* This is still work in progress.
Cloud computing is the in thing. It is now possible for anyone to manage, control and run software on multiple machines. Horizontal scaling is increasingly becoming the most used used,over-used and abused word in the industry. Every startup is doing something that involves horizontal scaling and is on the cloud.
There have been some amazing successes and companies are becoming more and more open. There has been a lot of activity towards open source software. And there has been a new generation of softwares and tools being open sourced on git.
One such contribution is statsd from etsy. Statsd is easily combined with grpahite to monitor several servers on a cluster. This all works, and works well on unix/linux and mac based machine. Open source does not really help on windows. Making any of these software work on windows is still a nightmare. Getting statsd to work on a ubuntu is as simple as following well defined set of instructions. All one needs to do is install graphite and then follow this statsd blog. These same steps become zillion more steps on windows.
This blog is an attempt to get graphite and statsd working on a windows 2003 server. We start from very basic, by installing python first and then installing graphite and statsd respectively.
I installed nodejs in its default directory on windows and it went into c:/program files. To run node, it needs to be in your PATH. Now we can test statsd and nodejs both with below command:
node stats.js yourConfileFileForStatsd.js
Most likely, one would get an error relating to problems with watchFile function.In node 0.6+ watchFile has been replaced with watch. To get rid of this error replace all occurrences of watchFile with watch.
Once done, we can see that node is able to run stats.js properly.
Now lets install graphite following instructions here. One of the very first errors we would get is related to txamqp dependency while installing carbon:
error: Unable to find vcvarsall.bat
I followed several articles on the internet and finally came to this stackoverflow link. This link suggests to install mingw32 to get rid of the error.
This error makes way for a new one:
gcc: error: unrecognized command line option '-mno-cygwin'
Another round of investigation leads to this stackoverflow question. (stackoverflow is the best thing that has happend to mankind since programming started). That answer gets us past that error.
But this is windows!, we are not yet done. We are still trying to install carbon!. Another error now (not really a window related though):
path '/opt/graphite/lib' cannot be absolute!
Fix for this one is to modify carbon source (downloaded with pip for python). Change setup.cfg file and change prefix variable to the correct one pointing to the local installation folder.
With the last carbon error past us, the installation should have been a success. As per the graphite installation instructions now download checkdependencies.py and run. Install any missing requirements. In my case i was missing cairo.
Install cairo from http://www.lfd.uci.edu/~gohlke/pythonlibs/#pycairo
Now install graphite-web for python. Since graphite-web is a web application for python, we would need a server to run it own, which is apache. To host python web application on apache there are two options:
Which one to choose? Stackoverflow to the rescue and we come to know that mod_python is not the way to go forward.
So we pick mod_wsgi and configure it as described here. Create a .wsgi script and create a virtual host in apache httpd.conf file. Check and see if we get a running wsgi script.
Now we need to create a graphite database as per the original graphite installation instructions. And the errors do not seem to go away. On running
sudo python manage.py syncdb
error on syncdb about timezone.
Install pytz for python with
pip install pytz
timezone errors are now gone and we get below error:
settings.DATABASE is not properly configured.
This error is due to wrong settings for dianjo in settings.py file. local_settings.py is not picked up for some reason. The error was due to a wrong graphite db path. Point that path to a local path and the error goes.
Finally, we are at a point where we can start carbon.
and…. another error!
no module pwd
pwd is unix module which is not available on windows.
Ang guess what….another error now in twisted module:
no module syslog
And this post continues….