gingerlime / graphite-fabric

fabric-graphite is a fabric script to install Graphite and all dependencies on a debian-based host (with optional vagrant support)

Graphite-Fabric - a fabric installer for Graphite

fabric-graphite is a fabric script to install Graphite, Nginx, uwsgi and all dependencies on a debian-based host.

Why?

I was reading a few interesting posts about graphite. When I tried to install it however, I couldn't find anything that really covered all the steps. Some covered it well for Apache, others covered Nginx, but had steps missing or assumed the reader knows about them etc.

I'm a big fan of fabric, and try to do all deployments and installations using it. This way I can re-run the process, and also better document what needs to be done. So instead of writing another guide, I created this fabric script.

Requirements

• Workstation running python (version 2.7 recommended). All platforms should be supported.
• Fabric - can be installed via pip install fabric or easy_install fabric
• a new VPS/Dedicated server running a Debian-based distribution (Debian, Ubuntu etc) - or
• Virtualbox with Vagrant (see below)

Target Host

Best to execute this on a clean virtual machine running Debian 6 (Squeeze). Also tested successfully on Ubuntu 12.04 VPS.

Installation Instructions

run fab graphite_install -H root@{hostname} (hostname should be the name of a virtual server you're installing onto)

It might prompt you for the root password on the host you are trying to instal onto.

You can use it with a user other than root, as long as this user can sudo.

During the installation, you would be asked to set up the django superuser account. You might want to create an account, but it's not strictly necessary. If you answer no, the installation will still work fine.

Vagrant

you can now install graphite using vagrant on your local virutalbox. If you have vagrant installed, simply use

$ pip install fabtools
$ vagrant up
...
$ fab vagrant graphite_install

Once installed, you can access your graphite server on http://localhost:8080/graphite

Giraffe Dashboard

Giraffe is also included and installed. It is installed under /opt/graphite/webapp/giraffe and is accessible on http://your-graphite-url/giraffe/.

OPTIONAL: Install statsd.

After installing graphite, run fab statsd_install -H root@{hostname}

statsd is a powerful, Node.js tool for aggregating data sent over UDP, which means it is fairly robust and allows for heaping large amounts of data into your stat backend (graphite) without fear of downtime or failures.

After Installation

Simply open your browser and go to http://[your-hostname]/graphite/ ! It should be up and running.

Of course there's a lot more configuration to be done, but at the very least you should have a working environment to play with Graphite.

Security

The scripts are meant to be used as a starting point for experimenting and trying out graphite and statsd, but they are by no means 'production-ready'. Further configuration would be required. This is of particular concern when considering security. The installed host(s) will have open ports listening, and access to the graphite web console is without any authentication.

Thanks

Contributors

Thanks to the authors of these online guides and resources who provided very useful information that I stitched together into this fabric script, and others who provided inspiration about Graphite in General:

• Graphite Docs
• frl1nuX - Graphite and Nginx
• Agile Testing - Installing and configuring Graphite
• Corey Goldberg - Installing Graphite 0.9.9 on Ubuntu 12.04 LTS
• Tom Purl - Installing Graphite on Ubuntu 10.4 LTS
• Bryan Helmig - for adding installation of statsd

Although not installed with this fabric script, I'd love to try these some time:

• Graphene
• GDash
• Logster

DISCLAIMER

Please try this at your own risk. Please run this only with a newly installed host that you can easily throw away! I tested it with both Debian 6 and Ubuntu 12.04 successfully. However, you may experience different results.

Help and Contribution

I'd be happy to try to help if I can, but given the complexity of linux-based operating-systems, and my limited time, I might not be able to know why a certain operation fails or an error is generated. Feel free to fork for your own special requirements or needs.

graphite-fabric's Issues

init.d/carbon pidfile logging prevents startup in Ubuntu

Hi there
Thanks very much for this fabric - I'm a windows dev who knows a very small amount of Linux to get by and this enabled me to get statsd/graphite going a few months ago in my dev environment.

I just built out the production server on a clean EC2 Ubuntu Server 12.04 instance and ran into a problem where carbon wasn't starting automatically on boot. The console.log showed permission denied trying to create the /var/run/carbon/carbon.pid file.

If I remove the pidfile bits from /etc/init.d/carbon that were added in this commit:
409bd88
then it starts up fine.

Just fyi - I really don't know what the pid file is for and maybe there is good reason to have it. I'm just fumbling in the dark here in Linux :)

ValueError: 'config/nginx' is not a valid local path or glob.

Running it on a fresh vagrant image running 12.04:

...
[[email protected]:2200] sudo: mkdir -p /etc/nginx
[[email protected]:2200] sudo: mkdir -p /var/log/nginx
[[email protected]:2200] sudo: chown -R www-data: /var/log/nginx
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/fabric/main.py", line 743, in main
    *args, **kwargs
  File "/usr/local/lib/python2.7/site-packages/fabric/tasks.py", line 368, in execute
    multiprocessing
  File "/usr/local/lib/python2.7/site-packages/fabric/tasks.py", line 264, in _execute
    return task.run(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/fabric/tasks.py", line 171, in run
    return self.wrapped(*args, **kwargs)
  File "/Users/josephg/Downloads/graphite-fabric/fabfile.py", line 61, in graphite_install
    put('config/nginx', '/etc/init.d/', use_sudo=True)
  File "/usr/local/lib/python2.7/site-packages/fabric/network.py", line 578, in host_prompting_wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/fabric/operations.py", line 381, in put
    raise ValueError(err)
ValueError: 'config/nginx' is not a valid local path or glob.
Disconnecting from 127.0.0.1:2200... done.

Fabric fails to start carbon-cache.py on Ubuntu because of missing twisted packages

Had the following failure, seems to work if I add "twisted python" to the list of installed packages in fabfile.py.

[root@precise64] sudo: /etc/init.d/carbon start
[root@precise64] out: Starting script carbon
[root@precise64] out: Traceback (most recent call last):
[root@precise64] out:   File "/opt/graphite/bin/carbon-cache.py", line 28, in <module>
[root@precise64] out:     from carbon.util import run_twistd_plugin
[root@precise64] out:   File "/opt/graphite/lib/carbon/util.py", line 21, in <module>
[root@precise64] out:     from twisted.scripts._twistd_unix import daemonize
[root@precise64] out: ImportError: cannot import name daemonize
[root@precise64] out: 

Solved issue: TypeError: fetch() takes at most 3 arguments (4 given)

If someone needs help with this JavaScript Console error after installing Graphite, I have solved it by upgrading Whisper to the latest version with this command:

sudo pip install 'git+https://github.com/graphite-project/whisper.git#egg=whisper' --upgrade

Tested on Ubuntu 12.04.