redhat-cip / numeter

Main repository for numeter poller / storage / webapp. Graphs system in Python.

Numeter is a new and dynamic graphing solution made by some of the folks at eNovance. We use it as part of our cloud solutions. It is based on Python, sexy and highly scalable.

Documentation : https://numeter.readthedocs.org

Features :

• Graphs with dygraphs
• User and group management
• Use graph template configuration give by poller
• Already works with munin-node
• Poller Keep datas in cache in case of network failure
• Scalable architecture

How to get started :

• Visit the Official Website : http://enovance.github.com/numeter/

License : AGPLv3

Quick overview :

• The Numeter infrastructure : instances of every component may be added to handle the load.
• All components can be installed on the same server or on multiple servers depending on their specifications.
• Numeter is written in Python and uses Redis. We are considering going from a pull mode to a push mode using rabbitmq.

Current Numeter architecture :

• Poller : An agent installed on the servers for which graphs are desired. It gathers data and send them to an rpc. In case of network failure all datas are preserved and sent when network is back.
• Rpc : Receive data from poller and provide them to storage.
• Storage : Data is fetched from the rpc and then stored in WSP files. An HTTP API allows access to the data.
• Webapp : A Django webapp displays data using the js library like dygraphs

Screenshots :

numeter's Issues

Rewrite myRedisConnect with logger

Delete print and replace with logger

Webapp install (chown www-data)

Add chown in webapp makeffile :

chown root:www-data -R /usr/share/numeter/web-app
chmod g+w -R /usr/share/numeter/web-app/app/tmp

But with a if group www-data exist ...

And add some explanations in docs/webapp/install like the default group is www-data but if this group not exist don't forgot to chown with your group like httpd, ...

Add an auto-update checkbox on graph

It should be nice if graphs could be automatically refreshed to have the new informations displayed without the need to click on another graph then come back.

Some graphing library already manage the streaming.

setup.py develop broken

Hi,

Following the pycon.fr talk about numeter I tried to install it and faced a couple of issues.
Narrowing them down, the main reason is setup.py does weird things during installation which doesn't work with develop. setup.py develop just puts the source directory as it is in the python path but, there's no numeter.poller package for example.

I was curious about the reason the packages aren't in the numeter directory. Even if using a virtual package namespace, you can put things there (which is what I usually do).

Virtual Host in the poller

Some devices can't have a poller on them (box, switches, etc).
The poller on a Host X should be able to act as a "virtual poller" for the Host Y

Issue on webapp setup

seraf@Tagada-server:/opt/numeter# numeter-webapp syncdb
CommandError: One or more models did not validate:
multiviews.skeleton: "name": CharField cannot have a "max_length" greater than 255 when using "unique=True".

I followed the documentation with the latest master branch :
8f9ef25

Uwsgi config import error

I have an error with storage/numeter_uwsgi.py

In uwsgi log :

  File "./numeter_uwsgi.py", line 19, in <module>
    from myRedisConnect import *
ImportError: No module named myRedisConnect

I think i have a problem with uwsgi config : (print sys.path)

['', '/usr/local/lib/python2.7/dist-packages/PyRRD-0.1.0-py2.7.egg', '/usr/local/lib/python2.7/dist-packages/numeter_common-0.2.3.4-py2.7.egg', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/pymodules/python2.7']

with uswgi :

['.', '', '/usr/lib/python2.7/', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload']

So for the moment we have a quick patch in the debian wheezy:

debian-wheezy/debian/patches/100-fixe-uwsgi-storage-script-python-path.patch

Update database format in all README (poller/collector/storage)

Things like host1 need to be updated by hostID

    +--------+         +-------------+ 
    |  HSET  |         | INFOS@Host1 | ----->  if   : {JSON:Value:ID:...}
    +--------+         +-------------+   |-->  df   : {JSON:Value:ID:...}

Add logrotate config in debian-squeeze/wheezy branch

Add logrotate configuration in debian package. Exemple a file like :

• debian/numeter-poller.logrotate

/var/log/numeter/numeter-poller.log {
    daily
    missingok
    rotate 7
    compress
    notifempty
    create 640 root adm
}

Poller redis config

Change the redis dir config

dir /var/lib/redis

You must check the new dir exist or create it.

Manual install send cfg in wrong directory

Exemple poller :
/usr/local/etc/numeter/numeter_poller.cfg

API endpoint to push data

I use Numeter to graph my home automation.

I have a motion sensor and I would like to graph it. The problem is : I can't get his state in a interval as it works with trigger.
My home automation box should be able to do a call on an http endpoint when the trigger is fired.

I was thinking to push info directly in the storage api when the trigger is fired to avoid polling it. I know it breaks the scalling schema of Numeter as only the poller should know which storage API to use ... but in the case of a trigger and the need to "push" data, it's the only way I see.

What do you think about this ?