diamond's People
Forkers
ooshlablu timwee oxcd8o robinedwards nareshov philipcristiano jbuchbinder jjhuff bastichelaar ross pablolb turbulenz suan tomplus dwatson marlier martindk80 steveberryman mechanism kippandrew unatine dotscreen microsigns saz bkjones welterde sbambach edolnx cbarraford adrianmoisey adreyer oldelvet richardc vimeo bcakipp patricksmith bfirsh gjedeer phobos182 konarkmodi songza sonatype dieterbe timdev arunthampi cosm boinger dqminh pandemicsyn meonkeys mate1 wonko nitinka nitink-agarwal omgjlk hedehung chiehwen servana-cookbooks keiouu drio richg obazoud chrisboulton tinle pulseenergy briceburg sckevmit bdelbosc ciranor mmorga bfritz lreed lambdafu tommyulfsparre omar-eyeviewdigital pombredanne barnybug yandex-load auser bearstech wadey bitprophet dreadatour classmarkets andreslagarcavilla jpmens revoohc mopub jorgebastida avereha mannya2k chamarag locationlabs jwilder captsens henner josexie benjamink erezzarum jkrauskadiamond's Issues
Is there a way to disable a collector from the config file?
I can just remove the collector ofcourse, but I wonder if there is a way to let's say, specify enabled = 1 in the config. I can build it in otherwise...
Diamond does not automatically logrotate its logs.
Given that Diamond uses Python's logging utility, perhaps it should implement log rotation using that?
Stop/start script does not work properly in Ubuntu
/etc/init.d/diamond stop does not work in ubuntu
upstart is handling these things in Ubuntu. while debian require /etc/init.d/*
I can provide an upstart script for ubuntu if you want.
I did not wanted to add it myself, because I got stuck in the Ant configurations... so if you have a way to separate debian from ubuntu, the package can contains an upstart script instead of init.d
PostgresqlCollector is missing unit test
Custom hostname for collectors
Hi,
Maybe I understood incorrectly how to use the custom hostname...
Or maybe it is broken?
Here how I got it to work...
Best regards,
Pablo
RabbitMQCollector - raise NetworkError("Error: %s %s" % (type(out), out))
Traceback (most recent call last):
File "/Users/kormoc/Dropbox/Projects/Diamond/src/diamond/collector.py", line 177, in _run
self.collect()
File "src/collectors/RabbitMQCollector/RabbitMQCollector.py", line 35, in collect
for queue in client.get_queues():
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pyrabbit/api.py", line 420, in get_queues
queues = self.http.do_call(path, 'GET')
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pyrabbit/http.py", line 97, in do_call
raise NetworkError("Error: %s %s" % (type(out), out))
NetworkError: Error: <class 'socket.error'> [Errno 61] Connection refused
LMSensorsCollector is missing unit test
Failed to import module: openvpn -- python 2.4 / centos 6.2 / x64
[2012-07-13 14:52:39,019] [MainThread] Failed to import module: openvpn. Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/diamond/server.py", line 156, in load_collectors
mod = import(modname, globals(), locals(), ['*'])
File "/usr/share/diamond/collectors/openvpn/openvpn.py", line 89
with open(filename, 'r') as fd:
^
SyntaxError: invalid syntax
i have default python running:
Python 2.4.3 (#1, Jun 18 2012, 08:55:23)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux2
PowerDNSCollector is missing unit test
Move Handlers
Move handlers to handlers/null.py from handlers/NullHandler.py
Debian update to run as diamond user by default?
Hi all,
I haven't used Diamond extensively, and therefore haven't tried the majority of plugins, but what are your thoughts around these types of changes to the Debian scripts?
https://github.com/mattt416/Diamond/commit/3eda762387c9a918ba6fc5b2c23adece1a14666b
It'd probably be wise to ignore the local mod I made to diamond.conf.example, as that would conflict with other installation mediums.
One additional thing I noticed is the pid file is getting created w/ a perm of 666, most likely due to os.umask(0) in /usr/bin/diamond. Should the pid file be locked down a bit more than that?
Anyway, perhaps someone with some Debian packaging experience could comment?
Thanks!
Question: collectors config ?
I suppose that in conf/collectors were example config files, but now this directory is empty and is listed in .gitignore ... I think it's mistake.
RedisCollector is missing unit test
Multiple Collector instances
Hi,
Today I was trying to setup Diamond to collect data from several servers, each containing multiple instances of memcached.
I cannot see how to accomplish this right now, as the collectors are added to a dict with their class name as key.
I have created a branch in my fork with a solution: https://github.com/pablolb/Diamond/tree/multiple-collector-instances.
But maybe you have a workaround or you'd prefer a better way to solve this...?
In the first commit (pablolb@f2397aa) I scan collectors config directory and schedule new instances of the collector.
In the second commit, I use MemcachedCollector as an example as how this can be used... I've seen this used in Genttoo network scripts I think (using symlinks and interpreting the actual name)
pablo@localhost collectors $ ls -l
total 4
-rw-r--r-- 1 root root 61 Mar 5 22:40 MemcachedCollector.conf
lrwxrwxrwx 1 root root 23 Mar 5 22:40 MemcachedCollector-localhost:11212.conf -> MemcachedCollector.conf
pablo@localhost collectors $ cat MemcachedCollector.conf
enabled=True
override_hostname=True
custom_name_is_host=True
Typo in init script causes daemon not to stop correctly on redhat systems
Therefore, restarts are broken too -- causing two instances of diamond to run.
At https://github.com/BrightcoveOS/Diamond/blob/master/dist/diamond.init.in#L77 :
-killproc -p ${DIAMODN_PIDFILE} $PROG +killproc -p ${DIAMOND_PIDFILE} $PROG
ConnTrackCollector - NameError: global name 'COMMAND' is not defined
Traceback (most recent call last):
File "/Users/kormoc/Dropbox/Projects/Diamond/src/diamond/collector.py", line 177, in _run
self.collect()
File "src/collectors/ConnTrackCollector/ConnTrackCollector.py", line 26, in collect
if not os.access(COMMAND[0], os.X_OK):
NameError: global name 'COMMAND' is not defined
reason for default precision to be 0?
Is there a reason for this.. I've noticed on servers with little traffic the bits are almost always 0 unless you change the precision to something like 6. It seems to break checks that might require a sub 1 metric since it will always 0 it out. I didn't see a config option for this also.
Debian package doesn't build
Trying to build .deb on Debian Squeeze, here's the last lines of output where the error occurs:
dh_usrlocal: debian/diamond/usr/local/bin/diamond is not a directory
rmdir: failed to remove `debian/diamond/usr/local/bin': Directory not empty
dh_usrlocal: rmdir debian/diamond/usr/local/bin returned exit code 1
make: *** [binary] Error 1
dpkg-buildpackage: error: fakeroot debian/rules binary gave error exit status 2
debuild: fatal error at line 1325:
dpkg-buildpackage -rfakeroot -D -us -uc failed
It appears that you're not allowed to install binaries in /usr/local/bin
on Debian? For now I'm not sure how to fix this...
PowerDNSCollector - OSError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/Users/kormoc/Dropbox/Projects/Diamond/src/diamond/collector.py", line 177, in _run
self.collect()
File "src/collectors/PowerDNSCollector/PowerDNSCollector.py", line 27, in collect
sp = subprocess.Popen([self.config['pdns_control'], "list"], stdout=subprocess.PIPE)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
VarnishCollector is missing unit test
Hadoop Collector issues
Per #72 (comment)
We need to change the way we build the metric keys for the Hadoop collector, as it collides on certain metrics lines in the log file.
tehmaze might be able to get a more official idea of what keys to use for the metrics.
Move collector tests
Move collector tests to something like "src/collectors/ExampleCollector/test/testexample.py". This is more pythonic,
and keeps tests and code separate.
UPSCollector is missing unit test
Force collector scan/reload conf files on sighup/siguser
It would be very useful to be able to send a signal to diamond to force a collector reload/config reload on demand rather then waiting for the interval to expire.
Move Collectors
Collectors should be
src/collectors/loadavg/loadavg.py
The class name can still be LoadAverageCollector.
config.py failing under python 2.4 / centos 6.2 / x64
cent02 Diamond]$ sudo python config.py
File "config.py", line 164
except IOError as (errno, strerror):
^
SyntaxError: invalid syntax
MongoDBCollector - raise AutoReconnect(', '.join(errors))
Traceback (most recent call last):
File "/Users/kormoc/Dropbox/Projects/Diamond/src/diamond/collector.py", line 177, in _run
self.collect()
File "src/collectors/MongoDBCollector/MongoDBCollector.py", line 35, in collect
conn = pymongo.Connection(self.config['host'],read_preference=ReadPreference.SECONDARY)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymongo/connection.py", line 290, in __init__
self.__find_node()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/pymongo/connection.py", line 586, in __find_node
raise AutoReconnect(', '.join(errors))
AutoReconnect: could not connect to localhost:27017: [Errno 61] Connection refused
Feature Request: over the network collector updates
Would be really fantastic to be able to push out collector scripts over the network to hosts, rather than my current setup, which is to create an rpm, and push that out.
With that feature, you could have a central console of the configs, and make it easy to push out changes, enable/disable plugins, etc.
Just a thought.
Sean
BindCollector - TypeError: %d format: a number is required, not str
Traceback (most recent call last):
File "/Users/kormoc/Dropbox/Projects/Diamond/src/diamond/collector.py", line 177, in _run
self.collect()
File "src/collectors/BindCollector/BindCollector.py", line 44, in collect
req = urllib2.urlopen('http://%s:%d/' % (self.config['host'], self.config['port']))
TypeError: %d format: a number is required, not str
network.py error on CentOS 6.2 x64
new to this.. so if a newbie error please excuse.
trying to get network.py working on a centos 6.2 64bit system and see an error...
Failed to import module: network. Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/diamond/server.py", line 156, in load_collectors
mod = import(modname, globals(), locals(), ['*'])
File "/usr/share/diamond/collectors/network/network.py", line 64
greed = '0-9' if self.config['greedy'].lower == 'true' else ''
^
SyntaxError: invalid syntax
my NetworkCollector.conf is such:
cat /etc/diamond/collectors/NetworkCollector.conf
byte_unit = megabyte
interfaces = eth0
enabled = True
greedy = true
i have default python running:
Python 2.4.3 (#1, Jun 18 2012, 08:55:23)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux2
CPUCollector
Maybe I'm dense, but it seems that this collector only uploads jiffies from /proc/stat, and I can't for the life of me figure out how to get graphite to display percentages.
What am I missing?
systemcollectors.CPUCollector missing newer Linux cpu metrics
CPUCollector currently collects user, nice, system, idle, iowait. It does not collect additional columns which appear with newer Linux kernels.
Here are the columns which may appear (in order):
- irq - since 2.6.0-test4
- softirq - since 2.6.0-test4
- steal - since 2.6.11
- guest - since 2.6.24
A sample line from 2.6.18-164.15.1.el5xen (only 8 columns):
cpu 54524917 1184 19017460 2207399508 25501525 0 1380737 5191044
A sample line from 3.2.0-25-generic-pae (10 columns!?!):
cpu 667245 6544 90394 33314088 111332 0 1240 0 0 0
SNMPInterfaceCollector inherits from unimportable class?
SNMPInterfaceCollector inherits from SNMPCollector, but I don't see how it's expected to be importable, since /usr/share/diamond/collectors (on my system) was not installed into any kind of Python-aware namespace. Browsing through some past commits, it seems that SNMPCollector used to live inside of diamond.collector, but it doesn't live there anymore.
As a result, when I start diamond, I get errors when it tries to load SNMPInterfaceCollector, saying it can't import SNMPCollector.
Standardize subprocess usage in collectors
We have a bunch of collectors that all spawn off subprocesses to gather metrics. We should standardize on a way to handle the following:
- Binary path configuration, including short $PATH style usage
- Missing binaries
HAProxyCollector is missing unit test
Feature Request: Auto disable of failed tests
How about a config parameter that can be set to a number of failures for a collector before diamond stop trying to run the collector. If I don't have mysql installed, run it X times, but then only check once per day/per hour/configurable time period.
UPSCollector - OSError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/Users/kormoc/Dropbox/Projects/Diamond/src/diamond/collector.py", line 177, in _run
self.collect()
File "src/collectors/UPSCollector/UPSCollector.py", line 24, in collect
p = subprocess.Popen(['/bin/upsc', self.config['ups_name']], stdout=subprocess.PIPE)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1249, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
No such file or directory: '/var/log/diamond/diamond.log'
Hi,
I run make buildrpm
, installed diamond on centos 6.2, copied the config and adapted it, then when trying to start it:
[root@nyvimeodfs1 ~]# /etc/init.d/diamond start
Starting diamond: Error occurs when initialize logging: [Errno 2] No such file or directory: '/var/log/diamond/diamond.log'
[ OK ]
[root@nyvimeodfs1 ~]# less /var/log/diamond/archive.log
/var/log/diamond/archive.log: No such file or directory
[root@nyvimeodfs1 ~]# pgrep -fl diamond
[root@nyvimeodfs1 ~]# mkdir /var/log/diamond/
[root@nyvimeodfs1 ~]# /etc/init.d/diamond start
Starting diamond: WARN: Bogus pid file was found. I deleted it.
[ OK ]
[root@nyvimeodfs1 ~]# less /var/log/diamond/archive.log
[root@nyvimeodfs1 ~]# pgrep -fl diamond
20168 /usr/bin/python /usr/bin/diamond -p /var/run/diamond.pid
so, when trying to start, it aborts because the log dir does not exist. a (temporary?) fix is to manually create the log dir as shown above.
DiskUsageCollector error
$ sudo diamond --configfile=/etc/diamond/diamond.conf --foreground -r /usr/share/diamond/collectors/DiskUsageCollector/DiskUsageCollector.py
[2012-05-28 01:20:17,585] [MainThread] Changed UID: 0 () GID: 0 ().
[2012-05-28 01:20:17,586] [MainThread] Loaded Handler: diamond.handler.GraphitePickleHandler.GraphitePickleHandler
[2012-05-28 01:20:17,587] [MainThread] GraphiteHandler: Failed to connect to brampton1-admin01.fatbox.ca:2004. [Errno -2] Name or service not known
[2012-05-28 01:20:17,587] [MainThread] Loaded Handler: diamond.handler.ArchiveHandler.ArchiveHandler
[2012-05-28 01:20:17,589] [MainThread] Loading Collectors from: /usr/share/diamond/collectors/DiskUsageCollector
[2012-05-28 01:20:17,589] [MainThread] Loaded Module: DiskUsageCollector
[2012-05-28 01:20:17,590] [MainThread] Loaded Collector: DiskUsageCollector.DiskUsageCollector
[2012-05-28 01:20:17,590] [MainThread] Initialized Collector: DiskUsageCollector
[2012-05-28 01:20:17,590] [MainThread] Scheduled task: DiskUsageCollector
[2012-05-28 01:20:17,590] [MainThread] Started task scheduler.
[2012-05-28 01:20:18,591] [Thread-1] Collecting data from: DiskUsageCollector
[2012-05-28 01:20:18,592] [MainThread] Stopping task scheduler.
[2012-05-28 01:20:18,593] [Thread-1] Traceback (most recent call last):
File "/usr/lib/pymodules/python2.6/diamond/collector.py", line 177, in _run
self.collect()
File "/usr/share/diamond/collectors/DiskUsageCollector/DiskUsageCollector.py", line 112, in collect
value = diamond.convertor.binary.convert(value = value, oldUnit = 'kB', newUnit = self.config['byte_unit'])
AttributeError: 'module' object has no attribute 'convertor'
[2012-05-28 01:20:18,593] [MainThread] Stopped task scheduler.
[2012-05-28 01:20:18,593] [MainThread] Exiting.
$ cat /etc/diamond/collectors/DiskUsageCollector.conf
enabled = True
$ dpkg -l diamond
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Description
+++-======================================-======================================-============================================================================================
ii diamond 0.2.0 System statistics collector for Graphite.
Sadness.
crashes when trying to run as user nobody
runs fine as root, but when trying to run as user nobody, the process stops and doesn't say why.
also, I needed to touch and chown the logfile first. see below:
[root@nyvimeostatsdev1 ~]# egrep 'user|group' /etc/diamond/diamond.conf
# Leave empty to use the current user
user = nobody
# Leave empty to use the current group
group = nobody
username = root
[root@nyvimeostatsdev1 ~]# /etc/init.d/diamond start
Starting diamond: Error occurs when initialize logging: [Errno 13] Permission denied: '/var/log/diamond/diamond.log'
[ OK ]
[root@nyvimeostatsdev1 ~]# ps aux | grep diamond
root 4488 0.0 0.0 103232 820 pts/0 S+ 05:01 0:00 grep diamond
[root@nyvimeostatsdev1 ~]# less /var/log/diamond/diamond.log
/var/log/diamond/diamond.log: No such file or directory
[[root@nyvimeostatsdev1 ~]# touch /var/log/diamond/diamond.log
[root@nyvimeostatsdev1 ~]# chown nobody:nobody /var/log/diamond/diamond.log
[root@nyvimeostatsdev1 ~]# /etc/init.d/diamond start
Starting diamond: [ OK ]
[root@nyvimeostatsdev1 ~]# ps aux | grep diamond
root 4542 0.0 0.0 103232 820 pts/0 S+ 05:01 0:00 grep diamond
[root@nyvimeostatsdev1 ~]# cat /var/log/diamond/diamond.log
[2012-07-17 05:01:37,251] [MainThread] Changed UID: 99 (nobody) GID: 99 (nobody).
[2012-07-17 05:01:37,251] [MainThread] Detaching Process.
[root@nyvimeostatsdev1 ~]#
ping collector issue?
Hi.
Is it ok that ping collector generate such path [path_prefix].[host].ping.ping.[target_host]?
NginxCollector is missing unit test
Failed to import module: hadoop -- python 2.4 / centos 6.2 / x64
[2012-07-13 14:52:38,946] [MainThread] Failed to import module: hadoop. Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/diamond/server.py", line 156, in load_collectors
mod = import(modname, globals(), locals(), ['*'])
File "/usr/share/diamond/collectors/hadoop/hadoop.py", line 51
with open(filename, 'r') as fd:
^
SyntaxError: invalid syntax
i have default python running:
Python 2.4.3 (#1, Jun 18 2012, 08:55:23)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux2
Debian Install has an error message
Imported from #43
Debian installs work fine, but throw this error on install. We should fix this.
vagrant@graphite:~/build/Diamond/build [master]$ sudo dpkg -i diamond_2.0.0_all.deb
Selecting previously deselected package diamond.
(Reading database ... 47779 files and directories currently installed.)
Unpacking diamond (from diamond_2.0.0_all.deb) ...
Setting up diamond (2.0.0) ...
insserv: warning: script 'K01statsd' missing LSB tags and overrides
insserv: warning: script 'statsd' missing LSB tags and overrides
Traceback (most recent call last):
File "/usr/bin/diamond", line 18, in <module>
from diamond.server import Server
File "/usr/lib/pymodules/python2.6/diamond/server.py", line 21, in <module>
from diamond.handler.Handler import Handler
ImportError: No module named handler.Handler
Processing triggers for python-support ...
MySQLCollector is missing unit test
Invoking diamond with -r (run once) doesn't actually run once
The docs say it will run the given collector once and exit, but it lies. It runs the collector forever.
Disk Collector only works in python2.6
Looks like namedtuple() was used, which breaks on 2.4 installations.
[2012-01-13 09:43:19,380] [MainThread] Failed to import module: DiskSpaceCollector. Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/diamond/server.py", line 143, in load_collectors
mod = import(modname, globals(), locals(), ['*'])
File "/usr/share/diamond/collectors/Disk/DiskSpaceCollector/DiskSpaceCollector.py", line 19
name = 'root' if name == '_' else name
^
SyntaxError: invalid syntax
[2012-01-13 09:43:19,380] [MainThread] Failed to import module: disk. Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/diamond/server.py", line 143, in load_collectors
mod = import(modname, globals(), locals(), ['*'])
File "/usr/share/diamond/collectors/Disk/disk.py", line 6, in ?
from collections import namedtuple
ImportError: cannot import name namedtuple
ImportError: No module named handler.Handler
Mac OS X Lion, installed Diamond from git. Running it gives this error:
$ diamond
Traceback (most recent call last):
File "/usr/bin/diamond", line 40, in
from diamond.server import Server
File "/Library/Python/2.7/site-packages/diamond/server.py", line 21, in
from diamond.handler.Handler import Handler
ImportError: No module named handler.Handler
Seems like I'm missing something?
Question: Dashboard for Diamond?
Hi,
we are currently deploying diamond on some of our production machines and the tons of data gathered by diamond paired with the performance of graphite are just awesome!
Now we came to the point where we need a easy to use and in the best case, an automatically managed dashboard for the machines monitored by diamond.
Are there any ready to use implementations?
If there aren´t any, i´m willing to build something for example using gdash if there is serious interest by the community - i would create a sperate github project for this.
I just want to ask first if there are any existing implementations i missed to find.
Regards,
Christian
`make buildrpm' doesn't succeed on CentOS 6
Hey,
The new setup.py based buildrpm target doesn't quite work: https://raw.github.com/gist/4210d96d443624174c74/7e98613585e9d2789420c5581538876337e1194b/gistfile1.txt
Opening an issue here while I explore fixes.
-Naresh
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.