graylog-labs / graylog-plugin-metrics

Graylog output plugin for Graphite and Ganglia

Metrics Plugin for Graylog

An output plugin for integrating Graphite, Ganglia and StatsD with Graylog.

Required Graylog version: 2.0 and later

Please use version 1.2.0 of this plugin if you run Graylog 1.x

Installation

Download the plugin and place the .jar file in your Graylog plugin directory. The plugin directory is the plugins/ folder relative from your graylog-server directory by default and can be configured in your graylog.conf file.

Restart graylog-server and you are done.

Build

This project is using Maven and requires Java 8 or higher.

You can build a plugin (JAR) with mvn package.

DEB and RPM packages can be build with mvn jdeb:jdeb and mvn rpm:rpm respectively.

Plugin Release

We are using the maven release plugin:

$ mvn release:prepare
[...]
$ mvn release:perform

This sets the version numbers, creates a tag and pushes to GitHub. TravisCI will build the release artifacts and upload to GitHub automatically.

graylog-plugin-metrics's Issues

Graylog node metrics to graphite

Is it possible to send graylog server node metrics to graphite via this plugin? If yes, how to do it?
For example org.graylog2.journal.entries-uncommitted metric.

Metrics are send always as Integer Values

We send decimal values in Gelf Messages to a Graylog2 Stream. Elasticsearch _mapping shows the field as a Double Type. The defined output to our graphite sends only Integer values.
We checked this with a tcpdump.

Is this a configuration issue ?

I tried to understand the code and it seams that a values are converted into long values.
....metricValue.longValue());

Thanks for reading.

Add spoof name configuration for ganglia export

Spoof name could be passed as last parameter of to GMetrcic constructor

Expose graylog metrics as jmx

Can we expose graylog metrics as JMX?

Support for counters

Please consider supporting counters as well as gauges ....

"impossible" results with graylog-plugin-metrics/graphite

We have just started using graylog-plugin-metrics to relay metrics to our graphite instance, and we're finding extremely curious results, like this (the times on this are at GMT-0600, so 10:00 on the graph is 16:00 GMT):

The graph is built from graylog absorption of nginx access.log entries, which have request_time and upstream_response_time fields. Since the upstream response time must be completely contained within the request time, request_time should be strictly greater than upstream_response_time (it is not possible for a request to have a greater request_time than an upstream_response_time).

Note that it is explicitly possible for a request to not have an upstream_response_time (if it is nginx serving a static file), but we've explicitly adjusted for that -- the graylog stream that we are dealing with restricts events to only those that include an upstream_response_time.

I've validated that within graylog itself, the reality holds and that upstream_response_time is always less than request_time:

From 16:07 to 16:08, mean request_time is 193.43 and mean upstream_response_time is 175.78.
From 16:08 to 16:09, mean request_time is 190.33 and mean upstream_response_time is 163.96.
From 16:09 to 16:10, mean request_time is 219.43 and mean upstream_response_time is 183.53.
From 16:10 to 16:11, mean request_time is 215.08 and mean upstream_response_time is 186.41.

... thus, the data in graylog looks good. The request_time and upstream_response_time fields are sent (as histograms, via graylog-plugin-metrics) to graphite, and result in the following graphite data, however:

$ whisper-fetch.py --from=1458230820 --until=1458231060 request_time/mean.wsp 
1458230880  192.900000
1458230940  152.620000
1458231000  221.990000
1458231060  206.160000

$ whisper-fetch.py --from=1458230820 --until=1458231060 upstream_response_time/mean.wsp 
1458230880  171.720000
1458230940  223.010000
1458231000  202.560000
1458231060  191.670000

As you can see, the data for 1458230940 (16:09 GMT) shows a significantly higher mean upstream_response_time than request_time. This should not be possible.

As the data in graylog appears correct, I can only assume that the metrics plugin is applying a transform to the data as it builds the necessary histogram, and that this is leading to incorrect output being sent to graphite.

Please let me know if there is further debugging that I can do on this front to help aid in diagnosing this issue.

Allow flexible metric name construction

Currently this plugin does not support using field content as part of the metric name that gets shipped to Graphite or Ganglia which in many cases is desirable ...

If a field called 'username' contains the value 'user_a' I would like to be able to create metrics such as 'authentication.user_a.failed' and increment that for every failed auth attempt.

Likewise for a field called 'src_ip' with the field content '1.2.3.4' I would like to be able to create a metric such as 'connections.1.2.3.4' and increment that for every failed auth attempt.

This could also be domain names, SSL certificates, port number, http error codes, etc

Please consider adding statsd to the list of supported protocols

There is a heap of tools that support statsd so please consider adding support to this plugin :)

Unable to send metrics to stats.

Hi,

We are trying to send some metrics to statsd on port 8125 .

Here are the plugin setup.

However I am getting following error on statsd logs.
12 Mar 10:57:04 - DEBUG: Bad line: 1 in msg "app.org.api_rpc_request.ur83dnu2fp4s80evc63n6z6t2mon05uo 1.00 1520852224"

We have tried changing URL from graphite://graphite.example.co:8125 to statsd://graphite.example.co:8125 but it wont send any data.

What are we doing wrong? Any help will be appreciated.

Thanks,
Vady

Feature Request: proceed number of matches to graphite etc.

First of all,
thank you for this plugin!

If I may issue a feature request: It would be great to optionally proceed only numbers of matches to graphite. For example, define a stream in graylog and push number of messages matching this stream to graphite/whisper. Or is it already implemented?
Many thanks and regards,
Ulf