davidb / metrics-influxdb

A reporter for metrics which announces measurements to an InfluxDB server.

To the extent possible under law, Novaquark has waived all copyright and related or neighboring rights to this work.

The library provide :

• a lighter client than influxdb-java to push only series to an InfluxDB server.
• A reporter for metrics which announces measurements.

The library provide a lighter client than influxdb-java to push only metrics.

Dependencies :

• slf4j-api for logging.
• metrics-core, to provide, if you use InfluxdbReporter.

Install:

Released

 dependencies {
	compile 'com.github.davidb:metrics-influxdb:0.9.3'
 }

Dev

 repositories {
    maven { url "https://jitpack.io" }
 }
 dependencies {
	compile 'com.github.davidb:metrics-influxdb:-SNAPSHOT'
 }

Usage :

Using the Builder API and its defaults, it is easy to use InfluxdbReporter:

ScheduledReporter reporter = InfluxdbReporter.forRegistry(registry).build();
reporter.start(10, TimeUnit.SECONDS);

With the previous simple configuration, all defaults will be used, mainly:

• protocol: HTTP
• server: 127.0.0.1
• port: 8086
• authentication: none
• database name: metrics
• rates: converted to TimeUnit.SECONDS
• duration: converted to TimeUnit.MILLISECONDS
• idle metrics: do not report
• influxdb protocol: v09 line protocol
• ...

But you are free of course to define all settings by yourself :

final ScheduledReporter reporter = InfluxdbReporter.forRegistry(registry)
    .protocol(new HttpInfluxdbProtocol("http", "influxdb-server", 8086, "admin", "53CR3TP455W0RD", "metrics"))
    .convertRatesTo(TimeUnit.SECONDS)
    .convertDurationsTo(TimeUnit.MILLISECONDS)
    .filter(MetricFilter.ALL)
    .skipIdleMetrics(false)
    .tag("cluster", "CL01")
    .tag("client", "OurImportantClient")
    .tag("server", serverIP)
    .transformer(new CategoriesMetricMeasurementTransformer("module", "artifact"))
    .build();
reporter.start(10, TimeUnit.SECONDS);

And if you are still using v08 influxdb

final InfluxdbReporter reporter = InfluxdbReporter
    .forRegistry(registry)
    .protocol(new HttpInfluxdbProtocol("influxdb-server", 8086, "admin", "53CR3TP455W0RD", "metrics"))
    .v08()
    .build();
...

metrics-influxdb's Issues

integrate with CI

would be great to see this lib integrated with a CI plateform.

Travis-ci is a popular one for github project ; as easy as pushing a .travis.yml file at the root of the project which conatins for example:

language: java
jdk:
  - oraclejdk8

It is enough to at least receive some compile & test feedback for each commit and pull-request.

HTTPS support

please, could you add support of HTTPS when sending to influxdb ?
It seems to be as simple as allowing https instead of http in front of the url (but I'm no java dev...)

Thanks.

Unable to report to influxdb in 0.9.6

Here is the exception.. looks like "series" api deprecated.
Instead of using http api, why not use https://github.com/influxdata/influxdb-java driver?

2016-01-07 19:41:15.827 [metrics-influxdb-reporter-thread-1] WARN metrics_influxdb.InfluxdbReporter [298] - Unable to report to InfluxDB. Discarding data.
java.io.IOException: Server returned HTTP response code: 404for URL: http://localhost:8086/db/rqsmetrics/series?u=rqsuser&p=rqsuser&time_precision=s with content :'Not Found'
at metrics_influxdb.InfluxdbHttp.sendRequest(InfluxdbHttp.java:152) ~[metrics-influxdb-0.7.0.jar:na]
at metrics_influxdb.InfluxdbReporter.report(InfluxdbReporter.java:295) ~[metrics-influxdb-0.7.0.jar:na]
at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:116) [metrics-core-3.0.2.jar:3.0.2]

Can you provide the examples for using your lib to insert the data

Dear Sir,
I like your lib but I can not find the example for inserting data to influxDB and can you please kindly provide one. I did use it to import data from mysql to influxDB but I am not sure I use the resetRequest() correctly.
Also, because I have a billion data need to insert to influxDB and do you have any recommend way to using your lib to do this . bulk insert maybe
Thanks in advance

Richard Huang

No data in counter timeseries

I have simple spring boot application.

My reporter:

ScheduledReporter reporter = InfluxdbReporter.forRegistry(metricRegistry)
//				    .protocol(new HttpInfluxdbProtocol("http", "monitoring-influxdb", 8086, "admin", "53CR3TP455W0RD", "metrics"))
					//servicename.namespace.svc.cluster.local
				    .protocol(new HttpInfluxdbProtocol("http", "monitoring-influxdb.kube-system.svc.cluster.local", 8086, "admin", "53CR3TP455W0RD", "metrics"))
				    .prefixedWith("services.microservice." + appName + "." + hostname)
				    .convertRatesTo(TimeUnit.SECONDS)
				    .convertDurationsTo(TimeUnit.MILLISECONDS)
				    .filter(MetricFilter.ALL)
//				    .skipIdleMetrics(false)
//				    .tag("cluster", "CL01")
//				    .tag("client", "OurImportantClient")
//				    .tag("hostname", hostname)
//				    .transformer(new CategoriesMetricMeasurementTransformer("module", "artifact"))
				    .build();
				reporter.start(30, TimeUnit.SECONDS);

Constructor

metricRegistry.register("MKCustomGaugeIndex",             new Gauge<Integer>() {
    @Override
    public Integer getValue() {
index++;
//  	  int randomNum = ThreadLocalRandom.current().nextInt(0, 300 + 1);
  	  System.out.println(new Date ()+" MKCustomGaugeXYZ generated "+index );
        return (int) new Integer(index);
    }
})
	;

In my rest code I have gauges and counters

public User getByEmail(@RequestParam(value="email", defaultValue=".")String email) 

{
....
	//------------------------ custom counter ------------------------//
	Counter mkCustomCounter = metricRegistry.counter("mkCustomCounter");
	mkCustomCounter .inc();
	//------------------------ custom counter ------------------------//
	Timer t=metricRegistry.timer("endpoint.MKget-by-email");
	Timer.Context c =t.time();
...
 this.counterService.increment("get-by-email.MKtxnCount");
 this.counterService.increment("get-by-email.MKtxnCount2");
	
 this.gaugeService.submit("get-by-email.MKcustomgauge", randomNum );
 this.gaugeService.submit("get-by-email.MKcustomgauge2", randomNum *2);

c.stop();
		
}

In infuxdb I can see that I have all timesieries, but data only in gauge timesies.
Counter timeseries are empty.

I,ve tested my code with two reporter, your influxdb and graphite.

I can see that this issue exsists only with influxdb rerpoter.
In graphite I have all data (gauges and counters).
In influxdb I have all timeseries definition but data only in gauges

Unable to use jitpack repo configuration in transitive dependency

We have this package in a 'core' project that other projects depend on. When we were still using net.alchim31:metrics-influxdb:0.7.0, the dependency is downloaded from maven so it's pretty convenient for us. However since the change to this package and jitpack repo, what happened was the 'core' project can be built just fine, however all projects depend on it, and in turn all projects depend on those projects also can't be built. This is real problem because the total number of projects (including transitive ones) are ~ 300 repos.

Is it possible to publish this package to maven central? Thanks.

missing maven central deployment

I think no version was published to maven central since you changed the project group.
Please consider pushing the project again to central.

TCP Support

Hi.

Since telegraf now supports listening to "influx-db protocol" in tcp (influxdata/telegraf#481) - I would like to send the data to telegraf in tcp.

Tell me if you like the idea, I can create the PR by myself.

Timestamp precision for InfluxDB v0.9

InfluxDB v0.9 expect timestamp in nanoseconds epoch time by default, instead of milliseconds as in v0.8.

The reporter builder and measure provide default clock from Metrics lib, which report time in milliseconds. The issue can be work around by providing a customized clock, which override Clock#getTime() to report in nanoseconds. However, it might be better to add a precision field to reporter and measure, set it default to nanoseconds for v0.9.

Make Influxdb interface public

The Influxdb interface has default scope which makes it only accessible within the package.

As a consequence any attempt to call build() on InfluxdbReporter.Builder results in an IllegalAccessError when running from clojure (using 1.7.0)

Measurement/metric mappings and gauge grouping

Here https://github.com/iZettle/dropwizard-metrics-influxdb there is feature that tries to solve inconsistence between default dropwizard metrics naming and the InfluxDB time series model. I think it is complementary to transformation feature and it is worth to consider implementing these feature.

Locked down implementation of MeasurementReporter means you cannot add your own overrides for certain metric types

Please see #60 as a potential fix to this problem

Background for the issue is that influx/grafana reporting for timers reports timer datapoints regardless of whether there has been a metric recorded in the previous time frame block. Making TimerMeasurementReporter extensible allows me to add my own implementation that fixes this issue

InfluxDB v 0.12

Hi,

Im using version 0.12

Strating reported with code:

object MetricRegistryHolder {
  val registry = new MetricRegistry()
  val metricBuilder = new MetricBuilder(MetricName(""), registry)

}

trait MetricInstrumented extends nl.grons.metrics.scala.InstrumentedBuilder {
  override lazy val metricBaseName = MetricName("")

  val metricRegistry = MetricRegistryHolder.registry
}

val reporter = InfluxdbReporter.forRegistry(MetricRegistryHolder.registry)
        .protocol(InfluxdbProtocols.http("localhost", 8086, "metrics"))
      .convertRatesTo(TimeUnit.SECONDS)
      .convertDurationsTo(TimeUnit.MILLISECONDS)
      .filter(MetricFilter.ALL)
      .transformer(new CategoriesMetricMeasurementTransformer("module", "artifact"))
      .build()
    reporter.start(1, TimeUnit.MILLISECONDS)

  val testTimer = metrics.timer("test")
    testTimer.update(1, TimeUnit.SECONDS)

but whats strange on update I have about 50 records.. its so strange. Is it ok ?
and btw I have to use scala-influxdb-client just to connect to infliuxDB is it ok ?
Does new version supports v0.12 ?

Connected to http://localhost:8086 version 0.9
InfluxDB shell 0.12.0

Library does not seem to respect metric prefix configuration

ScheduledReporter reporter = InfluxdbReporter.forRegistry(metricRegistry)
                .protocol(new HttpInfluxdbProtocol(properties.getScheme(), properties.getHost(), properties.getPort(), "root", "root", properties.getDb()))
                .prefixedWith("test-prefix")
                .build();

Using the configuration above, I have observed that the prefix is not respected when pushing the metrics to influxdb. This seems to be the case for both v08 and v09.

Some other package name?

possible it is good idea to change package name to something classical like "org.bla-bla.metrics.influxdb"

Allow the ScheduledExecutorService used by InfluxdbReporter to be customized

InfluxdbReporter's builder has a lot of customization options, but it doesn't appear possible to set the ScheduledExecutorService that gets used under the hood. I need the ability to provide my own executor here since my application requires some custom logic to be run whenever threads are instantiated.

Release 0.8.3?

Is there a timeline for this? There are several commits since 0.8.2 that I'd like to be able to use.

Thanks!

Count values for Timers should use Timer.getCount

Reporting the count values for Timers currently use the value obtained from snapshot.size().
This should be changed to use the getCount() method on the Timer, which accurately returns the number of times the event has occurred.

Support for DNS name entry changes in UdpInlineSender

If the desire is for this client to be able to work with services that dynamically change their DNS name entries (many in AWS, GCP, etc), we need to update UdpInlineSender.

Inn UdpInlineSender, recreating the InetSocketAddress passed in to channel.send would allow for dynamic DNS resolution.

In the HTTP sender, I believe we are in good shape. The URL class's DNS lookup functionality (as strange as this is) is only used for equals and hashcode...hence opening each new connection as the code in this library does should result in fresh DNS lookups against JVM cache.

Rewrite on official java API?

Seems to be good idea to move from manual json creation to java API library https://github.com/influxdb/influxdb-java

naming of metric values is not consistent with other Reporters

If we look at some standard metrics reporters (CSV, graphite, ganglia, ...), the naming of values is different than the one in metrics-influxdb

For example histogram values in standard reporters: count,max,mean,min,stddev,p50,p75,p95,p98,p99,p999
in InfluxDBReporter
run-count,max,mean,min,std-dev,50-percentile,75-percentile,95-percentile,99-percentile,999-percentile.

It is even worst for count as in InfluxDBReporter it is the number of samples in the Snapshot instead of the metric calls.

I would propose to correct this by default and to introduce a backward compatible setting via the builder.

Allow not sending some parts of the metrics

Hi, I would like to send only some parts of the metrics.

for example, for timers I don't want to send the one_minute, five_minute and fifteen_minute mean_rate and run_count. Those values does not matter to me because I'm using HdrHistogramResetOnSnapshotReservoir (see https://bitbucket.org/marshallpierce/hdrhistogram-metrics-reservoir and http://taint.org/2014/01/16/145944a.html) which resets everything every time a snapshot is taken, So the sample size is the rate since last report.

If you'd like I can do a pull request by my own but please guide me where do you want me to make the change.

Thanks.

JsonBuilderDefault does not handle Double.NaN or Float.NaN

Those are reported as NaN and that' s not valid json. Serializing Nan as "NaN" works with the influx parser. I looked a bit at https://jira.codehaus.org/secure/attachment/43247/JACKSON-139.patch and it seem to handle it in the same way. I'll give you a PR soon.

Library generates huge amount of requests if publishing rejected metrics

The error handling code in HttpInlinerSender/QueueableSender can lead to a huge amount of writes being pushed to influx in certain situations.

More specifically what happens:
Assuming our service is producing a metric which is rejected (in our case we were trying to write an int metric to a metric with type float), then every time HttpInlinerSender::doSend is called, the entire batch will be rejected. The issue arises once the measures queue is full. At that point every time QueueableSender.send is called, it will try to send the entire batch (5000 items). Assuming this batch continually gets rejected, that means for each reporting interval, for each metric, the library will try and send the entire batch.

Some back of the envelope estimates - assuming a reporting interval of 10 seconds, with 10 metrics, we would expect to see (5000 measures * 10 metrics)/10 seconds = 5000 measurements/second written to influx. This is a huge amount of writes to push to influx. Even if your influx cluster can handle that load OK, it still introduces quite a bit of resource overhead for the service doing the reporting.

So there are two distinct issues here:

• The metrics that were failed to be written actually connected to Influx correctly, but a 400 was returned. However, on a non 200/204 return code HttpInlinerSender.java:81 tries to log the following:
  LOGGER.info("failed to send {} Measures to {}://{}:{}, HTTP CODE received: {}\n", measures.size(), writeURL.getProtocol(), writeURL.getHost(), writeURL.getPort(), responseCode, Miscellaneous.readFrom(con.getInputStream()));
  In our case the con.getInputStream() was failing (presumably the connection was closed on the server's end). Which then went to the catch block which caused the metrics to never be cleared. This issue is likely most easily fixed by just removing the Miscellaneous.readFrom(con.getInputStream()) (especially since it seems to not be logged). I will submit a pull request for this once we test it out internally.

• I am less sure whether this is actually an issue - the catch block causes the metric back to never be cleared. If there is another scenario (outside of the one described in the last bullet) that would cause the requests to be sent to influx but also cause the library to catch an IOException we would expect to see the same behavior. If that's possible it is probably worth adding something to handle expiring failed measurements after a number of failed sends.

Add support for influxdb v0.9

Influxdb's new version has a different API http://influxdb.com/docs/v0.9/concepts/reading_and_writing_data.html

HttpDatabaseCreator must use POST method instead GET

Hello.
This manual https://docs.influxdata.com/influxdb/v1.7/tools/api/#verb-usage pointing us that we must use POST instead GET method. It doesn't create new DB when using GET method.

NaN float values breaks json

In case of any NaN value in metrics json broken (need to be skipped probably)

BoundedFIFO.add to delegate (LinkedBlockingQueue) throws exception

in BoundedFIFO.add() function, one check if add succeeded in a while loop.
if insert succeeded this.delegate.add(e) will return true, but in case it won't it will throws IllegalStateException: "Queue full".
add function on LinkedBlockingQueue will always return true or throw an exception (never return false)

Support different precisions

Hi,

Seems that influx-db compresses data better when using second as precision :

influxdata/telegraf#159

Collection in points breaks reporting

Some metrics contains collections in points (for example "jvm.thread-states.deadlocks"). It is reported as [] and breaks influxdb import.

Timestamp issue with InfluxDB 0.9

Hey, thanks for the library.

The documention of InfluxDB states a timestamp:

The timestamp - supplied at the end of the line in Unix time in nanoseconds 
since January 1, 1970 UTC - is optional. If you do not specify a timestamp InfluxDB 
uses the server’s local nanosecond timestamp in Unix epoch. Anything that has to 
do with time in InfluxDB is always UTC.

However MeasurementReporter is using getTime which is equals to System.currentTimeMillis. Am I missing something?

Project unable to open in IntelliJ IDEA

I am unable to import this project into my idea using gradle. I've tried using the wrapper and local installation for importing but both result in an error that prevents the project from importing.

I don't have any issue importing other gradle projects.

Error Log: http://pastebin.com/r8fvJcXC

Could Support JDK1.7?

Hi:
The project use jdk8, but our project use jdk1.7. Could you release the bytecode support jdk7?
Thank you!

TFAR on Server

After joining the Server: "no Task Force Radio deployed on Server"
I have no SW Radio on my Server, but LR works normal.

Can anybody help me pleas.

best regards

Andi

Unable to report to InfluxDB, forgot data.

I'm trying to use this library, but I always get this exception:

2014-05-22 17:56:22,626 WARN  [Thread-1] metrics_influxdb.InfluxdbReporter (()) - Unable to report to InfluxDB, forgot data
java.io.IOException: Server returned HTTP response code: 400for URL: http://localhost:8086/db/bfrs/series?u=bfrs&p=bfrs&time_precision=s with content :'Bad Request'
    at metrics_influxdb.Influxdb.sendRequest(Influxdb.java:151)
    at metrics_influxdb.InfluxdbReporter.report(InfluxdbReporter.java:274)
    at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:116)
...

I've set up my reporter like this:

...
    final Influxdb influxDb = new Influxdb("localhost", 8086, "blah", "blah", "blah", TimeUnit.SECONDS);
    dbReporter = InfluxdbReporter.forRegistry(metrics).convertRatesTo(TimeUnit.MILLISECONDS).convertDurationsTo(TimeUnit.SECONDS).filter(MetricFilter.ALL).build(influxDb);
    dbReporter.start(500, TimeUnit.MILLISECONDS);

after I've definitely collected some metrics (during a unit test, and I can see them if I use just a slf4jreporter), at the end of the tests, I do:

      dbReporter.stop();
      dbReporter.report();

and that's when I get the exception. I even ran the code in debug mode, and I see there are metrics and the:

    influxdb.appendSeries(prefix, name, ".meter", COLUMNS_METER, pointsMeter);

line gets called with data to append.

Any ideas what I might be doing wrong?

Thanks.

Support for 0.12.

I get couldn't sent metrics to http://[my servername omitted]:8086, reason: Server returned HTTP response code: 500 for URL: http://[my servername omitted]:8086/write?precision=ms&db=service
when I am trying to use the library (version 0.8.2) against influxdb 0.12. I hope this is not a local issue for our setup. The server seems to work as telegraf is reporting its metrics correctly.

Unfurtunatly, I have no time to investigate this further right now...

btw @davidB thanks for your efforts

Auto create db

Hi,

Could you please confirm whether the reporter will auto create the influxdb db if it doesn't exist?

Regards
Brian

Jitpack.io is not reliable publisher

I have experienced several times issues with jitpack.io site when resolving snapshot version.
https://jitpack.io/com/github/davidb/metrics-influxdb/-SNAPSHOT/maven-metadata.xml gives:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<metadata modelVersion="1.0.0">
    <groupId>com.github.davidb</groupId>
    <artifactId>metrics-influxdb</artifactId>
    <version>-0.8.2-g570144b-13</version>
    <versioning>
        <snapshot>
            <timestamp>0.8.2-g570144b</timestamp>
            <buildNumber>13</buildNumber>
        </snapshot>
    </versioning>
</metadata>

while url:
https://jitpack.io/com/github/davidb/metrics-influxdb/-SNAPSHOT/metrics-influxdb--0.8.2-g570144b-13.pom returns 404.
It is not the first time when I can not build a project because this dependency resolution fails.
Would you consider publishing at more reliable location please?