viniciusccarvalho / elasticsearch-newrelic

Plugin to push elasticsearch data to newrelic

elasticsearch-newrelic

Elasticsearch version	branch
0.90.0 -> 0.90.2	master
0.20.1 -> 0.20.5	0.20-version

This plugin export node stats (indices, pool, network, http) to a newrelic account. The plugin works at a node level, new relic will act as an agreggator for metrics of each node inside your cluster.

The plugin does not collect jvm metrics, as those are already collected by the newrelic agent by default.

The plugin has a default of collecting all metrics every 10 seconds (please note that newrelic transmit those values at a different rate, usually 60 seconds)

There's an endpoint where you can turn on/off collection per stats as well change the interval (in seconds, minimum: 1s)

Installation:

The plugin can be built using [gradle] (http://www.gradle.org/) using the jar task.

from your $ES_HOME run: bin/plugin -url file://${jar_location} -install newrelic

You will also need a [new relic] (http://www.newrelic.com) agent. Please refer to their documentation on how to get your agent along with your yml config file.

Assuming you will deploy the newrelic directory inside your ElasticSearch home folder, add this line to your bin/elasticsearch.in.sh

JAVA_OPTS="$JAVA_OPTS -javaagent:/$ES_HOME/newrelic/newrelic.jar"

Configuration:

There's an endpoint _newrelic that will return the configuration using GET:

curl -XGET http://localhost:9200/_newrelic?pretty
 {
  "configuration" : {
    "agents" : {
      "http" : true,
      "pool" : true,
      "transport" : true,
      "fs" : true,
      "indices" : true,
      "network" : true
    },
    "refreshInterval" : 10
   }
  }

You can also set each parameter individually by issuing a POST:

curl -XPOST "http://localhost:9200/_newrelic?http=false&pool=false&refreshInterval=5"

Or set all to be on/off by issuing:

curl -XPOST "http://localhost:9200/_newrelic?all=false"

Metrics:

The plugin collects six main categories: indices, pool, network, transport, http and filesystem. The metrics are created using newrelic hierarchical approach, so for example:

indices/search/total --> Total number of searches executed

This way, once you are on your custom dashboard, you can benefit from the auto complete feature by just typing the name of the main category followed by a / :

Screenshots:

elasticsearch-newrelic's Issues

Ideally post metrics in the Custom/ space

While it's not the end of the world, it's generally better style for inter-operation with New Relic to start all reported metrics with Custom/ so for example Custom/indices/search/average_time_millis for example.

The drawback here is that making this change breaks any currently set up custom dashboards within New Relic, but if you forsee your number of metrics growing at all, I would recommend making this change.

Compiled .jar

Why not profile a compiled .jar like https://github.com/polyfractal/elasticsearch-inquisitor does?

Thanks for the plugin.

Filter evictions are being collected as field evictions

Publish this plugin

It'd be great if you decided to share this plugin with the world by publishing it to Plugin Central! Simply click the "Publish" button on your plugin.

Search per second and average search times metrics are wrong

Wrong math, needs to fix it :D

After installing the plugin starting elastic search returns an error

The error I'm getting is:
Failed to find Premain-Class manifest attribute in //usr/share/elasticsearch/newrelic/newrelic.jar
Error occurred during initialization of VM
agent library failed to init: instrument

We're currently running ES v0.90.1 and the latest version of the plugin.

Any help would be appreciated.

ES 1.0.x support