staslev / storm-metrics-reporter

A Storm metrics reporter module.

storm-metrics-reporter

A Storm metrics reporter capable of sending both Storm built-in metrics, as well as custom ones, to various metrics reporters such as Graphite or plain JMX.

Please take a look at the Wiki for further details.

Usage

Add a pom dependency (hosted by maven central):

<dependency>
  <groupId>com.github.staslev</groupId>
  <artifactId>storm-metrics-reporter</artifactId>
  <version>1.5.0</version>
</dependency>

Or, in case you wish to build the jar yourself:

git clone https://github.com/staslev/storm-metrics-reporter.git
cd storm-metrics-reporter
mvn install

Screenshots

Disclaimers

• This project was inspired by storm-metrics-statsd, thanks jt6211!
• The screenshots above were generated using a custom StormMetricProcessor implementation, not the one included in the sources. It depicts a particular Graphite naming convention (the host for instance, is not seen in the metric hierarchy chart as it's located higher in the hierarchy). Metrics naming styles are subject to change in other environemnts.
• storm-metrics-reporter currently supports reporting metrics to Graphite only, but it should be pretty straight forward to extend it to support other metrics reporting mechanisms.

Further Reading

• storm-metrics-reporter wiki
• Sending out Storm metrics
• Storm Metrics How-To
• Sending metrics from storm to graphite
• Storm documentation

storm-metrics-reporter's Issues

release for JMX Reporter

Hi,

the JMX reporter was added a while back, but there hasn't been a release since.

Could you please create one, so we don't have to use local jars?

Thanks,
Cheers!

When I use a customized scheduler Storm does not collect metrics

Hi, when I use a customized scheduler which implements IScheduler interface I cannot see any metrics on Graphite anymore. I tested without a customized scheduler and it works. However it stops to work when I just configure a customized scheduler on the storm.yaml

supervisor.scheduler.meta:
  site: "cluster"
storm.scheduler: "org.sense.storm.scheduler.SiteAwareScheduler"

Do you have any hint of what could be happening?
Thanks

Can't see Storm in my graphite UI

Hi,

I like to report Storm's built-in metrics using the storm-metrics-reporter, here how I make it, first

1. add dependency in my topology pom
   
   com.github.staslev
   storm-metrics-reporter
   1.0
   
2. add this snippet my code:

import com.github.staslev.storm.metrics.graphite.SimpleGraphiteStormMetricGauge;
import com.github.staslev.storm.metrics.MetricReporterConfig;
import com.github.staslev.storm.metrics.MetricReporter;

conf.put(MetricReporter.METRICS_HOST, "10.100.70.29");
conf.put(MetricReporter.METRICS_PORT, "2003");
conf.registerMetricsConsumer(MetricReporter.class,
new MetricReporterConfig("(.execute.|.latency.|.capacity.)",
SimpleGraphiteStormMetricGauge.class.getCanonicalName()),
1);
I successfully build the jar and submit the topology.

1. Now I assume my topology has registered the storm-metrics-reporter and sending buildin metrics to graphite.
2. This is how I start graphite-web
   PYTHONPATH=pwd/storage/whisper ./bin/run-graphite-devel-server.py --port=8085 --libs=pwd/webapp /opt/graphite 1>/opt/graphite/storage/log/webapp/process.log 2>&1 &
3. carbon.conf
   [cache]
   LINE_RECEIVER_INTERFACE = 0.0.0.0
   LINE_RECEIVER_PORT = 2003
   CACHE_QUERY_INTERFACE = 0.0.0.0
   CACHE_QUERY_PORT = 7002

However, I can't see storm metrics in graphite, any ideas?

thanks

AL