camitz / aws-cloudwatch-statsd-backend

Backend for use with statsd (statistics aggregator for Node.js) targeting AWS CloudWatch.

StatsD backend for AWS CloudWatch

Overview

StatsD is a smart Node.js package that collects and aggregates statistics from differents apps sent over the UDP protocol. At a set time interval it forwards the aggregated data to a configured backend. It is pluggable with several backends available, the most popular being Graphite, a python/django monitoring tool.

With aws-cloudwatch-statsd-backend you can replace Graphite in favour of AWS Cloudwatch for your monitoring purposes, appropriate for sites on the Amazon EC2 cloud.

Counters, timers, gauges and sets are all supported.

Installation

You need node.js installed on your system aswell as StatsD. Follow the instructions on their sites or see this blog post/tutorial on how to install these components on a Windows system.

The CloudWatch backend is an npm package that can be installed with the npm command which comes with your installation of node.js. Go to the npm site for more information.

npm install aws-cloudwatch-statsd-backend

The package has two depdencies that should be installed automatically, awssum and fmt. Awssum is a node.js package encapsulating the AWS API.

Configuration

The StatsD and its backends are configured in a json object placed in a file supplied to StatsD at the command line. For example, start StatsD with the following.

node ./stats.js ./myConfig.js

The following demonstrates the minimum config for the CloudWatch backend.

{
    backends: [ "aws-cloudwatch-statsd-backend" ],
    cloudwatch: 
    {
        accessKeyId: 'YOUR_ACCESS_KEY_ID', 
        secretAccessKey:'YOUR_SECRET_ACCESS_KEY', 
        region:"YOUR_REGION"
    }
}

The access keys can be you personal credentials to AWS but it is highly recommended to create an ad hoc user via Amazon's IAM service and use those credentials.

The region is for example EU_WEST_1 or US_EAST_1. Region should be in capital letter and separated by _ instead of -.

The above will create a metric with the default namespace, AwsCloudWatchStatsdBackend, and send an http request to CloudWatch via awssum.

See the CloudWatch documentation for more information on these concepts.

The metric name, unit and value depends on what you send StatsD with your UDP request. For example, given

gorets:1|c

the Unit will be Counter, the metric name gorets. The value will be the aggregated count as calculated by StatsD.

ms corresponds the unit Milliseconds. *s and g to None.

Warning Indescriminate use of CloudWatch metrics can quickly become costly. Amazon charges 50 cents for each combination of namepace, metric name and dimension per month. However, the 10 first per month are free.

Additional configuration options

The cloudwatch backend provides ways to override the name and namespace by cofiguration. It can also capture these components from the bucket name.

The following overrides the default and any provided namespace or metric name with the specified.

{
    backends: [ "aws-cloudwatch-statsd-backend" ],
    cloudwatch: 
    {
        accessKeyId: 'YOUR_ACCESS_KEY_ID', 
        secretAccessKey: 'YOUR_SECRET_ACCESS_KEY', 
        region: 'YOUR_REGION',
        namespace: 'App/Controller/Action', 
        metricName: 'Request'
    }
}

Using the option processKeyForNamespace (default is false) you can parse the bucket name for namespace in addition to metric name. The backend will use the last component of a bucket name comprised of slash (/), dot (.) or dash (-) separated parts as the metric name. The remaining leading parts will be used as namespace. Separators will be replaced with slashes (/).

{
    backends: [ "aws-cloudwatch-statsd-backend" ],
    cloudwatch: 
    {
        accessKeyId: 'YOUR_ACCESS_KEY_ID', 
        secretAccessKey: 'YOUR_SECRET_ACCESS_KEY', 
        region: 'YOUR_REGION',
        processKeyForNames:true
    }
}

For example, sending StatsD the following

App.Controller.Action.Request:1|c

is will produce the equivalent to the former configuration example. Note that both will be suppressed if overriden as in the former configuration example.

Whitelisting Metrics

Using cloudwatch will incur a cost for each metric sent. In order to control your costs, you can optionally whitelist (by full metric name) those metrics sent to cloudwatch. For example:

{
    backends: [ "aws-cloudwatch-statsd-backend" ],
    cloudwatch: 
    {
        accessKeyId: 'YOUR_ACCESS_KEY_ID', 
        secretAccessKey: 'YOUR_SECRET_ACCESS_KEY', 
        region: 'YOUR_REGION',
        whitelist: ['YOUR_FULL_METRIC_NAME']
    }
}

The above configuration would only sent the metric named 'YOUR_FULL_METRIC_NAME' to cloudwatch. As this is an array, you can specify multiple metrics. This is useful if you are using multiple backends e.g. mysql backend and want to send some metrics cloudwatch (due to the associated cost) and all the metrics together to another backend. It is also useful if you want to limit the metrics you use in cloudwatch to those that raise alarms as part of your wider AWS hosted system.

Using AWS Roles to obtain credentials

A preferable approach to obtaining account credentials is instead to query the Metadata Service to obtain IAM security credentials for a given role. If iamRole is set to 'any' then any available credentials found on the metadata service will instead be used. For example:

{
    backends: [ "aws-cloudwatch-statsd-backend" ],
    cloudwatch:
    {
        iamRole: 'YOUR_ROLE_NAME',
        region: 'YOUR_REGION',
        whitelist: ['YOUR_FULL_METRIC_NAME']
    }
}

Multi-region support

If you wish to send cloudwatch metrics to multiple regions at once, instead of

{
    backends: [ "aws-cloudwatch-statsd-backend" ],
    cloudwatch: 
    {
        accessKeyId: 'YOUR_ACCESS_KEY_ID', 
        secretAccessKey:'YOUR_SECRET_ACCESS_KEY', 
        region:"YOUR_REGION"
    }
}

you can use the instances key under cloudwatch to configure a list of configurations.

{
    backends: ["aws-cloudwatch-statsd-backend"],
    cloudwatch: {
        instances: [{
            accessKeyId: 'YOUR_ACCESS_KEY_ID',
            secretAccessKey: 'YOUR_SECRET_ACCESS_KEY',
            region: "YOUR_REGION_1",
            whitelist: ['YOUR_FULL_METRIC_NAME1']
        }, {
            accessKeyId: 'YOUR_ACCESS_KEY_ID',
            secretAccessKey: 'YOUR_SECRET_ACCESS_KEY',
            region: "YOUR_REGION_2",
            whitelist: ['YOUR_FULL_METRIC_NAME2']
        }]
    }
}

Tutorial

This project was launched with a following blog post/tutorial describing the implementation chain from log4net to Cloudwatch on a Windows system.

Also in the series:

Improving the CloudWatch Appender

A CloudWatch Appender for log4net

aws-cloudwatch-statsd-backend's Issues

Support IAM Roles for authentication

Currently it is required to specify accessKeyId and secretAccessKey. It would be great if this backend could utilize IAM roles for authentication when running on EC2.

For more info see: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html

invalid security token

Hi, I am following your docs for running with the most basic configuration described in your readme but am unable to send metrics to cloudwatch:

Err : { StatusCode: 403,
  Headers:
   { 'x-amzn-requestid': '-',
     'content-type': 'text/xml',
     'content-length': '311',
     date: 'Tue, 03 Jan 2017 19:03:25 GMT' },
  Body:
   { ErrorResponse:
      { '@': { xmlns: 'http://monitoring.amazonaws.com/doc/2010-08-01/' },
        Error:
         { Type: 'Sender',
           Code: 'InvalidClientTokenId',
           Message: 'The security token included in the request is invalid' },
        RequestId: '-' } } }

I am able to get statsD metrics publishing to the console backend and have verified that my access key/secret key are correct by describing cloudwatch using AWS cli:

aws cloudwatch describe-alarms

(works)

Update documentation to cover latest version

Although this repository has been updated to use the aws-sdk node package rather than awssum, the documentation is still tuned to awssum. This led me to have an error when I provided the region as "US_EAST_1" rather than "us-east-1".

Please update the documentation prior to making a more recent release.

Use putMetricData optimally.

putMetricData() method allows to pack up to 40 KB of metrics. In case of producing huge number of them packing metrics in single requests could give 10-20x costs reduction.

Links to external information are dead

Please update these if you think they are helpful :)

1.3.0 is not in npm

as above

Optimize putMetrics - to avoid Cloudwatch throttling

This might be related to #7

We see this in statsd.log - perhaps packing metrics might help?

Err : { StatusCode: 400,
  Headers:
   { 'x-amzn-requestid': '43732e6e-4618-11e6-9498-bf6b383bb542',
     'content-type': 'text/xml',
     'content-length': '261',
     date: 'Sat, 09 Jul 2016 21:01:10 GMT' },
  Body:
   { ErrorResponse:
      { '@': { xmlns: 'http://monitoring.amazonaws.com/doc/2010-08-01/' },
        Error:
         { Type: 'Sender',
           Code: 'Throttling',
           Message: 'Rate exceeded' },
        RequestId: '43732e6e-4618-11e6-9498-bf6b383bb542' } } }

Bumping up default statsd flushInterval to 20, 30 s does not seem to help

Documentation Error: Region option is not correctly specified

The region option has been changed inside code or something. When I checked the code the region option they were expecting was in the format AP_SOUTH_1, AP_SOUTHEAST_1 instead of ap-south-1, ap-southeast-1. And inside the doc its given as latter but in code they are expecting former.