Coder Social home page Coder Social logo

yaorg / node-measured Goto Github PK

View Code? Open in Web Editor NEW
518.0 12.0 53.0 4.45 MB

A Node metrics library for measuring and reporting application-level metrics, inspired by Coda Hale, Yammer Inc's Dropwizard Metrics Libraries

Home Page: https://yaorg.github.io/node-measured/

License: MIT License

JavaScript 97.05% Shell 2.95%
nodejs node metrics signalfx datadog dropwizard meter gauge counter timer

node-measured's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

node-measured's Issues

Create Type File for Typescript compatibility

I think it would be cool to have an official type file either included with this project or available via

npn install @types/measured

I have made the following in my project which is probably almost everything that is needed

/**
 * Everything in this file is sourced from the [measured readme]{@link https://github.com/felixge/node-measured}
 */
declare module 'measured' {
    /**
     * Values that can be read instantly.
     *
     * Gauges take a function as parameter which needs to return their current value.
     */
    export class Gauge {
        constructor(callBack: () => number)

        /**
         * Gauges directly return their currently value.
         */
        public toJSON(): JSON
    }

    /**
     * Things that increment or decrement.
     */
    export class Counter implements IMetric {
        /**
         * @param {CounterProperties} properties
         */
        constructor(properties?: CounterProperties)

        /**
         * Increment the counter by n. Defaults to 1.
         * @param {number} n
         */
        public inc(n?: number): void

        /**
         * Decrement the counter by n. Defaults to 1.
         * @param {number} n
         */
        public dec(n?: number): void

        /**
         * Resets the counter back to count Defaults to 0.
         * @param {number} count
         */
        public reset(count?: number): void

        /**
         * Counters directly return their currently value.
         */
        public toJSON(): JSON
    }

    /**
     * Properties to create a [Counter]{@see Counter} with.
     */
    export class CounterProperties {
        /**
         * count An initial count for the counter. Defaults to 0.
         */
        public count: number
    }

    /**
     * Things that are measured as events / interval. Example:
     */
    export class Meter implements IMetric {
        /**
         * rateUnit The rate unit. Defaults to 1000 (1 sec).
         * tickInterval The interval in which the averages are updated. Defaults to 5000 (5 sec).
         * @param {MeterProperties} properties
         */
        constructor(properties?: MeterProperties)

        /**
         * Register n events as having just occured. Defaults to 1.
         * @param {number} n
         */
        public mark(n: number)

        /**
         * Resets all values. Meters initialized with custom options will be reset to the default settings (patch welcome).
         */
        public reset(): void

        /**
         * Unrefs the backing timer. The meter will not keep the event loop alive. Idempotent.
         */
        public unref(): void

        /**
         * Refs the backing timer again. Idempotent.
         */
        public ref(): void

        /**
         * toJSON Output
         *
         * <li> mean: The average rate since the meter was started.
         * <li> count: The total of all values added to the meter.
         * <li> currentRate: The rate of the meter since the last toJSON() call.
         * <li> 1MinuteRate: The rate of the meter biased towards the last 1 minute.
         * <li> 5MinuteRate: The rate of the meter biased towards the last 5 minutes.
         * <li> 15MinuteRate: The rate of the meter biased towards the last 15 minutes.
         *
         * @return {JSON}
         */
        public toJSON(): JSON
    }

    /**
     * Properties to create a [Meter]{@see Meter} with.
     */
    export class MeterProperties {
        /**
         * The rate unit. Defaults to 1000 (1 sec).
         */
        public rateUnit: number

        /**
         * The interval in which the averages are updated. Defaults to 5000 (5 sec).
         */
        public tickInterval: number
    }

    /**
     * Keeps a resevoir of statistically relevant values biased towards the last 5 minutes to explore their distribution.
     *
     * {@link https://github.com/felixge/node-measured#histogram}
     */
    export class Histogram implements IMetric {
        /**
         * @param {HistogramProperties} properties
         */
        constructor(properties?: HistogramProperties)

        /**
         * Pushes value into the sample. timestamp defaults to Date.now().
         * @param {number} value
         * @param {Date} timestamp
         */
        public update(value: number, timestamp?: Date): void

        /**
         * Whether the histogram contains values.
         * @return {boolean}
         */
        public hasValues(): boolean

        /**
         * Resets all values. Histograms initialized with custom options will be reset to the default settings (patch welcome).
         */
        public reset(): void

        /**
         * toJSON output:
         *
         * <li> min: The lowest observed value.
         * <li> max: The highest observed value.
         * <li> sum: The sum of all observed values.
         * <li> variance: The variance of all observed values.
         * <li> mean: The average of all observed values.
         * <li> stddev: The stddev of all observed values.
         * <li> count: The number of observed values.
         * <li> median: 50% of all values in the resevoir are at or below this value.
         * <li> p75: See median, 75% percentile.
         * <li> p95: See median, 95% percentile.
         * <li> p99: See median, 99% percentile.
         * <li> p999: See median, 99.9% percentile.
         *
         * @return {JSON}
         */
        public toJSON(): JSON
    }

    /**
     * Properties to create a [Histogram]{@see Histogram} with.
     */
    export class HistogramProperties {
        /**
         * The sample resevoir to use. Defaults to an ExponentiallyDecayingSample.
         */
        public sample: object
    }

    /**
     * Timers are a combination of Meters and Histograms. They measure the rate as well as distribution of scalar events.
     * Since they are frequently used for tracking how long certain things take, they expose an API for that:
     *
     * {@see https://github.com/felixge/node-measured#timers}
     */
    export class Timer implements IMetric {
        /**
         * @param {TimerProperties} properties
         */
        constructor(properties?: TimerProperties)

        /**
         * @return {StopWatch} Returns a Stopwatch that has been started.
         */
        public start(): StopWatch

        /**
         *  Updates the internal histogram with value and marks one event on the internal meter.
         * @param {number} value
         */
        public update(value: number)

        /**
         * Resets all values. Timers initialized with custom options will be reset to the default settings.
         */
        public reset(): void

        /**
         * Unrefs the backing timer. The meter will not keep the event loop alive. Idempotent.
         */
        public unref(): void

        /**
         * Refs the backing timer again. Idempotent.
         */
        public ref(): void

        /**
         * toJSON output:
         *
         * <li> meter: {@see Meter} toJSON output docs above.
         * <li> histogram: {@see Histogram} toJSON output docs above.
         *
         * @return {JSON}
         */
        public toJSON(): JSON
    }

    /**
     * Properties to create a [Timer]{@see Timer} with.
     */
    export class TimerProperties {
        /**
         * The internal meter to use. Defaults to a new Meter.
         * {@see Meter}
         */
        public meter: Meter

        /**
         * The internal histogram to use. Defaults to a new Histogram.
         * {@see Histogram}
         */
        public histogram: Histogram
    }

    /**
     * Created by the Timer Metric when start() is called
     */
    export class StopWatch {
        /**
         * Called to mark the end of the timer task
         * @return {number} the total execution time
         */
        public end(): number
    }

    /**
     * Creates a collection that can create and keep track of metrics
     */
    export const createCollection: (name?: string) => Collection

    /**
     * Collection class that keeps track of Metrics that it has created
     *
     * @param {string} name Optional name to use for the collection
     */
    export class Collection {
        constructor(name: string)

        /**
         * {@see https://github.com/felixge/node-measured/blob/master/lib/Collection.js}
         * @return {JSON} JSON object with all the metrics in the collection
         */
        public toJSON(): JSON

        /**
         * Iterates through the metrics in the collection and
         * calls end() in the metric, if the metric has an end() method.
         */
        public end(): void

        /**
         * Creates a new Gauge and registers it with the metrics collection
         *
         * {@see Gauge}
         *
         * @param {() => number} callBack
         * @return {Gauge}
         */
        public guage(callBack: () => number): Gauge

        /**
         * Creates a new Counter and registers it with the metrics collection
         *
         * {@see Counter}
         *
         * @param {CounterProperties} properties optional counter properties.
         * @return {Counter}
         */
        public counter(properties?: CounterProperties): Counter

        /**
         * Creates a new Meter and registers it with the metrics collection
         *
         * {@see Meter}
         *
         * @param {MeterProperties} properties
         * @return {Meter}
         */
        public meter(properties?: MeterProperties): Meter

        /**
         * Creates a new Histogram and registers it with the metrics collection
         *
         * {@see Histogram}
         *
         * @param {HistogramProperties} properties
         * @return {Histogram}
         */
        public histogram(properties?: HistogramProperties): Histogram

        /**
         * Creates a new Timer and registers it with the metrics collection
         *
         * {@see Timer}
         *
         * @param {TimerProperties} properties
         * @return {Timer}
         */
        public timer(properties?: TimerProperties): Timer
    }

    /**
     * All metrics implement toJSON()
     */
    export interface IMetric {
        /**
         * See the toJSON metric of the implementing classes
         * @return {JSON}
         */
        toJSON(): JSON
    }
}

I think a modified version of the above could be included with index.js

Checkout what AWS is doing with their Node SDK
https://github.com/aws/aws-sdk-js

If this is something you are open to I could make a PR.
I think this would benefit all users regardless if they use typescript, as it will add java docs and return types to their IDEs

"_unknown" uri passed for POSTs to signalfx

https://github.com/yaorg/node-measured/blob/master/packages/measured-node-metrics/lib/nodeHttpRequestMetrics.js#L32
req.route is always undefined here when I make a POST, so uri is sent as "_unknown". Docs don't mention this. Would be nice if they did, but a working middleware would be even better :)

Minimal reproducible example:

  1. start with https://yaorg.github.io/node-measured/packages/measured-signalfx-reporter/tutorial-SignalFx%20Express%20Full%20End%20to%20End%20Example.html
  2. s/app.get/app.post/g
  3. make a post call to '/hello'
    4a) observe uri dimension is not passed to SFX
    4b) observe status code is passed as 200 even if you change it to return something else

Fix:
req.url seems to be available, but since you're trying to avoid that, I would suggest setting a property on req, possibly triggered by an event like 'data' or 'end'

Speaking of which, it might be helpful to mention that the middleware needs to be added before any bodyParser middleware as 'data' and 'end' get swallowed during POSTs.

Histograms - possible to show windowed averages

I have a use case where I would like to show the average over different timeframes, rather than simply for all time. Does that seem like something worth adding into the histogram output?

Support for Value and Rate in Histogram

I have a requirement where I would like to know the last value recorded for a metric and rate of a particular metric

To get the Latest value I can use a gauge but it is synchronous and requires developers to maintain the values they require to measure

Rate of a metric is not possible right now. If I want to understand rate of a metric I have to use counters and calculate rate of change manually.

Let me know if the 2 requirements make sense and I can raise a Pull request for the same.

Why isn't DimensionAwareMetricsRegistry a subclass of Collection?

I notice there's a lot of duplicated logic between DimensionAwareMetricsRegistry in measured-reporting and Collection in measured-core. Is there a reason DimensionAwareMetricsRegistry was not just created as a subclass of Collection?

What about SelfReportingMetricsRegistry? It currently seems to contain a DimensionAwareMetricsRegistry as a field by default, allowing a custom registry to be given, but if a custom registry is given it needs to conform to the same API including the dimension-specific parts. The dimensions concern also seems to leak out into SelfReportingMetricsRegistry so it seems to me like maybe those should just be merged or subclassed?

What do you think? I can put together some PRs to clean that up. I just want to know what your perspective is on it first.

toJSON() is named incorrectly - doesn't return JSON

It's really bugging me that this method is called toJSON but it's not actually returning JSON - it returns an Object - there's a big difference.

I don't want it to return JSON, I just mean to say that the method name is wrong.

Usage example in README doesn't make sense

Your code in step 2 refers to a collection object which was not defined in step 1.

The usage example differs significantly from the example code in example/http_requests_per_second.js

High resolution timer in Stopwatch

It would be a quite nice feature to use process.hrtime() to have a even more precise time.

Than it would be:

{  
  histogram:
  { min: 0.45434,
  ...
  }
}

instead of:

{  
  histogram:
  { min: 0,
  ...
  }
}

Unfortunately sinon does not support the mocking of process.hrtime, so all tests would fail...

How to use this with Graphite?

The README has a TODO about instructions for using this module with Graphite. Can someone tell me roughly how I should do this? I could then figure out the details and write nicer instructions. Thanks! :)

docs: document release process

Hi @fieldju,

Apologies about the issues around #64. I'm not sure if I understand the current release process and somehow Travis kept adding tags so I cancelled the last build.

Probably the root cause is that I manually created a tag? Is there any documentation about how the release process works?

Histogram - quantiles weighting

Currently in the Histogram the result of priority sampling (from ExponentiallyDecayingSample) is treated like uniform sampling with discarded items weights.

The Cormode et al. Forward Decay paper defines a quantile as the smallest item v satisfying equation:

quantiles

max,min and other values are also not calculated using the formulas from the paper therefore are not biased towards recent values. Might raise a separate PR/issue for that.

See also dropwizard/metrics#421

Invert Control for Gauge

It looks like the Gauge currently polls the provided function to obtain the current represented value. I think it also makes sense to provide a set method to follow node's evented model. I might want a Gauge of number of open connections in a mysql pool, but I know when that number increases or decreases, so it would be better to just tell the Gauge when that number changes. In some cases, though, like in the case of memoryUsage, it does make sense to poll for changes, so both usages should be supported IMO.

var gauge = new metrics.Gauge(function() {
  return process.memoryUsage().rss;
});

var gauge = new metrics.Gauge();

gauge.set(pool.openConnections());

pool.on('openConn', function() {
  gauge.set(pool.openConnections());
});

pool.on('closeConn', function() {
  gauge.set(pool.openConnections());
});

// or even

var gauge = new metrics.Gauge(function() {
  return pool.openConnections();
});

pool.on('openConn', function() {
  gauge.pull();
});

pool.on('closeConn', function() {
  gauge.pull();
});

Decide whether or not this should be hard forked or add new maintainers.

There was a bit of a conversation around this in PR #31, I want to help maintain this library as I need it and would like some fresh features.

I guess I need to know whether or not this repo can have new maintainers that are interested in helping out or if this should get hard forked.

It might be nice to move this repo to an organization and invite the maintainers rather than have it live under a single devs personal space?

https://help.github.com/articles/transferring-a-repository-owned-by-your-personal-account/#transferring-to-an-organization

[RFC] 3.0?

I am thinking I want to port this lib to Typescript and add some more integrations namely Prometheus.
Maybe finally clean up somethings on the Metrics objects, such as toJson() being the primary manner of getting data when half the time it doesn't return JSON.

As I am basically the only one care and feeding this lib, I will leave this issue open for a while and see if anyone has comments and see if I have time to start this process.

Closes #32

In the process maybe we can get some integration tests up and running so that PRs like dependency auto bumps can be streamlined and we can make sure we stay up to date to prevent security issues.

I think porting to TS can be done 1 package at a time starting with core.

I am going to CC anyone who has ever contributed for visibility:

CC: @felixge, @mantoni, @Qard, @oliverzy, @csabapalfi, @dohse, @anacasner, @horte, @bnoordhuis, @tuckbick, @tlisonbee, @joeybaker, @hugebdu, @avolfson, @arlolra, @OlegIlyenko

Cannot use EWMA with Histogram

const measured = require('measured');

const metrics = measured.createCollection();

const ewmaTimer = metrics.timer('perf.(algo=ewma).getModel', {
        histogram: new measured.Histogram({
            sample: new measured.ExponentiallyMovingWeightedAverage()
        })
    }).start()


ewmaTimer.end();

ewmaTimer.toJSON(); // fall down go boom

Am I misunderstanding how this is supposed to be used?

Get max value of a counter - Feature request (or potentially stupid user)

Analogy to keep things simple..

I use counters && counter.inc/dec to have stats how many active connections we have.

I need to know what the maximum connections have been since application start. I wonder if this is something that node-measured provides and I'm just missing it. Obivously I can write logic on inc event => if value is > value in variable => overwrite variable value with newValue but I want to keep things clean and inside measured if I can...

Sorry again if I'm missing something obvious here..

Unbounded memory usage

I'm noticing issues where Node processes consuming histograms (and surprisingly) meters are growing memory in a linear, unbounded fashion. Removing measured fixes this issue. I'm not very surprised that histograms are memory-hungry given they're backed by a binary heap, I am much more surprised the meters seem to be as well.

Is this something you've encountered before?

bunyan dependency on dtrace-provider is problematic

Apparently bunyan has an optional dependency on dtrace-provider. While optional dependencies are generally okay, it's problematic in this case because dtrace-provider is a native module and therefore requires Python to be installed on the system to even attempt the install. This is especially problematic on Windows because it does not include Python by default therefore measured fails to install. Perhaps we should switch to another logger, like pino?

It has caused no end of issues opened on the bunyan repo... https://github.com/trentm/node-bunyan/issues?q=is%3Aissue+dtrace

Re-hydration from previous state

Right now, I'm accessing the "private" members directly, and assigning them manually. would be good to have an static method, like fromJSON(theOneReturnedFromtoJSON) to initialize the classes using a previous state.

Counter not working

HI, I installed measured using this:

npm install measured@latest
[email protected] node_modules/measured
└── [email protected]

Then calling node console (0.12.1):

> var m = require ('measured')
undefined
> var counter = new m.Counter()
undefined
> counter
{ _count: 0 }
> util.inspect (counter)
'{ _count: 0 }'

The functions inc(), dec(), toJSON() are missing. I looked quickly into the source code in Github - but could not see where the problem is. Any idea?

Object.values is not a function on Nodejs 6.9.0

According to its package.json, measured-core 1.11.2 should support Node >= 5.12.

When testing Etherpad 1.7 against Node 6.9.0 (ether/etherpad-lite#3459), I found that measured-core uses Object.values(), a feature that was introduced in Node 7.x.

const metricTypeValues = Object.values(MetricTypes);

For Etherpad we resolved using a polyfill (ether/etherpad-lite@fc14f60).

For next node-measured version, you may want to update package.json (requiring node >= 7.x). Or - if you want to continue supporting Node 5 - either use a polyfill or change the code to not use Object.values().

Thank you for your library!

Add clear to Histogram metric

I'm curious why there is not a clear or reset function on the histogram. If you are periodically taking a snapshot to publish to graphite, etc., wouldn't you want to reset the counts and such?

Curious, what are you doing now for transloadit?

counter.inc(0) will increment by 1

In my application I'm dynamically incrementing a counter based on the size of a collection, e.g.

var counter = new measured.Counter();
...
counter.inc(array.length);

I got wrong counts, and the issue is that if array.length is 0, the counter is still incremented by 1.

This is due to the inc and dec functions using the || logical-or operator to fill in a default value if the parameter n is not passed. However, 0 || 1 will evaluate to 1, which is probably not what was intended. It would be safer to update the inc and dec functions to explicitly test whether n is undefined (i.e. n === undefined ? 1 : n)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.