fl0w / api-benchmark

A node.js tool to benchmark APIs

api-benchmark

A node.js tool that measures and compares performances of single and multiple apis inspired by BenchmarkJs.

Why all of this? (blog post)

To see an example of a request/response look at this gist.

If you want to benchmark your api via grunt take a look at grunt-api-benchmark.

1. Requirements
2. Installation
3. Usage
4. API * measure * compare * getHtml
5. The Route object
6. The Options object
7. Tuning your machine to benchmark
8. Contributing
9. Why all of this?

Requirements

Node version: min: 0.8.0, recommended: >=0.10.13

Build status: Unix: | Windows:

Installation

npm install api-benchmark

Usage

measure(service, routes, [options, ] callback)

Measures performances of a given api for multiple routes

var apiBenchmark = require('api-benchmark');

var service = {
  server1: "http://myserver:myport/mypath/"
};

var routes = { route1: 'route1', route2: 'route2' };

apiBenchmark.measure(service, routes, function(err, results){
  console.log(results);
  // displays some stats!
});

compare(services, routes, [options, ] callback)

Compares performances of a given list of api servers with the same routes. Useful in case of load balancers, globalised services, deployment of new versions.

var apiBenchmark = require('api-benchmark');

var services = {
  server1: "http://myserver:myport/mypath/",
  server2: "http://myserver2:myport2/mypath2/",
};

var routes = { route1: 'route1', route2: 'route2' };

apiBenchmark.compare(services, routes, function(err, results){
  console.log(results);
  // displays some stats, including the winner!
});

All the Http verbs and headers are supported.

var apiBenchmark = require('api-benchmark');

var services = {
  server1: "http://myserver:myport/mypath/",
  server2: "http://myserver2:myport2/mypath2/",
};

var routes = {
  route1: {
    method: 'get',
    route: 'getRoute',
    headers: {
      'Cookie': 'cookieName=value',
      'Accept': 'application/json'
    }
  },
  route2: 'getRoute2',
  route3: {
    method: 'post',
    route: 'postRoute',
    data: {
      test: true,
      moreData: 'aString'
    }
  }
};

apiBenchmark.compare(services, routes, function(err, results){
  console.log(results);
  // displays some stats, including the winner!
});

getHtml(results, callback)

Given a results object, gets the html report.

var apiBenchmark = require('api-benchmark');

var service = {
  server1: "http://myserver:myport/mypath/"
};

var routes = { route1: 'route1', route2: 'route2' };

apiBenchmark.measure(service, routes, function(err, results){
  apiBenchmark.getHtml(results, function(error, html){
    console.log(html);
    // now save it yourself to a file and enjoy
  });
});

Route object

method

(String, default 'get'): Http verb.

route

(String): the route to benchmark

headers

(Object): the headers to send. In case of function (that has to return an object) it will be evaulated for each request.

data

(Object): the data sent with the request. In case of function (that has to return an object) it will be evaulated for each request.

query

(Object): the query sent with the request. In case of function (that has to return an object) it will be evaulated for each request.

expectedStatusCode

(Number, default null): if it is a number, generates an error when the status code of the response is different

maxMean

(Number, default null): if it is a number, generates an error when the mean value for a benchmark cycle is major than the expected value

maxSingleMean

(Number, default null): if it is a number, generates an error when the mean across all the concurrent requests value is major than the expected value

Options object

debug

(Boolean, default false): Displays some info on the console.

runMode

(String, default 'sequence'): Can be 'sequence' (each request is made after receiving the previous response) or 'parallel' (all requests are made in parallel).

maxConcurrentRequests

(Number, default 100): When in runMode='parallel' it is the maximum number of concurrent requests are made.

delay

(Number, default 0): When in runMode='sequence', it is the delay between test cycles (secs).

maxTime

(Number, default 10): The maximum time a benchmark is allowed to run before finishing (secs). Note: Cycle delays aren't counted toward the maximum time.

minSamples

(Number, default 20): The minimum sample size required to perform statistical analysis.

stopOnError

(Boolean, default true): Stops the benchmark as soon as it receives an error. When false, the benchmark goes on and the errors are collected inside the callback.

Tuning

You should tune your machine to remove any OS limits in terms of opening and quickly recycling sockets.

Linux and Mac OS X

sudo sysctl -w kern.maxfiles=25000
sudo sysctl -w kern.maxfilesperproc=24500
sudo sysctl -w kern.ipc.somaxconn=20000
ulimit -S -n 20000

Docker version

Containerized version of api-benchmark is available here: docker-api-benchmark

Contributing

For the latest updates and release information follow @matteofigus on twitter. Feel free to open new Issues in case of Bugs or Feature requests. Pull requests are welcome: first run all the tests locally doing npm test.

Contributors:

• Derek Myers
• Michael Sanford

Tests

npm test

TODO

• Command-line simple interface
• Multi-thread requests
• SOAP
• Killer mode - ask for details

License

MIT