bestiejs / benchmark.js

A benchmarking library. As used on jsPerf.com.

BestieJS

A collection of "Best in Class" modules. This means we promote solid browser/environment support, ES5 precedents, unit testing, and plenty of documentation.

Contributors

John-David Dalton	Benjamin Tan	Kit Cambridge	Mathias Bynens

benchmark.js's Issues

the third test failed in IE11, windows7 64-bit

3.Benchmark: supports loading Platform.js as a module (1, 0, 1)Rerun
1.failed Expected: "string"

Result: "object"

Diff: "string" "object"

Source: at Anonymous function (http://benchmarkjs.com/_js/test.js?t=1388900753472:113:9)

Benchmarking Canvas Operations

Hey Guys,

I've been having a crack at using benchmark.js and I really like the API and the general operation of the library. One thing I'm noticing though is that when I'm trying to test things like canvas operations the browser essentially freezes up.

The same problem occurs on jsperf also. To replicate try something like:

context.clearRect(0, 0, canvas.width, canvas.height);

I'm going to keep looking to see if I can work out going on, and if I find something I'll let you know...

Cheers,
Damon.

library size?

I can't find the library size on your homepage or README - care to add it? Would be nifty if you could read the result from closure compiler and write it into the README

`setup` not consistently called before `fn`

Test case:

var Benchmark = require('benchmark');
var suite = new Benchmark.Suite('suite');
suite.add('test', {
  fn: function() {
    if (!this.wasSetup) {
      console.error('fail');
      process.exit(1);
    }
    console.log('fn');
    this.wasSetup = false;
  },
  setup: function() {
    this.wasSetup = true;
    console.log('setup');
  }
});
suite.run();

And here's the output:

% node test.js
setup
fn
setup
fn
setup
fn
setup
fn
fail

Tested with v1.0.0 of benchmark from npm.

In general, I found that setup still does get called on occasion, but only after long runs of fn first.

Add library to any public CDN

Now we have public:

//cdnjs.cloudflare.com/ajax/libs/benchmark/0.3.0/benchmark.js

but better to have any new version of library on CDN.

Add local copys of the dependencies

Could you add a directory that contains local copys of the dependencies.
Mainly because if I download the benchmarkjs repository as a zip or tar.gz, the vendor folder contains emtpy folders.
This causes for test page (test\index.html) to fail since require.js doesn't exist.
I'm unable to use GIT due to firewall restrictions, so I can't fetch the dependencies automatically.

Benchmark .add() is undefined (nodejs)?

var Benchmark = require('benchmark');
console.log(Benchmark.add);
console.log(Benchmark.Benchmark.add);
console.log(Benchmark().add);

git clone --recursive stops with the error:

Cloning into vendor/platform.js...
remote: Counting objects: 691, done.
remote: Compressing objects: 100% (308/308), done.
remote: Total 691 (delta 431), reused 633 (delta 373)
Receiving objects: 100% (691/691), 2.02 MiB | 443 KiB/s, done.
Resolving deltas: 100% (431/431), done.
fatal: reference is not a tree: 61dad43825ebb1f391dc79900744fc884029aa7f
Unable to checkout '61dad43825ebb1f391dc79900744fc884029aa7f' in submodule path 'vendor/platform.js'

asnyc suite not aborting deferred benchmarks

Here is a test case: http://jsfiddle.net/9bcVB/18/

I run two deferred tests in an async suite and trigger suite.abort() before the suite can finish, which correctly emits an abort event.
But the benchmarks still run until the suite is complete, firing a complete event.
I first thought this was a requestAnimFrame error, but it also occurs for plain setTimeout-based async benchmarks.

Is this behaviour expected, or did I find i bug? I crawled through the docs, but info on aborting and deferred runs is sparse.

Comparing async and sync benchmarks

May be I did not enough RTFM, but this seems strange:

async x 119,606 ops/sec ±2.39% (76 runs sampled)
sync x 69,544,898 ops/sec ±3.24% (66 runs sampled)

Code to reproduce:

var Benchmark = require('benchmark'),
    suite = new Benchmark.Suite;

suite
    .add('async', {
        defer: true,
        fn: function(deferred) {
            deferred.resolve();
        }
    })
    .add('sync', function() {
        return true;
    })
    .on('cycle', function(event) {
        console.log(String(event.target));
    })
    .run();

Strange results

Why is benchmark.js not getting accurate performance data? I rewrote the test to use the Date object and it's able to determine which method is faster.

• test.js
• test_date.js

Use `process.hrtime()` in Node

Introduced in Node 0.8; appeared in several talks at NodeConf. Will tackle when I have some time...

strange results from async benchmark again

var Bench = require('benchmark')

Bench.support.timeout = false

function each(fns, index, cb) {
  var sync = true
  for (var i = index; i < fns.length; i++) {
    var done = false
    fns[i](function() {
      done = true
      if (sync) return
      each(fns, i + 1, cb)
    })
    sync = done
    if (!sync) return
  }
  cb()
}

function noop(cb) {
  cb()
}

function asyncNoop(cb) {
  setImmediate(cb)
}

var sync = [noop, noop, noop, noop, noop]

var async = [noop, noop, asyncNoop, noop, noop]

var suite = new Bench.Suite

suite.add('sync', function() {
  each(sync, 0, function() {})
})

suite.add('async', function(deferred) {
  each(async, 0, function() {
    deferred.resolve()
  })
}, {
  defer: true
})

suite.add('setImmediate()', function(deferred) {
  setImmediate(function() {
    deferred.resolve()
  })
}, {
  defer: true
})

suite
.on('cycle', function(event) {
  console.log(String(event.target));
})
.run({async: true})

Output

sync x 5,450,974 ops/sec ±0.25% (99 runs sampled)
async x 38,051 ops/sec ±2.67% (74 runs sampled)
setImmediate() x 305,534 ops/sec ±0.24% (85 runs sampled)

We see just a huge difference between async and setImmediate() for no reason. Actually neither benchmarking via Http server nor via adhoc script do not show such difference.

aync func handling

Hi,
Can i do something like this :

benchmark.add('test db call', function(done) {
//make the io (db call)
//and once finished, call done()
});

i would like to do a benchmark of db operations which we do, wanted to know if async support like above is possible.

Thanks.

show: a standalone runner/ui

hey

Im developping a lib for webgl https://github.com/jeromeetienne/tquery and im using benchmark.js to bench it. The library is real nice but i failed to find an easy way to display the results. So i did a runner. You can find it here https://github.com/jeromeetienne/benchrunner

If you like it, i will be more than happy to modify it to match the need of benchmark.js project. and do a pull request

Jerome

Add Benchmark.Event#target and currentTarget

Instead of passing the benchmark as a second argument as

suite.on('cycle', function(event, bench) {
  console.log(bench);
});

we should use

suite.on('cycle', function(event) {
  console.log(event.target);
});

and also use event.currentTarget to reference the suite instance.

Disambiguate `Benchmark#cycles` and the `onCycle` event from `Benchmark#stats.samples.length`

The number of times onCycle is called doesn't match up with Benchmark#cycles because onCycle is also called per sample run as well.

Make benchmark.js work in Adobe AIR with async cycles.

A regression in Benchmark.js 0.3.0 was introduced.
While synchronous cycles, the default, is supported asynchronous cycles are not.
We should have the fallback avoid use of the Function constructor so we don't throw a JavaScript security error in AIR.

Benchmark.Suite setup/teardown ?

is there a setup/teardown mechanism at the suite level ?

i looked and didnt see it

The fastest function is the first .add()

Why is it that it's always returning the first function added as the fastest?

asynchronous benchmark setup.

I have some things I need to do asynchronously in a setup function. As far as I can tell, there's no way to do this -- if there is, I apologize!

Here's an example (for node) of the kind of thing I want to be able to do:

var fs = require("fs");
var Benchmark = require("benchmark");

new Benchmark.Suite().
  add("Test with asynchronous setup", {
    'defer': true,
    'setup': function (deferred) {
      var _this = this;
      fs.readFile("data.txt", function (_, data) {
        _this.data = data;
        deferred.resolve();
      });
    },
    'fn': function (deferred) {
      console.log(this.data);
      deferred.resolve();
    }
  }).
  run();

What is the unit of Benchmark.prototype.stats.mean?

Remove `Math.random` from the example in `/example/jsperf/index.html`

Remove Math.random from the example in /example/jsperf/index.html.

Async function

Hi,

I want to test some async functions (database inserts).

There are two ways of testing:

1. doing all the inserts of one type ('async fct 1' for instance) in parallel, and stop the clock when all the functions have finished (callback)
2. doing all the inserts of one type in sequence (synchronous), and wait the last callback.

I try with both 'async' options (true/false), but I notice than the benchmark doesn't wait the callbacks to give the result.

Is there any function like "done()" to call as callback?

suite.add('async fct 1', function() {
  setTimeout(function() {
    return true;
  }, 10)
 })
 .add('async fct 2', function() {
  setTimeout(function() {
    return true;
  }, 20)
 })
 .on('complete', function() {
    console.log('1 : Fastest is ' + this.filter('fastest').pluck('name'));
  })
 .run({
    'async' : true/false
 });

Thank you,
Philmod

Add Benchmark `assert` option to pre-run test that a function returns an expected result.

Add Benchmark assert option to pre-run test that a function returns an expected result.
This can be useful to ensure two snippets being tested at least return the same result.

Fix testing on Opera Mini :)

something kills Opera Mini on its proxy's side.

Benchmark.options.onStart and onComplete are called multiple times per Benchmark run

Consider the following code:

var optionsStartCount = 0,
    benchStartCount = 0,
    optionsCompleteCount = 0,
    benchCompleteCount = 0;
Benchmark.options.onStart = function () {
    optionsStartCount++;
};
Benchmark.options.onComplete = function () {
    optionsCompleteCount++;
};
Benchmark("start tracker", function () {
    Math.pow(20,500);
})
.on("complete", function (e) {
    benchCompleteCount++;
    console.log("options start count " + optionsStartCount);
    console.log("bench start count " + benchStartCount);
    console.log("options complete count " + optionsCompleteCount);
    console.log("bench complete count " + benchCompleteCount);
})
.on("start", function () {
    benchStartCount++;
}).run({async: true});

Why is the Benchmark.options.onStart and onComplete functions called several times? Shouldn't they be called just as many times as the benchmark's own start and complete event listeners?

Event.target not set correctly

Benchmark looks great.

Unfortunately, the example on the home page doesn't seem to work out of the box on node v0.8.

I tried making a new file and running it in node

var Benchmark = require('./benchmark.js');

var suite = new Benchmark.Suite;

// add tests
suite.add('RegExp#test', function() {
  /o/.test('Hello World!');
})
.add('String#indexOf', function() {
  'Hello World!'.indexOf('o') > -1;
})
.add('String#match', function() {
  !!'Hello World!'.match(/o/);
})
// add listeners
.on('cycle', function(event) {
  console.log(String(event.target));
})
.on('complete', function() {
  console.log('Fastest is ' + this.filter('fastest').pluck('name'));
})
// run async
.run({ 'async': true });

which will output this:

undefined
undefined
undefined
Fastest is String#indexOf

It looks like the process is working correctly, but the Event objects are not being formed correctly. The cycle event is missing the target property and looks like this:

{ type: 'cycle' }

I can see in the source that event.type is set explicitly (e.g., line 1659 for the cycle event) while event.target is coming from elsewhere (last, bench). Nothing jumps out to me about any of this though, so I was hoping someone else might see something.

Also, I'm running on the brand new node v0.8 (released yesterday), in case that could be related.

Getting ReferenceError when using setup with Nodejs

This test works correctly:

var Benchmark = require('benchmark'),
    schedule = require('../index'),
    suite = new Benchmark.Suite('dependency-graph');

suite
.add('create graph', function() {
    var tasks = ['A', 'B', 'C', 'D', 'E'];  
    schedule.dependencyGraph(tasks);
})
.on('error', function(event) {
  console.log(event);
})
.on('cycle', function(event) {
  console.log(String(event.target));
})
.run({async: false});

However, when I try and move the tasks setup to setup I get a reference error on schedule.

var Benchmark = require('benchmark'),
    schedule = require('../index'),
    suite = new Benchmark.Suite('dependency-graph');

suite
.add('create graph', function() {
    schedule.dependencyGraph(tasks);
},
{
    'setup': function() {
        var tasks = ['A', 'B', 'C', 'D', 'E'];  
    }
})
.on('error', function(event) {
  console.log(event);
})
.on('cycle', function(event) {
  console.log(String(event.target));
})
.run({async: false});

The error event reports:

error: [ReferenceError: schedule is not defined],

Looks like there is a scope issue that I've introduced by specifying setup but I'm not sure how to resolve. What's the correct way to run setup code before every test when using Nodejs?

Newbie question

Hi. I've just started using this library and it's been great. I was just wondering, is it possible to set a specific number of times an operation should run, then calculate the time taken? Sorry if it's written in the documentation; I wasn't sure where to look.

Remove the vendor folder and use npm’s `node_modules` folder instead (through `dependencies`/`devDependencies`)

I’ve been doing this for luamin and it feels so good. Updating a “submodule” is just a matter of running npm update, and not having the submodule changes show up in the commits/diffs is another plus.

What do you think, @jdalton?

Benchmark.options.onComplete is called by the internal cloned benchmarks.

Benchmark.options.onComplete is called by the internal cloned benchmarks.
The internal cloned benchmarks should not execute the onComplete event listeners of the host benchmark.

Fix `window.init()`

http://twitter.com/jsprf/status/21334855147
http://twitter.com/jsprf/status/21334917319
http://jsperf.com/hasownproperty-vs-for-in/5#count=500
mathiasbynens@85b0374

How to share code between test runs?

Here is a sample benchmark I am trying to make work:

"use strict";

var Benchmark = require("benchmark");

function makeArray(length) {
    var array = new Array(length);
    for (var i = 0; i < length; ++i) {
        array[i] = 1;
    }
}

var suite = new Benchmark.Suite("Loops with arrays");

var stepValues = [1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024];

stepValues.forEach(function (stepValue) {
    suite.add(stepValue + " at a time", function () {
        var array = makeArray(64 * 1024 * 1024);
        for (var i = 0; i < array.length; ++i) {
            array[i] *= 3;
        }
    });
});

suite.on("complete", function () {
    console.log(Object.getPrototypeOf(this));
    this.forEach(function (result) {
        if (result.error) {
            console.error(result.name + ": ", result.error);
        } else {
            console.log(result.name + ": ", result.stats);
        }
    });
});

suite.run();

This doesn't work because of the function decompilation; it gives a bunch of ReferenceError: makeArray is not defined. OK, but how can I make this work? I tried attaching to this inside the setup option, but that was a no-go.

[Question] How is "hot" code prevented?

I'm giving a 6 minute talk on jsPerf in a few days, & like to know how is "hot" code prevented. please? Seems that FireFox has several optimization layers, & I'd expect tight loops will make FF jump to optimize early.

From your talks I see you use several loops of different lengths, so my brief bullet points would be:

• Minor prevention of "hot" code with variable-length loops
• Sometimes best to test on larger chunks of real-world code than tiny isolated routines

Do you think this is fair/accurate please? BTW, I'm not saying to never test on tiny routines, but my ASM programming experience & lessons on ninja JS engine optimization tricks, indicates code context is important.

Add more forOwn unit tests to check enumeration issues.

Add more forOwn unit tests to check enumeration issues.

Document dependency on platform.js

The Benchmark.platform object needs a dependency on platform.js in order to work. Make note of this in the documentation to avoid confusion.

Fatal Bug in the function interpolate (cannot work with '$')

If you have for example console.log('$') in a benchmark code, it fails to run the benchmark suite.
cf. https://developer.mozilla.org/en/docs/JavaScript/Reference/Global_Objects/String/replace

To fix this, apply the following patch. Thanks.

--- benchmark.js.bak    2013-01-03 13:06:16.739109681 +0900
+++ benchmark.js        2013-01-03 15:06:32.846285741 +0900
@@ -1630,7 +1630,7 @@
   function interpolate(string, object) {
     forOwn(object, function(value, key) {
       // escape regexp special characters in `key`
-      string = string.replace(RegExp('#\\{' + key.replace(/([.*+?^=!:${}()|[\]\/\\])/g, '\\$1') + '\\}', 'g'), value);
+      string = string.replace(RegExp('#\\{' + key.replace(/([.*+?^=!:${}()|[\]\/\\])/g, '\\$1') + '\\}', 'g'), value.replace(/\$/g, '$$$$'));
     });
     return string;
   }

strange async test results

var Suite = require('benchmark').Suite

var suite = new Suite

suite.add('async test', function(deferred) {
  deferred.resolve()
}, {
  defer: true
})

suite
.on('cycle', function(event) {
  console.log(String(event.target));
})
.on('complete', function() {
  console.log('Fastest is ' + this.filter('fastest').pluck('name'));
})
.run({async: true})

Outputs the following:

async test x 781 ops/sec ±0.49% (11 runs sampled)

Which is obviously no sense. What's going on here?

Add benchmark and suite `score` API.

Add benchmark and suite score API.

Add a benchmark `assert` option to pre-run and test that it returns an expected result.

Add a Benchmark assert option to pre-run and test that it returns an expected result.
This can be useful to ensure snippets being tested at least return the same result.

bower distribution?

Are there any plans to distribute benchmark.js in bower and other package managers? Likewise you are doing with lodash.

Make use of the high resolution timer

currently in chrome canary as window.performance.webkitNow()

It's a floating point millisecond measurement since the page was created.

spec here: http://www.w3.org/TR/hr-time/ (out of date)

Update by @mathiasbynens: The above spec link is the out of date version. Check out the latest editor’s draft here: http://dvcs.w3.org/hg/webperf/raw-file/tip/specs/HighResolutionTime/Overview.html

IE10 will probably have it as well.

quick async example in readme

to get started would be useful. Can you pass a callback to Suite#add()'s function or how does the async support work? The readme example has async: true but the code isn't async

Code coverage

Istanbul ftw.

Would it be possible to add per-sample setup/teardown?

I could use such an option when testing a snippet of JS that has to run on a large DOM structure. In those cases, the DOM tree must be cloned, appended to the document, and only then can the timed code run once on the clone. After it has completed, the modified tree must be discarded.

Currently, if the DOM tree is too large, the majority of time is spent appending it to the document, so the amount of time spent executing the JS which modifies the tree pales in comparison to the amount of time spent actually attaching the tree.

Update version number in code

Benchmark.version still says 0.2.2.

Add unit tests for Suite array methods like shift(), splice(), slice() and others.

Add unit tests for Suite array methods like shift(), splice(), slice(), and others.
And test that we cover IE issues with array-like-objects and splice() / shift().

[Question] Async setup and onStart methods

Do the Benchmark.setup and onStart methods support async methods ?
The code would be something like

new Benchmark.Suite({}).add({
    'setup' : function(){
        window.setTimeout(function(){
            var i = 10;
        },1000);
    },
    defer: true
    'fn': function(def) {
        window.setTimeout(function() {
            console.log(i); // Do somethign with 'i', but only when it is defined by setup
            def.resolve();
        }, 100);
    }
});

The setTimeout will be replaced with an async method.

P.S: Not able to find a google groups for questions, so asking here

Add Benchmark#clone unit tests.

Add Benchmark#clone unit tests.