Strange results about benchmark.js

[jdalton]

Your code on line 14-16 should go into the test setup function because the benchmark test is repeated several times to create a sample.

[jdalton]

Documentation on the setup option can be found here.

[bootstraponline]

Thank you! That worked.

[bootstraponline]

I thought it fixed the issue but it didn't. Emscript is reported as fastest on the first page load, which is accurate.

emscript x 2,921 ops/sec ±74.94% (94 runs sampled) test.js:40
pagedown x 450 ops/sec ±2.12% (90 runs sampled) test.js:40
Fastest is emscript

Then on refresh it changes to:

emscript x 243 ops/sec ±131.18% (96 runs sampled) test.js:40
pagedown x 433 ops/sec ±2.27% (86 runs sampled) test.js:40
Fastest is pagedown 

The hand written date test always returns the correct perf data (emscript is faster). Is there still something missing from the benchmark.js test?

[jdalton]

You're probably exceeding the length of the text so it's measuring less work. Benchmark.js automatically creates a test loop with an iteration that could be several times higher than your MAX of 10000. You could try moving the for-loop into the test too.

[bootstraponline]

Adding a for loop with 10 iterations in each benchmark produced consistent results. Now emscript is always measured as faster.

What does it mean that the length of the text is being exceeded? The text in this benchmark is static despite the code being written to handle dynamic text.

[jdalton]

What does it mean that the length of the text is being exceeded? The text in this benchmark is static despite the code being written to handle dynamic text.

It looks like that if ( text.length > allocSize ) { is determining if it needs to walk text moving the pointer. Eventually it gets to the end say in 10 iterations, so the rest of the say 20,000 or so are doing less work.

[bootstraponline]

If I comment out all the length checks though the problem still occurs. It's almost as if the benchmark is too fast so the for loop is required to slow it down.

emscript x 242 ops/sec ±131.37% (95 runs sampled) test.js:44
pagedown x 459 ops/sec ±1.70% (88 runs sampled) test.js:44
Fastest is pagedown 

[jdalton]

I don't know enough about your code to help you out. I know it's most likely a problem with your test setup as Benchmark.js is used by jsPerf and other projects so something like basic functionality/accuracy is pretty well covered.

[bootstraponline]

How many other projects are using it to test emscripten code? I think having to slow down a test case to get consistent results is a bug in benchmark.js considering it works fine when measuring time with the Date object.

Specifically the bug is benchmark.js reports correct results on the first page load and then fails to measure properly on subsequent refreshes. This isn't a problem with the tests because measuring the time with new Date is consistent.

[jdalton]

I think having to slow down a test case to get consistent results is a bug in benchmark.js considering it works fine when measuring time with the Date object.

Things off the top of my head that could be causing an issue are things like global variable references, like text as some engines (TraceMonkey, JaegerMonkey) don't like that, and garbage collection pauses. Benchmark.js calculates the number of iterations it needs to get an accurate reading usually within a 50ms time frame.

Google, jQuery, MooTools, Underscore, Esprima, and the like use Benchmark.js (directly or via jsPerf) but as always you are free to use whatever you want.

[bootstraponline]

It takes about 21 ms for the call to complete. Is there a best practice for using benchmark.js on very fast code? I can't be the first to run into this issue.

[jdalton]

It takes about 21 ms for the call to complete. Is there a best practice for using benchmark.js on very fast code?

Benchmark.js is used to test tons of micro-optimizations on jsPerf.com.

I can't be the first to run into this issue.

Tinker around with it a bit and try to produce a small test case and a link to running code that shows the problem.

A test compiles to something like

<setup>
<start timer>
while (++i < count) {
<test>
}
<stop timer>
<teardown>

If you are in Node make sure to expose your variables on the global object.

[bootstraponline]

The compiled test doesn't seem helpful.

(Benchmark.uid1340958730535runScript || function() {
})();
Benchmark.uid1340958730535createFunction = function(t1340958730535) {
    var r1340958730535, s1340958730535, m1340958730535 = this, f1340958730535 = m1340958730535.fn, i1340958730535 = m1340958730535.count, n1340958730535 = t1340958730535.ns; // Must be in setup function or test results will be invalid.
    // http://benchmarkjs.com/docs#Benchmark
    // http://benchmarkjs.com/docs#prototype_setup
    // Emscripten
    var Pointer_stringify = Module['Pointer_stringify'];
    var _str_to_html = Module['_str_to_html'];
    var malloc = Module._malloc;
    var realloc = Module._realloc;
    var writeStringToMemory = Module.writeStringToMemory;
    var free = Module._free;

    // Move text into memory.
    var allocSize = text.length; //+ 1024; // at least 1024 chars
    var pointer = malloc(allocSize);
    s1340958730535 = n1340958730535.webkitNow();
    while (i1340958730535--) { /*
    // In production the text is not static.
    // Must check for text increase and
    // write the string to memory.
    if ( text.length > allocSize ) {
      allocSize = text.length << 1; // double
      pointer = realloc( pointer, allocSize );
    }
*/
        writeStringToMemory(text, pointer);
        Pointer_stringify(_str_to_html(pointer));
    }
    r1340958730535 = (n1340958730535.webkitNow() - s1340958730535) / 1e3; // no operation performed
    return {elapsed: r1340958730535,uid: "uid1340958730535"}
}

I think I figured out the reason for the inconsistency.

200 iterations
emscript is slower by 141 ms
emscript is: 552
pagedown is: 411

300 iterations
emscript is faster by 44 ms
emscript is: 561
pagedown is: 605

In my hand written test I used a fixed number of iterations and emscript is always measured as fastest. Benchmark.js seems to dynamically select an amount where pagedown is sometimes faster and sometimes slower. If I adjust the Benchmark.options then Benchmark.js will run the test longer, generate a better ± percentage, and reach the same result.

So there's no bug in Benchmark.js, I just have to set the options so it takes longer to measure this particular function.

It looks like adding minTime fixed the consistency problem.

emscript x 4,681 ops/sec ±0.96% (5 runs sampled) test.js:44
pagedown x 569 ops/sec ±0.09% (7 runs sampled) test.js:44
Fastest is emscript 

[jdalton]

The compiled test doesn't seem helpful.

You may also get the compiled code after a benchmark has run by accessing the benchmark's "compiled" property.

So there's no bug in Benchmark.js, I just have to set the options so it takes longer to measure this particular function. It looks like adding minTime fixed the consistency problem.

Good digging! You might also try adjusting the maxTime to get a larger sample size. The Benchmark.options object is the default options object used when creating new benchmarks. You may also pass these option properties via the Benchmark constructor or Benchmark.Suite#add.

[bootstraponline]

Thanks for your help and suggestions.