Benchmark unfairly excludes compilation time. about optimize-js

[nolanlawson]

So you've highlighted an important issue here; there's a measurement that the benchmark is not taking into account and this does have the potential to unfairly inflate optimize-js's numbers.

I think the responsible thing would be to add an additional number to measure the compilation time (which indeed should go up). But the question is how best to observe compilation time from JavaScript, which turns out to be a difficult question to answer.

1․ Start measurement from the moment that the script element is added to the DOM.

As you say this would get muddied by network overhead, which would make the tests inconsistent and unreliable. You could use a cached version of the script, but the problem is that some engines may pre-JIT a cached script (I'm told V8 does this).

2․ Start measurement from the moment that the script element is added to the DOM, but subtract the time from the Resource Timing API.

Besides the Safari problem, there's actually no guarantee that the browser will only begin parsing once it's received the last byte of JavaScript – it can start parsing the moment it starts to receive script.

3․ Download the script with XHR/fetch, and call eval() on it.

eval() has side effects and so isn't exactly equivalent to <script> tags. Also <script> tags are more the norm on the web, so it's better to test those.

Due to all these issues, I'd say your first suggestion is probably the best one, but we should do the following:

1. Download the script as text/plaintext (to avoid parsing)
2. Convert it to a blob/dataURL (not sure which one is best but I imagine blob since otherwise browsers may cache the dataURL. we will probably want to append random whitespace to the script as well)
3. Set the blob/dataURL as the source of a non-async non-deferred script and add it to the DOM
4. Measure the time from adding the script to the DOM and the first line of code in the script as "load/compile time" and the current measured time as "run/execute time"

I think reporting both numbers is important to ensure we don't mislead people on the benefits of optimize-js, so we should definitely do this. 👍

/cc @toddreifsteck who helped walk me through some of the tricky bits of this :)

[nolanlawson]

OK, I believe I have found a better system. By creating script elements, setting their textContent to the script source, then randomizing the source by adding a random string, then doing document.body.appendChild(script), we can fully measure compile+execute without measuring the network.

I set a mark right before appendChild(script), and another mark right at the end of the script. I confirmed in the Chrome timeline and Edge (via Windows Performance Analyzer) that the performance measures include parsing and executing. I can also confirm that Chrome is re-parsing every time due to the randomization (which it wouldn't otherwise; it would cache the parsed/JITed code).

I'll update the benchmark and post new numbers and a new version of optimize-js as soon as I can. Testing so far indicates my numbers were indeed overly optimistic, but optimize-js still comes out on top for the most part. Thank you for raising this issue! 😃

[aickin]

Oooooh, I love the idea of a blob/dataURL; hadn't occurred to me, and I am agreed on all counts about the imperfections of the three ideas I proposed. I didn't even know about using blobs as URLs; that's really neat!

And (perhaps obviously) I am +1 that this is important to do to make sure the numbers are fair. Thanks!

[nolanlawson]

This is my highest-priority task for optimize-js before merging in recent changes. I'm pretty sure the current benchmarks are far too optimistic.