japgolly / scalajs-benchmark

Benchmarks: write in Scala or JS, run in your browser. Live demo:

scalajs-benchmark

Benchmarks: write in Scala or JS, run in your browser.

See an online demo here: https://japgolly.github.io/scalajs-benchmark/.
The tiny source code is here: gh-pages/demo.

Changelogs — Latest: 0.10.0.

How do I use it?

1. Include scalajs-benchmark as a dependency in your Scala.JS project.

libraryDependencies += "com.github.japgolly.scalajs-benchmark" %%% "benchmark" % "0.10.0"

1. You write benchmarks.

import japgolly.scalajs.benchmark._
import japgolly.scalajs.benchmark.gui._

object Example {
  val suite = GuiSuite(
    Suite("Example Benchmarks")(

      // Benchmark #1
      Benchmark("foreach") {
        var s = Set.empty[Int]
        (1 to 100) foreach (s += _)
        s
      },

      // Benchmark #2
      Benchmark("fold") {
        (1 to 100).foldLeft(Set.empty[Int])(_ + _)
      }
    )
  )
}

(Hey, can you make that 1 to 100 configurable in the GUI? You sure can.)

1. Add a tiny loader HTML like this.

2. Create a main app and point scalajs-benchmark at your suite of benchmarks.

import org.scalajs.dom.document
import japgolly.scalajs.benchmark.gui.BenchmarkGUI

object Main {

  def main(args: Array[String]) = {
    val body = document getElementById "body"
    BenchmarkGUI.renderSuite(body)(Example.suite)
  }
}

If you have a library of different benchmarks, instead of using renderSuite, you use renderMenu to create an app that lets the user navigate the library and run benchmarks.

Example:

BenchmarkGUI.renderMenu(body)(
  suites.example.Examples.all,
  suites.scala.all)

1. Compile; run in browser. Done.

Support

If you like what I do —my OSS libraries, my contributions to other OSS libs, my programming blog— and you'd like to support me, more content, more lib maintenance, please become a patron! I do all my OSS work unpaid so showing your support will make a big difference.

---

Note: This is not affiliated with the official Scala.JS project; it's just named to be informative rather than interesting.

scalajs-benchmark's Issues

Log10 scale option for the main score wanted

It will allow easily compare scores for benchmarks parametrized by geometric sequences (10, 100, 1000, etc.).

Also, it would be helpful in case of too slow outliers when winners are represented by tiny bars above zero.

Add query params

• autostart for batch mode
• all for batch mode
• zip for batch mode
• wi/i like jmh

Add ability to export results

Output in a similar format to jmh

Allow modification of EngineOptions from GUI

Unexpected linker error with JDK 8

I'm trying to use the Scala.js benchmark in this PR: plokhotnyuk/jsoniter-scala#526

Currently, build sbt jsoniter-scala-benchmarkJS/fullOptJS fails with the following error:

[error] java.nio.file.FileSystemLoopException: /META-INF/resources/webjars/jstat/1.9.3
[error]         at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:294)
[error]         at java.nio.file.FileTreeWalker.next(FileTreeWalker.java:372)
[error]         at java.nio.file.Files.walkFileTree(Files.java:2706)
[error]         at org.scalajs.linker.PathIRContainer$.org$scalajs$linker$PathIRContainer$$walkIR(PathIRContainer.scala:95)
[error]         at org.scalajs.linker.PathIRContainer$JarIRContainer.$anonfun$sjsirFiles$2(PathIRContainer.scala:67)
[error]         at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
[error]         at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1$$anon$2.block(ExecutionContextImpl.scala:75)
[error]         at java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3313)
[error]         at scala.concurrent.impl.ExecutionContextImpl$DefaultThreadFactory$$anon$1.blockOn(ExecutionContextImpl.scala:87)
[error]         at scala.concurrent.package$.blocking(package.scala:146)
[error]         at org.scalajs.linker.PathIRContainer$JarIRContainer.$anonfun$sjsirFiles$1(PathIRContainer.scala:59)
[error]         at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
[error]         at scala.util.Success.$anonfun$map$1(Try.scala:255)
[error]         at scala.util.Success.map(Try.scala:213)
[error]         at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
[error]         at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
[error]         at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
[error]         at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
[error]         at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1402)
[error]         at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
[error]         at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
[error]         at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
[error]         at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
[error] (jsoniter-scala-benchmarkJS / Compile / scalaJSIR) java.nio.file.FileSystemLoopException: /META-INF/resources/webjars/jstat/1.9.3

All is fine when JDK 11+ is used:

sbt -java-home /usr/lib/jvm/openjdk-15 jsoniter-scala-benchmarkJS/fullOptJS

Log {fast,full}OptJS info to JMH JSON

Release for Scala.js 1.0

Suboptimal layout of bars in some charts

Top 1/3 of the chart is empty in the following example:

Add ability to automate running and saving results

Benchmarking `async` functions?

Hello again! 😁

We're looking for something to benchmark Cats Effect on JS, and of course you have the perfect project for this :) The trickyness for our use-case however is that we need to be able to benchmark an async function, e.g. something that returns a Future or a js.Promise or asynchronously invokes a callback. I took a walk through your source, which is really excellent btw, but it seems like to accommodate this would require quite a bit of changes to the Benchmark, Engine, and Clock interfaces/implementations at least. Is this something you'd be open to, or is there a problem with benchmarking an asynchronous function that I'm not realizing?

Thanks in advance!

Add CSV export

Remove blanks from param names in JMH JSON

jmh visualiser doesn't like

eg. "Sample data" should be "Sample_data"

Task Shootout: FS2 / Cats / Scalaz / Monix

Ability to export results into JSON representation of JMH format wanted

Use cases:

1. plotting with external tools like jmh-visualizer
2. importing results into benchdb

Shootout: Nyaya / ScalaCheck / ScalaProps

Append fastOptJS (or similar) to filenames when built with fastOptJS

10x times difference in results between online and file accesed benchmark page

Please help me to find source of differences why the benchmark page loaded from HTTP URL shows about 10x times worse results comparing to loaded from file URL.

Chrome

Firefox

Getting started is challenging

Hello!

This looks like a great project ...but I found it difficult getting something simple up and running and gave up. The details in the readme seem to be inadequate and the arrangement of the demos/examples is too involved for a quick start.

Examples:

1. The readme does not include a simple build.sbt arrangement.
2. How to run it - I went for fastOptJS and a local http server but is that right?
3. It doesn't mention @JSExportTopLevel("main") - I should have known about that one but I still tripped over it.

Doing the prescribed minimum for me produces:

Should I be exporting modules? The HTML file examples suggest not?

I'll have another go some other time, just thought the feedback might be useful as it may be a barrier to adoption. This is what I did (nothing clever, just followed/interpreted the readme):

PurpleKingdomGames/indigo@ea622ea

All the best,

Dave

Wrong conversion to hours and days

scalajs-benchmark/benchmark/src/main/scala/japgolly/scalajs/benchmark/engine/TimeUtil.scala

Lines 39 to 40 in 26a8881

	case TimeUnit.HOURS => _ / 3660000
	case TimeUnit.DAYS => _ / 3660000 / 24

The 3600000 divisor should be used instead of 3660000.

Exclude param to BM graph label when param is Unit

See https://japgolly.github.io/scalajs-benchmark/#/demos/no_params

Saving batch results to one file wanted

The Safari browser asks to allow downloading for each file when the page is opened locally.

Automatic allowing of downloads works only for websites in it.

It is not a problem to press the Allow button for each suite and then merge results by a command like jq -s '[.[][]]' sjsbm-*.json > safari.json.

But it starts to be unpractical when there are a lot of suites because the browser blocks execution of benchmarks until the Allow button will be pressed.

BTW: the Brave browser doesn't block running of benchmarks if the multi-file download was allowed but it asks for saving of each file.

Allow modification of EngineOptions in batch mode

Division by 0 error during deselection of all benchmarks on the batch mode page

It happens with Chrome and Opera browsers and broke the benchmark page.

I started to notice it when restarting a part of jsoniter-scala benchmarks in the batch mode due to those browsers fail the page during benchmark running with the "Aw, snap!" or "Page crashed" errors correspondingly.

Yep, the benchmarks pass unit tests with Node.js and run with Firefox successfully but this year started to fail sporadically with Chrome and Opera.

Include various iteration examples

Dirty secret of Scala collections is that while-loops are much faster than foreach, and for comprehensions. Would be interesting to see those in the benchmark.

Include folder name in output filename by default

Wrong mode is serialized to JMH JSON when OpsPerSec is selected

Below is a sample from the resulting output:

{
  "benchmark": "ArrayOfDoublesReading.jsoniterScala",
  "mode": "avgt",
  "threads": 1,
  "forks": 1,
  "jdkVersion": "1.8",
  "vmName": "Scala.JS",
  "vmVersion": "1.1.0",
  "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36",
  "warmupIterations": 5,
  "warmupTime": "1 s",
  "warmupBatchSize": 1,
  "measurementIterations": 5,
  "measurementTime": "1 s",
  "measurementBatchSize": 1,
  "primaryMetric": {
    "score": 42313.722900788394,
    "scoreError": 409.4079368962774,
    "scoreConfidence": [
      41904.314963892124,
      42723.13083768468
    ],
    "scoreUnit": "ops/s",
    "rawData": [
      [
        42293.15468674238,
        42417.35945548029,
        42247.57714402512,
        42183.73921242692,
        42426.78400526727
      ]
    ]
  },
  "secondaryMetrics": {}
},

It leads to that the JMH Visualizer renders a chart for ops/s using wrong metric captions and hints like here:

Make useful for perf regression testing

use case: Every time library X is release, BM results are recorded and compared with history.

Will be hard to account for environmental changes.

Anyhoo, would be a nice-to-have.

More Scalaz/Cats comparison

I'd like to see:

• StateT + EitherT (cmp with my own StateEither)
• Traverse with List|Vector|Stream

AsyncEffectShootout with MacrotaskExecutor

Follow-up to #206: would be interesting to run the AsyncEffectShootout with the new MacrotaskExecutor just released:
https://github.com/scala-js/scala-js-macrotask-executor

This is the polyfill mentioned in #203 (comment). Should level the playing field for CE3 hopefully 😅

Not sure how straightforward it is to swap out the executor for the various effect implementations 🤔

Add checkboxes to completed BMs to hide from graph

Remove blank lines from JMH JSON

Switch to ops/s as the main score metric in JMH JSON wanted

It will be easier to compare winners on charts.

Please, consider making throughput scores (ops/s) as default.

Middle-click for new tab now working in Chrome

JMH JSON generation slow for big result sets

The Batch mode serializes benchmarks of suites in a random order

Expected order is the same as an execution order.

Switch dots & commas according to locale

Use standard high-resolution timer in browsers

http://www.w3.org/TR/hr-time/#sec-DOMHighResTimeStamp

http://caniuse.com/#feat=high-resolution-time

window.performance.now();

Show library versions in UI

Eg. Scalaz is currently 7.1.3, Shapeless 2.2.5, etc.

For sec in [1,10) use 3 dp instead of 2

Add a copy-to-clipboard button to result text

Too big values in ops/s error column

Show ETA before/during/after

By before I mean it should show how long it will take before a user hits start