bencheeorg / benchee_html Goto Github PK

Someone* reached out to me after reading WHAT’S THE FASTEST DATA STRUCTURE TO IMPLEMENT A GAME BOARD IN ELIXIR? and made a very nice graphic:

Having this graph built-in would be very nice and probably not too hard as I suspect plotly supports this.

*there are multiple people by their name on github so I don't dare to gamble on the @mention

Part of the Benchee Suite is system data including information about CPU, Memory, OS, Elixir and Erlang version.

We already display the Elixir and Erlang version on the top right. It'd be nice to add the other information as well. The top right isn't big enough for it so we'd need another place.

As it's part of the main content I'd suggest another panel with that information perhaps as a table or something else. Not entirely sure tbh.

Also not sure if that panel should come first or last or where it should appear. In the console formatter that information comes first but only because we can get it right away, I think the benchmark results are still most important :)

Provide the possibility to open the HTML in the default browser once HTML generation has finished.

An option inline all assets (CSS/JavaScript) would be great. This way the files could be totally independent moved and copied around without worrying that all the appropriate asset folders have to be present.

From twitter:

Would it be possible to set a scale of each chart the same by default ?

What I think this would/could entail is to set the scale of all raw tun time charts to be the same, probably also the same for the box plots. To do this effectively, it'd be convenient if benchee provided some meta statistics aka what are the minimum/maximum over the whole sample set (or well minimum of minimums/maximum of maximums).

Probably best implemented as an option, setting it as the default might also cause problems if the different jobs are wildly differing in how fast/slow they are.

Also, maybe @fazibear has some input as to what exactly he wishes for.

This will allow me to use benchee_html in Poison itself. :)

Seems I finally found one of the differences of poison 3 vs. poison 2.

The problem is our new percentiles that have integers as keys in the maps:

iex(1)> Poison.encode! %{1 => "lol"}
** (Poison.EncodeError) expected string or atom key, got: 1
    (poison) lib/poison/encoder.ex:188: Poison.Encoder.Map.encode_name/1
    (poison) lib/poison/encoder.ex:212: anonymous fn/4 in Poison.Encoder.Map.encode/3
    (poison) lib/poison/encoder.ex:214: Poison.Encoder.Map."-encode/3-lists^foldl/2-0-"/3
    (poison) lib/poison/encoder.ex:214: Poison.Encoder.Map.encode/3
    (poison) lib/poison.ex:41: Poison.encode!/2

vs.

iex(1)> Poison.encode! %{1 => "lol"}
"{\"1\":\"lol\"}"

This should trigger in benchee_json, but right now this code is still inlined in benchee_html:

job_json = JSON.encode!(%{
        statistics: scenario.run_time_statistics,
        run_times: scenario.run_times
      })

We can't switch to a newer poison version as lots of people rely on Poison < 3 in some phoenix versions etc.

Mitigation:

• manually stringify keys
• switch to a different json library that handles these things like jason (not sure if it handles them/worth a try)

Like benchmarks/benchmark.html - makes for a better first time user experience imo and documentation on how to change it is still there.

Right now the raw run time charts all go up to a certain level which is understandable as there is a minimum execution time. I think it'd be better if benchee_html set a custom start value for the y-axis that is something like minimum - 10% * minimum so that the height of the chart is used more effectively to show the differences.

what might help here from plotly.js (oe well hopefully will) is setting y0.

Example of what's "broken":

When the names are rather long and we scale the graph down from width it seems they can end up cut off. Maybe the height can be set higher or some other layout option?

Would be nice if the stats from memory_time: were a part of the HTML output :).

Is there any way to configure benchee or benchee_html to compare the inputs on one graph?

I can do this with something like the following:

Benchee.run(
  %{
    "tx_benchmark 100" => fn -> TraditionalLedger.run_benchmark_file("./data/transactions_100.csv") end,
    "tx_benchmark 1000" => fn -> TraditionalLedger.run_benchmark_file("./data/transactions_1000.csv") end,
    "tx_benchmark 10000" => fn -> TraditionalLedger.run_benchmark_file("./data/transactions_10000.csv") end,
  },
  formatters: [
    {Benchee.Formatters.Console, extended_statistics: true},
    {Benchee.Formatters.HTML, file: "benchmarks/results.html"},
  ],
  after_each: fn _input -> reset_database() end,
)

But doing it like the following "feels" better in terms of the code, but the HTML comparison charts will put these on separate pages:

Benchee.run(
  %{
    "tx_benchmark" => fn input -> TraditionalLedger.run_benchmark_file(input) end,
  },
  formatters: [
    {Benchee.Formatters.Console, extended_statistics: true},
    {Benchee.Formatters.HTML, file: "benchmarks/results.html"},
  ],
  inputs: %{
    "100" => "./data/transactions_100.csv",
    "1_000" => "./data/transactions_1000.csv",
    "10_000" => "./data/transactions_10000.csv",
  },
  after_each: fn _input -> reset_database() end,
)

And output the following:

But I think this may just be a personal preference, but I could see there a use case for comparing inputs against each other in this fashion for benchmarking performance for different applications.

Need a new formatter_option like auto_open or something that is either true or false. If it's false open_report should just not open the report in a browser.

Most importantly, please set the option to false in all tests so that my test don't open all the browser tabs all the time :D

When particularly fast functions are benchmarked for long, the sample size gets too large and plotly.js basically crashes or takes VERY LONG to render.

Current work arounds:

• reduce :time to indirectly reduce sample size
• remove the individual graphs from the resulting HTML by hand (they are most of the cause of this happening) and maybe also the box plot (is also fed raw run times)

Proposed solutions (might go into separate issues)

• print a warning when sample sizes exceed a certain threshold (like say 100_000)
• provide the option to split the individual graphs into separate files and then link to them
• render the box plot giving manually calculated quadrilles on the elixir side see this thread

Just small notes/links in the footer so that it can be traced back which benchee/benchee_html version created this HTML page, best also linking back to the projects github pages :)

There should be a way of graphing the execution time as a function of the size of the input, so that one can demonstrate the time complexity of the algorithm empirically in graphs like this:

The size of the input can't be determined automatically, but it could given by the user in the form of a tuple: {input_size, input_value} or some similar API. I understand that this might require a new top-level Benchee function, and that might be a change too big to implement.

But even if this change isn't possible, just plotting the execution time of all the inputs in the same graph would be a big improvement.

In my concrete example, I want to show that a an implementation of a programming language lexer (let's say, implementation 1) has complexity roughly linear on the size of the file while the alternative impelmentation (say, implementation 2) has supralinear complexity. I can see how, as the input size rows, implementation 2 gets slower and slower than implementation 1, but I can't compare the execution time of the implementation 1 with itself as the input grows.

When rendering multiple inputs benchee_html produces multiple output htmls, an overview page with the original file name linking to all of them would be very nice

E.g. to show units in milliseconds and others instead of the base units!

While I love the library don't really want to get them to their service :)

According to the documentation this is relatively easy to achieve or they make it seem so :)

talking about this link:

I'm thinking of including some benchmarks as part of the docs as a way of justifying some implementation choices and a s a teaching tool on how to extend my library.

You can access Benchee's HTML pages form the README (here). I'd like to change the template so that I could add other information, such as a title for the benchmark.