restarting development and criterion's future about criterion.rs

[bheisler]

Hello everyone! I'm bheisler, and with @japaric's permission I've volunteered to take over maintenance of Criterion-rs. At least to start with, I'll probably be following the plan posted here relatively closely. I hope to wrap up release 0.1 at some point in the next few weeks.

[jonhoo]

I'd be happy to invest some time in a PR or two if it would be of any use.
Anything in particular you'd like help with?

[mersinvald]

@bheisler hello, I have a quick question about criterion-rs future: are the overall trend graphs planned?
It would be nice to have a graph showing how the performance evolved throughout development time, not just compared to the previous build. (or am I missing something that is already there?)

[droundy]

One advantage of the long-term performance graph is that it allows one to easily see beyond the noise. Yes, you can just sample longer to reduce the noise, but that isn't always feasible. Also, it allows you to see whether you have fixed a regression. The short tweak optimization cycle works as long as you're only optimizing, but once you intersperse new functionality it becomes trickier.

[japaric]

@faern will, hopefully this week, send us some PR to get us close to compiling on a recent nightly. 👍

[japaric]

Next actions:

• publish criterion dependencies
• write great docs
• publish criterion 0.1.0

@faern if you can land the multiple functions benchmarking method before 0.1.0, excellent. If not it can wait until 0.1.1.

[faern]

@japaric I did some rebasing of that on top of your latest merge this morning. I will see when I can get it ready. Hopefully within a day.

I have another suggestion. How about we run rustfmt on the entire crate before 0.1.0? My Rust editor runs that automatically on save so I have a hard time to not commit unrelated changes :) Also good to comply to the standards.

[japaric]

@japaric I did some rebasing of that on top of your latest merge this morning. I will see when I can get it ready. Hopefully within a day.

👍

I have another suggestion. How about we run rustfmt on the entire crate before 0.1.0?

Sounds good to me. But last time I ran cargo fmt the source files related to plotting looked really bad; they were heavily indented IIRC.

Something I forgot to mention. What do you think about changing the signature of methods like Criterion::warm_up_time to accept a Duration instead of a u64 that represents nanoseconds? We are already using Duration internally. It would be the last breaking change before 0.1, or we could hold wait until 0.2 to land it.

I'll be publishing the criterion dependencies today when I get free time.

[faern]

PR created: #64
Except the benchmarking over multiple functions it adds another iter method. Tell me if you rather skip that one..

Yes, we should be consistent and use Duration everywhere. I think we can cram that into this release!

[faern]

cargo fmt made some plotting stuff weird yes, but that was because the code is a little bit strange from the start :P The correct solution is still to format it, but we might want to save that until it can be refactored into not looking like this.

(Does not stop us from committing formatting on the other files now)

[japaric]

PR created: #64

Will leave comments over there.

Yes, we should be consistent and use Duration everywhere. I think we can cram that into this release!

👍

re: cargo fmt. We can put a #![rustfmt_skip] on the plotting modules and cargo fmt the rest. Let's do that after the other PRs land to avoid rebasing them.

[faern]

I did put #![rustfmt_skip] on the plot files and ran cargo fmt. I didn't want to submit a PR since if it was my project I would want to do such large change myself, but feel free to merge in my master branch if you want to.

What's more left now? I see you did some commits on testing, but didn't check those boxes yet.

[japaric]

The CI stuff is done. I forgot to check those boxes. Dependencies have been released to crates.io as well. Only docs are missing. I'm trying to figure out the simplest way to run benchmarks in a cargo project.

[ms705]

Hi!

Recent adopter of criterion here -- nice work!

Some comments on things that would be helpful for our (@jonhoo's and my) use case:

Was the performance regression detection reliable, or did you get false positives?

We would prefer if this the comparison step could be disabled as part of the configuration, or at least if it avoided failing hard. The experiments we're running are often somewhat noisy, and we run multiple benchmarks in sequence, so it's annoying to fail because of a false positive regression on one of them.

What functionality do you think is missing?

One additional feature (on top of being able to disable the regression testing) that would really help us is the ability to invoke a one-off setup routine, which is only executed once for all benchmarks.

In our use case, we create a benchmark database and populate it with some basic data, but this can take a while. Having to do it once per benchmark seems wasteful, so we're working around this by having multiple Criterion::default().bench_function(...) invocations in a single test routine. However, this is not ideal, because it's now no longer possible for us to run a single benchmark in isolation.

[hauleth]

How is it going?

[bheisler]

Hmm. I haven't really given that one much thought, to be honest. My workflow when benchmarking is more based on the Profile/Tweak/Benchmark loop rather than tracking performance over days or weeks. I'm not sure what one would want in a long-running trend graph or if Criterion.rs is the right place to generate one. On the other hand, I know that some people are using Criterion.rs for that purpose already, so it might make sense for that workflow to be more directly supported.

Can you describe in more detail what you're looking for?

[mersinvald]

Okay, mine is structured alike, more or less, but I am constantly running into this scenario:

1. Run benchmark
2. Tweak
3. Run benchmark, see that performance has regressed
4. Now I need to check out initial version and rerun benchmark to be able to compare the next tweak
   As benchmarks are quite complex and take much time, this is not very comfortable.

And lastly, having some squished down version of the report in git tree just for sake of seing overall performance trend would be cool, but that's a bit another story.

I'd be happy to see something similar to graphs that Jenkins-Performance has in their Web UI (https://imgur.com/wavTvXf)
Graph legend shown in the picture is just an example.