Comments (17)
Hello everyone! I'm bheisler, and with @japaric's permission I've volunteered to take over maintenance of Criterion-rs. At least to start with, I'll probably be following the plan posted here relatively closely. I hope to wrap up release 0.1 at some point in the next few weeks.
from criterion.rs.
I'd be happy to invest some time in a PR or two if it would be of any use.
Anything in particular you'd like help with?
from criterion.rs.
@bheisler hello, I have a quick question about criterion-rs future: are the overall trend graphs planned?
It would be nice to have a graph showing how the performance evolved throughout development time, not just compared to the previous build. (or am I missing something that is already there?)
from criterion.rs.
One advantage of the long-term performance graph is that it allows one to easily see beyond the noise. Yes, you can just sample longer to reduce the noise, but that isn't always feasible. Also, it allows you to see whether you have fixed a regression. The short tweak optimization cycle works as long as you're only optimizing, but once you intersperse new functionality it becomes trickier.
from criterion.rs.
@faern will, hopefully this week, send us some PR to get us close to compiling on a recent nightly. 👍
from criterion.rs.
Next actions:
- publish criterion dependencies
- write great docs
- publish criterion 0.1.0
@faern if you can land the multiple functions benchmarking method before 0.1.0, excellent. If not it can wait until 0.1.1.
from criterion.rs.
@japaric I did some rebasing of that on top of your latest merge this morning. I will see when I can get it ready. Hopefully within a day.
I have another suggestion. How about we run rustfmt on the entire crate before 0.1.0? My Rust editor runs that automatically on save so I have a hard time to not commit unrelated changes :) Also good to comply to the standards.
from criterion.rs.
@japaric I did some rebasing of that on top of your latest merge this morning. I will see when I can get it ready. Hopefully within a day.
👍
I have another suggestion. How about we run rustfmt on the entire crate before 0.1.0?
Sounds good to me. But last time I ran cargo fmt
the source files related to plotting looked really bad; they were heavily indented IIRC.
Something I forgot to mention. What do you think about changing the signature of methods like Criterion::warm_up_time
to accept a Duration
instead of a u64
that represents nanoseconds? We are already using Duration
internally. It would be the last breaking change before 0.1, or we could hold wait until 0.2 to land it.
I'll be publishing the criterion dependencies today when I get free time.
from criterion.rs.
PR created: #64
Except the benchmarking over multiple functions it adds another iter method. Tell me if you rather skip that one..
Yes, we should be consistent and use Duration everywhere. I think we can cram that into this release!
from criterion.rs.
cargo fmt
made some plotting stuff weird yes, but that was because the code is a little bit strange from the start :P The correct solution is still to format it, but we might want to save that until it can be refactored into not looking like this.
(Does not stop us from committing formatting on the other files now)
from criterion.rs.
PR created: #64
Will leave comments over there.
Yes, we should be consistent and use Duration everywhere. I think we can cram that into this release!
👍
re: cargo fmt. We can put a #![rustfmt_skip]
on the plotting modules and cargo fmt
the rest. Let's do that after the other PRs land to avoid rebasing them.
from criterion.rs.
I did put #![rustfmt_skip]
on the plot files and ran cargo fmt
. I didn't want to submit a PR since if it was my project I would want to do such large change myself, but feel free to merge in my master branch if you want to.
What's more left now? I see you did some commits on testing, but didn't check those boxes yet.
from criterion.rs.
The CI stuff is done. I forgot to check those boxes. Dependencies have been released to crates.io as well. Only docs are missing. I'm trying to figure out the simplest way to run benchmarks in a cargo project.
from criterion.rs.
Hi!
Recent adopter of criterion here -- nice work!
Some comments on things that would be helpful for our (@jonhoo's and my) use case:
Was the performance regression detection reliable, or did you get false positives?
We would prefer if this the comparison step could be disabled as part of the configuration, or at least if it avoided failing hard. The experiments we're running are often somewhat noisy, and we run multiple benchmarks in sequence, so it's annoying to fail because of a false positive regression on one of them.
What functionality do you think is missing?
One additional feature (on top of being able to disable the regression testing) that would really help us is the ability to invoke a one-off setup routine, which is only executed once for all benchmarks.
In our use case, we create a benchmark database and populate it with some basic data, but this can take a while. Having to do it once per benchmark seems wasteful, so we're working around this by having multiple Criterion::default().bench_function(...)
invocations in a single test routine. However, this is not ideal, because it's now no longer possible for us to run a single benchmark in isolation.
from criterion.rs.
How is it going?
from criterion.rs.
Hmm. I haven't really given that one much thought, to be honest. My workflow when benchmarking is more based on the Profile/Tweak/Benchmark loop rather than tracking performance over days or weeks. I'm not sure what one would want in a long-running trend graph or if Criterion.rs is the right place to generate one. On the other hand, I know that some people are using Criterion.rs for that purpose already, so it might make sense for that workflow to be more directly supported.
Can you describe in more detail what you're looking for?
from criterion.rs.
Okay, mine is structured alike, more or less, but I am constantly running into this scenario:
- Run benchmark
- Tweak
- Run benchmark, see that performance has regressed
- Now I need to check out initial version and rerun benchmark to be able to compare the next tweak
As benchmarks are quite complex and take much time, this is not very comfortable.
And lastly, having some squished down version of the report in git tree just for sake of seing overall performance trend would be cool, but that's a bit another story.
I'd be happy to see something similar to graphs that Jenkins-Performance has in their Web UI (https://imgur.com/wavTvXf)
Graph legend shown in the picture is just an example.
from criterion.rs.
Related Issues (20)
- --profile-time doesn't work HOT 1
- panic in drop
- Custom Iter with async setup and non-async benched value
- "cargo bench -- <filter>" does not actually filter the benchmarks
- Can't call a function for the benchmark from the crate inside a Rust workspace HOT 1
- Quickstart instructions still use criterion 0.4 HOT 1
- Benchmarking With One function result as Input
- Help needed? HOT 5
- Found argument '--bench' which wasn't expected
- Benchmarks not running when specifying benchmark name
- Skip setup for filtered out benchmarks
- Allow running benchmarks as tests under Miri
- Segfault in code that passes valgrind/asan HOT 1
- Save baseline does not work in 0.5 HOT 4
- Can clap be made optional? HOT 1
- Can I limit number of threads to use in benchmark? HOT 1
- Enable Github "Discussions"
- "How Should I Benchmark Small Functions?" section in book is nonsense HOT 1
- script / plugin to easily interpret benchmark with OpenAI / LLM
- Does criterion store the binary some for later investigations with valgrind or something else?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from criterion.rs.