`--benchmark-samples` does not seem to work as expected about catch2

[horenmar]

You can't do this with Catch2, because the benchmark will always run the function at least twice. Once for the sample you ask for, but before that happens it will run at least once for the warmup. --benchmark-warmup-time sets the minimal time spent in warmup, but even setting it to 0 does not skip it.

--benchmark-resamples 1 does not do anything when combined with --benchmark-no-analysis, as these change how (and whether) the bootstrapping analysis from the taken samples happen.

[Megaloblastt]

And wouldn't it be relevant to specifically test if --benchmark-warmup-time 0 was set, and if yes, simply skip the warmup?

[horenmar]

That could be done, but that is a very specific, and frankly weird, use case. Catch2 primarily targets microbenchmarking and attempts to provide high quality statistical output. Doing only one sample, especially without warm up, can't provide that, and the only advantage of using BENCHMARK over auto t1 = std::chrono::steady_clock::now(); ... ; auto t2 = is that the output will go through the reporter.

Furthermore, if this is a function that can only be called once in program's lifetime (e.g. libfoo_init), trying to call it inside a benchmark/test is generally a bad idea, as it makes it easy to skip calling it or call it multiple times through test/benchmark selection. If, on the other hand, you have a function that can only be called once per set-up (e.g. member function on a class that steals internal state with a check that it was not called multiple times), you should instead provide sufficient setup through the advanced benchmarking facilities (this is also used e.g. for benchmarks of destructive algorithms).

[Megaloblastt]

Without entering the details of my use case, it consists on few steps, running in a given order, each of them being internally protected against replay attacks. And I want to benchmark each of the steps, and the benchmark results to be included in the report (thus go through the reporter).
If there is a way to do so (maybe using BENCHMARK_ADVANCED, but I don't see how), please let me know.

But most important, independently of my use case, I would normally expect --benchmark-warmup-time 0 to skip the warmup. Or maybe accept a negative number if, internally, the code does something like

while (current_warmup_time <= max_warmup_time) {
// do something
}

so that --benchmark-warmup-time -1 would actually skip it.