Incorrect CI PR benchmark comparison about package-benchmark

[hassila]

Thanks for the report, unfortunately away for a couple of weeks and won't have time to check on this until then, sorry for delay.

[vsarunas]

Hi @ptoffy,

Yes I believe it is due to running on GitHub's CI infrastructure and "noise neighbours" for the most part.

Can verify locally by creating 3 runs on where development was done:

swift package --disable-sandbox benchmark baseline update run1 --target Signing
swift package --disable-sandbox benchmark baseline update run2 --target Signing
swift package --disable-sandbox benchmark baseline update run3 --target Signing

And comparing them:

swift package benchmark baseline compare run1 run2
swift package benchmark baseline compare run1 run3
swift package benchmark baseline compare run2 run3

The metrics on macOS have less variations than on Linux for me it seems; but they're a lot less than the linked workflow result.

You may find that increasing the number of samples by increasing the test duration will also create more stability:

Benchmark.defaultConfiguration.maxDuration = .seconds(10)

[hassila]

Hi @ptoffy , yeah @vsarunas is correct, running time/throughput measurements on GitHub infrastructure isn't a good proposition (which is why we recommend a standalone runner for such metrics if you want them in CI) - the environment is simply not controlled enough. We are considering adding e.g instruction count also (in #242) which could be a better proposition (if we can get those stats on the runner, remains to be seen..) as it then is a proxy of the runtime in practice. Other metrics such as malloc count and sys calls works fine on GitHub ci though. Closing this for now, feel free to open up follow up issues as needed.