Epic - Move benchmark code out of metapackage about qiskit-metapackage

[jakelishman]

I'm not sure I actually see the repo move affecting any of those three points, but I do agree that it makes logical sense for the benchmarks to be in Terra's repo. An argument against that originally was that the benchmarks were for testing cross-repo performance, so they weren't localised to Terra, but now with the exception of a small dependency on qiskit-experiments (iirc, it's just for generating a couple of circuits and isn't involved in the benchmark), it's all Terra.

A benchmark suite run takes ~3 hours; it's not something we can ever run against PRs (so no CI failures from it, etc). Admittedly that'll fall drastically once we cut the massive waste of time that's the qiskit.assemble benchmarks, but even then it'll still be ~45 minutes. Almost all of the infrastructure is necessarily separate, too; asv is configured through its own asv.conf.json file, and the machine that actually runs the benchmarks is set up completely separately on the IBM cloud; multi-use public CI VMs aren't suitable for use in benchmarking because we can't control the loading of them.

[Eric-Arellano]

I'd bias towards the benchmarks living in Qiskit/qiskit-terra. You get monorepo benefits. Here, in particular:

1. More likely people will pay attention to the benchmarks. Multiple people are using Terra every day, so we will notice e.g. when CI fails
2. We have the option to have benchmarks run against the latest HEAD / PR. Meaning, we can see how a PR impacts performance, which gives us more granularity. (This is an option, not required; we can also choose to only run benchmarks on other events, like Git tags).
3. Less infrastructure, like duplicating Tox files etc.