Add benchmarking to Travis gate job? about sanic

[brian-bates]

We could probably get the best of both worlds (free shared hardware + short runs) by normalizing the run with a fixed control. For example, say that our control takes 5 seconds to run on ideal hardware. Suppose we then run the benchmark on shared hardware alongside the control and it takes 10 seconds. We can then normalize the benchmark by multiplying by current_execution_time / execution_time_on_ideal_hardware or 10 seconds / 5 seconds. What do you guys think?

[ludovic-gasc]

I'm fully agree for this idea, but, the main issue is travis-ci: Based on my experience to validate techempower framework benchmarks, I never have two times the same timing between each run, even if I change nothing, because the tests run in a container, executed in a VM: You can't predict the load average of others containers and VMs.

To have a predictable test, you need to use hardware without virtualization and follow the good practices from @Haypo with perf: http://perf.readthedocs.io/en/latest/
Moreover, you need to test always on the same hardware, to enable you to compare between two executions.

Is it a travis-ci like service proposes a service like that ?

[channelcat]

Agreed, we need a standard way of doing this. My tests on AWS are not stable and involve me running them over a long period of time until I get consistent results. Even the m4 instances which aren't supposed to burst fluctuate in performance about 1-3k req/sec.

It seems to me like we have 2 options:

1. Dedicated hardware - Currently AWS charges $2/hr + fees for dedicated hardware. Unless we can come up with a cheaper solution, I think we'd have to resort to running it less often. Maybe once a week or before releases.
2. Shared hardware + long-running tests - Since we're just looking for relative difference in speed between 2 branches, we could just run the tests and compare averages/best run/top percentile results (whichever is most telling). My guess is we'd have to run tests for about 5 minutes to get some consistent results. This may not be that bad, but would definitely be annoying to wait for if the committer is relying on travis to check if if their pull request passed.

I'm inclined to go with option #2, unless there's a solution that lets us spin up dedicated hardware cheaply. What do you guys think?

[ludovic-gasc]

option 2 seems to be more realistic.
Nevertheless, we might have a new business model just here to launch a new company ;-)
More seriously, it might be a question I could raise to PSF, because they have dedicated hardware to test and benchmark CPython. I'll try to take the temperature.

[katlol]

@channelcat you can use Scaleway.com for cheap dedicated hardware; they have €3/month ARM dedicated servers (basically a hosted RPi alternative), or the cheap dedicated servers offered by Kimsufi.com / Online.net.

[jpiasetz]

profile (and cProfile) might be useful. Could look at changes in number of calls or percentage of time in sanic files relative to overall.

[seemethere]

@brian-bates I definitely think that, that idea maybe mixed with @jpiasetz idea could really be helpful for completing this.

[sjsadowski]

Closing due to no follow-up in 2 years. There are discussions on benchmarking in the current community forums (https://community.sanicframework.org)