Monitor performance in CI about spin

[lann]

A first step toward this is basic http trigger benchmarks merged in #185. Those are run nightly by the spin-benchmarks repo and published to https://fermyon.github.io/spin-benchmarks/criterion/reports/

[lann]

@vdice
maybe a Slack bot approach that shows the results and/or diff week-over-week could be neat, kinda like the comms website interaction diff messages we get

The cargo-criterion tool can dump events to stdout that would be perfect for this:

{"reason":"benchmark-complete","id":"startup/spin-executor", [...CLIP...], "change":"NoChange"}

[radu-matei]

Using a Spin build based on #76, with the following Spin configuration (running components that use the Wagi executor, so this is as close as possible to running Wagi):

name = "fermyon.com"
version = "1.0.0"
description = "A simple application showing both Spin and Wagi components."
authors = [ "Radu Matei <[email protected]>" ]
trigger = {type = "http", base = "/" }


[[component]]
source = "bartholomew.wasm"
id = "bartholomew"
files = [ "content/**/*" , "templates/*", "scripts/*", "config/*"]
[component.trigger]
route = "/..."
executor = "wagi"

[[component]]
source = "fileserver.gr.wasm"
id = "fileserver"
files = ["static/**/*"]
environment = { PATH_PREFIX = "static/" }
[component.trigger]
route = "/static/..."
executor = "wagi"

This is the result of running a load test locally with Spin:

# application running using Spin, with components that use the Wagi executor
 ➜ bombardier -c 100 -n 10000 http://localhost:3000/blog/2022-02-08-hello-world
Bombarding http://localhost:3000/blog/2022-02-08-hello-world with 10000 request(s) using 100 connection(s)
 100.00% 371/s 26s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec       374.55     104.01     912.78
  Latency      266.33ms   107.40ms      0.97s
  HTTP codes:
    1xx - 0, 2xx - 10000, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:     6.81MB/s

And the same application, this time run with Wagi:

# application running using Wagi
➜ bombardier -c 100 -n 10000 http://localhost:3000/blog/2022-02-08-hello-world
Bombarding http://localhost:3000/blog/2022-02-08-hello-world with 10000 request(s) using 100 connection(s)
 100.00% 358/s 27s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec       361.79     132.12     844.97
  Latency      276.70ms   111.16ms   800.17ms
  HTTP codes:
    1xx - 0, 2xx - 10000, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:     6.56MB/s

For reference, updating the application to components that use the Spin HTTP executor (as opposed to the Wagi executor):

[[component]]
source = "spin-bartholomew.wasm"
id = "bartholomew"
files = [ "content/**/*" , "templates/*", "scripts/*", "config/*"]
[component.trigger]
route = "/..."
executor = "spin"

[[component]]
source = "spin_static_fs.wasm"
id = "fileserver"
files = ["static/**/*"]
environment = { PATH_PREFIX = "static/" }
[component.trigger]
route = "/static/..."
executor = "spin"

The performance is slightly better again:

# application running using Spin, with Spin HTTP components
➜ bombardier -c 100 -n 10000 http://localhost:3000/blog/2022-02-08-hello-world
Bombarding http://localhost:3000/blog/2022-02-08-hello-world with 10000 request(s) using 100 connection(s)
100.00% 377/s 26s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec       381.55     115.34     897.43
  Latency      261.63ms   102.31ms   782.53ms
  HTTP codes:
    1xx - 0, 2xx - 10000, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:     6.92MB/s

These results are on my machine, for 100 concurrent connections.

[radu-matei]

Running the tests in the same order as before, this time with a single concurrent connection, the results are almost identical regardless of Wagi vs. Spin (regardless of the executor and component types used): ~1MB/s, ~17ms average latency:

# application running using Spin, with components that use the Wagi executor
➜ bombardier -c 1 -n 1000 http://localhost:3000/blog/2022-02-08-hello-world
Bombarding http://localhost:3000/blog/2022-02-08-hello-world with 1000 request(s) using 1 connection(s)
100.00% 57/s 17s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec        58.43      21.06     289.53
  Latency       17.24ms     1.39ms    47.11ms
  HTTP codes:
    1xx - 0, 2xx - 1000, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:     1.06MB/s

# application running using Wagi
➜ bombardier -c 1 -n 1000 http://localhost:3000/blog/2022-02-08-hello-world
Bombarding http://localhost:3000/blog/2022-02-08-hello-world with 1000 request(s) using 1 connection(s)
100.00% 57/s 17s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec        57.80      19.25     105.59
  Latency       17.37ms     1.08ms    39.69ms
  HTTP codes:
    1xx - 0, 2xx - 1000, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:     1.05MB/s

# application running using Spin, with Spin HTTP components
➜ bombardier -c 1 -n 1000 http://localhost:3000/blog/2022-02-08-hello-world
Bombarding http://localhost:3000/blog/2022-02-08-hello-world with 1000 request(s) using 1 connection(s)

100.00% 56/s 17s
Done!
Statistics        Avg      Stdev        Max
  Reqs/sec        56.78      19.86     314.47
  Latency       17.74ms     1.49ms    37.87ms
  HTTP codes:
    1xx - 0, 2xx - 1000, 3xx - 0, 4xx - 0, 5xx - 0
    others - 0
  Throughput:     1.02MB/s

Of course, as we've seen with 100 concurrent connections, the time can add up to significant latencies, but for an application that makes heavy IO use, the results seem to be almost identical.

[radu-matei]

These test results are purely informative, as we need to run them in a reproducible manner, and my machine isn't currently the most reliable at that..

[radu-matei]

A good resolution for this issue would be a suite of tests that we run as part of CI.

[lann]

I'm going to put this on hold until I get my desktop delivered.

[radu-matei]

Perfect reason to build a Spin component that writes to Slack.

[radu-matei]

The benchmarks we have been already been publishing for a while now address this.
See https://fermyon.github.io/spin-benchmarks/criterion/reports.