Comments (9)
A first step toward this is basic http trigger benchmarks merged in #185. Those are run nightly by the spin-benchmarks repo and published to https://fermyon.github.io/spin-benchmarks/criterion/reports/
from spin.
@vdice
maybe a Slack bot approach that shows the results and/or diff week-over-week could be neat, kinda like the comms website interaction diff messages we get
The cargo-criterion
tool can dump events to stdout that would be perfect for this:
{"reason":"benchmark-complete","id":"startup/spin-executor", [...CLIP...], "change":"NoChange"}
from spin.
Using a Spin build based on #76, with the following Spin configuration (running components that use the Wagi executor, so this is as close as possible to running Wagi):
name = "fermyon.com"
version = "1.0.0"
description = "A simple application showing both Spin and Wagi components."
authors = [ "Radu Matei <[email protected]>" ]
trigger = {type = "http", base = "/" }
[[component]]
source = "bartholomew.wasm"
id = "bartholomew"
files = [ "content/**/*" , "templates/*", "scripts/*", "config/*"]
[component.trigger]
route = "/..."
executor = "wagi"
[[component]]
source = "fileserver.gr.wasm"
id = "fileserver"
files = ["static/**/*"]
environment = { PATH_PREFIX = "static/" }
[component.trigger]
route = "/static/..."
executor = "wagi"
This is the result of running a load test locally with Spin:
# application running using Spin, with components that use the Wagi executor
➜ bombardier -c 100 -n 10000 http://localhost:3000/blog/2022-02-08-hello-world
Bombarding http://localhost:3000/blog/2022-02-08-hello-world with 10000 request(s) using 100 connection(s)
100.00% 371/s 26s
Done!
Statistics Avg Stdev Max
Reqs/sec 374.55 104.01 912.78
Latency 266.33ms 107.40ms 0.97s
HTTP codes:
1xx - 0, 2xx - 10000, 3xx - 0, 4xx - 0, 5xx - 0
others - 0
Throughput: 6.81MB/s
And the same application, this time run with Wagi:
# application running using Wagi
➜ bombardier -c 100 -n 10000 http://localhost:3000/blog/2022-02-08-hello-world
Bombarding http://localhost:3000/blog/2022-02-08-hello-world with 10000 request(s) using 100 connection(s)
100.00% 358/s 27s
Done!
Statistics Avg Stdev Max
Reqs/sec 361.79 132.12 844.97
Latency 276.70ms 111.16ms 800.17ms
HTTP codes:
1xx - 0, 2xx - 10000, 3xx - 0, 4xx - 0, 5xx - 0
others - 0
Throughput: 6.56MB/s
For reference, updating the application to components that use the Spin HTTP executor (as opposed to the Wagi executor):
[[component]]
source = "spin-bartholomew.wasm"
id = "bartholomew"
files = [ "content/**/*" , "templates/*", "scripts/*", "config/*"]
[component.trigger]
route = "/..."
executor = "spin"
[[component]]
source = "spin_static_fs.wasm"
id = "fileserver"
files = ["static/**/*"]
environment = { PATH_PREFIX = "static/" }
[component.trigger]
route = "/static/..."
executor = "spin"
The performance is slightly better again:
# application running using Spin, with Spin HTTP components
➜ bombardier -c 100 -n 10000 http://localhost:3000/blog/2022-02-08-hello-world
Bombarding http://localhost:3000/blog/2022-02-08-hello-world with 10000 request(s) using 100 connection(s)
100.00% 377/s 26s
Done!
Statistics Avg Stdev Max
Reqs/sec 381.55 115.34 897.43
Latency 261.63ms 102.31ms 782.53ms
HTTP codes:
1xx - 0, 2xx - 10000, 3xx - 0, 4xx - 0, 5xx - 0
others - 0
Throughput: 6.92MB/s
These results are on my machine, for 100 concurrent connections.
from spin.
Running the tests in the same order as before, this time with a single concurrent connection, the results are almost identical regardless of Wagi vs. Spin (regardless of the executor and component types used): ~1MB/s, ~17ms average latency:
# application running using Spin, with components that use the Wagi executor
➜ bombardier -c 1 -n 1000 http://localhost:3000/blog/2022-02-08-hello-world
Bombarding http://localhost:3000/blog/2022-02-08-hello-world with 1000 request(s) using 1 connection(s)
100.00% 57/s 17s
Done!
Statistics Avg Stdev Max
Reqs/sec 58.43 21.06 289.53
Latency 17.24ms 1.39ms 47.11ms
HTTP codes:
1xx - 0, 2xx - 1000, 3xx - 0, 4xx - 0, 5xx - 0
others - 0
Throughput: 1.06MB/s
# application running using Wagi
➜ bombardier -c 1 -n 1000 http://localhost:3000/blog/2022-02-08-hello-world
Bombarding http://localhost:3000/blog/2022-02-08-hello-world with 1000 request(s) using 1 connection(s)
100.00% 57/s 17s
Done!
Statistics Avg Stdev Max
Reqs/sec 57.80 19.25 105.59
Latency 17.37ms 1.08ms 39.69ms
HTTP codes:
1xx - 0, 2xx - 1000, 3xx - 0, 4xx - 0, 5xx - 0
others - 0
Throughput: 1.05MB/s
# application running using Spin, with Spin HTTP components
➜ bombardier -c 1 -n 1000 http://localhost:3000/blog/2022-02-08-hello-world
Bombarding http://localhost:3000/blog/2022-02-08-hello-world with 1000 request(s) using 1 connection(s)
100.00% 56/s 17s
Done!
Statistics Avg Stdev Max
Reqs/sec 56.78 19.86 314.47
Latency 17.74ms 1.49ms 37.87ms
HTTP codes:
1xx - 0, 2xx - 1000, 3xx - 0, 4xx - 0, 5xx - 0
others - 0
Throughput: 1.02MB/s
Of course, as we've seen with 100 concurrent connections, the time can add up to significant latencies, but for an application that makes heavy IO use, the results seem to be almost identical.
from spin.
These test results are purely informative, as we need to run them in a reproducible manner, and my machine isn't currently the most reliable at that..
from spin.
A good resolution for this issue would be a suite of tests that we run as part of CI.
from spin.
I'm going to put this on hold until I get my desktop delivered.
from spin.
Perfect reason to build a Spin component that writes to Slack.
from spin.
The benchmarks we have been already been publishing for a while now address this.
See https://fermyon.github.io/spin-benchmarks/criterion/reports.
from spin.
Related Issues (20)
- inconsistencies when making outbound http requests HOT 1
- Custom trigger with other WASI worlds requires trait implementation for `spin_core::Data<T>` HOT 2
- SQLITE error after update from 2.5.1 to 2.6.0 HOT 3
- spin can not login docker private registry while docker login success HOT 5
- Follow OCI container image annotation standards when generating containers HOT 2
- Generate and sign SBOM through spin HOT 5
- Maintain a CHANGELOG.md file? HOT 2
- Spin (Re)Factors tracking issue
- Add native support for Golang, Python, and JavaScript Azure SDKs HOT 3
- Add native support for Golang and Python AWS SDKs
- Registry metadata support in application manifest HOT 3
- add MAINTAINERS.md file to each repository in Spin project HOT 1
- `spin update` An easier way to update to latest spin version HOT 1
- Endless re-compile in `spin watch` for Spin Apps that consist of multiple components HOT 1
- Hide old plugin versions in `spin plugins list`
- Spin-telemetry: consider eliminating the use of `set_global_default` HOT 4
- inconsistent preopened dirs between wasmtime and spin HOT 8
- Abstract pg client type in factor-outbound-pg HOT 2
- Need CORS config for HTTP trigger HOT 1
- Spin Docker image HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spin.