rousan / rust-web-frameworks-benchmark

A hello world benchmark for the available Rust Web Frameworks: hyper vs gotham vs actix-web vs warp vs rocket

Benchmarking Rust Web Frameworks: hyper vs gotham vs actix-web vs warp vs rocket

I was building a very performance sensitive API service using Rust and I was trying to find out which http framework I should use as there a lot of frameworks out there for Rust. So, I just shortlisted some of them and I ran a hello world route for every frameworks and its result is very interesting.

Benchmarking Tool

I used wrk to test those http servers:

$ wrk --latency -t4 -c200 -d8s http://127.0.0.1:8081

Results

I ran all those tests on:

Intel Core i7 CPU @ 3.20GHz × 12, 16GB of RAM

Before run, I created a release build:

$ cargo build --release

Framework 1: hyper (https://github.com/hyperium/hyper)

Requests/sec: 520718.02 and Latency: 368.31us

Running 8s test @ http://127.0.0.1:8082
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   368.31us  327.11us   8.11ms   92.48%
    Req/Sec   131.79k     6.77k  147.18k    77.50%
  Latency Distribution
     50%  295.00us
     75%  390.00us
     90%  595.00us
     99%    1.64ms
  4196014 requests in 8.06s, 352.14MB read
Requests/sec: 520718.02
Transfer/sec:     43.70MB

Framework 2: actix-web (https://github.com/actix/actix-web)

Requests/sec: 562315.64 and Latency: 365.86us

Running 8s test @ http://127.0.0.1:8080
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   365.86us  659.57us  18.39ms   94.46%
    Req/Sec   142.44k     7.76k  150.25k    92.19%
  Latency Distribution
     50%  238.00us
     75%  328.00us
     90%  451.00us
     99%    3.35ms
  4534091 requests in 8.06s, 557.80MB read
Requests/sec: 562315.64
Transfer/sec:     69.18MB

Framework 3: warp (https://github.com/seanmonstar/warp)

Requests/sec: 506776.84 and Latency: 393.21us

Running 8s test @ http://127.0.0.1:3030
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   393.21us  487.15us  21.55ms   95.16%
    Req/Sec   128.68k     7.57k  146.42k    79.06%
  Latency Distribution
     50%  305.00us
     75%  410.00us
     90%  620.00us
     99%    1.80ms
  4098580 requests in 8.09s, 320.51MB read
  Non-2xx or 3xx responses: 4098580
Requests/sec: 506776.84
Transfer/sec:     39.63MB

Framework 4: gotham (https://github.com/gotham-rs/gotham)

Requests/sec: 300483.70 and Latency: 677.14us

Running 8s test @ http://127.0.0.1:8081
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   677.14us  513.16us  15.78ms   87.00%
    Req/Sec    76.36k     6.20k   88.57k    74.38%
  Latency Distribution
     50%  554.00us
     75%  816.00us
     90%    1.19ms
     99%    2.66ms
  2431445 requests in 8.09s, 384.92MB read
Requests/sec: 300483.70
Transfer/sec:     47.57MB

Framework 5: Rocket (https://github.com/SergioBenitez/Rocket)

Requests/sec: 115231.95 and Latency: 412.75us

Running 8s test @ http://127.0.0.1:8000
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   412.75us    1.35ms 204.10ms   99.83%
    Req/Sec    29.20k     1.60k   31.79k    85.94%
  Latency Distribution
     50%  312.00us
     75%  551.00us
     90%  737.00us
     99%    0.99ms
  929792 requests in 8.07s, 129.46MB read
  Socket errors: connect 0, read 929766, write 0, timeout 0
Requests/sec: 115231.95
Transfer/sec:     16.04MB

Caveat

In order to run a fair comparison a suitable environment variable should be set while running, ROCKET_ENV=prod ./rocket. Indeed, by default is set to dev, and it is spending a lot of time logging everything on the console.

Conclusion

hyper -> Perfect for an app which requires more controls on http and performance. It provides async with tokio.rs.

gotham -> A high-level version of hyper.

actix-web -> Provides a very high-level API and injects some boilerplate code as well as runtime. It provides async with tokio.rs.

warp -> A very high level version of hyper and easy to use and it requires a few lines of codes to start a server. It provides async with tokio.rs.

rocket -> High level, actively developed and quite used. Nevertheless it has not the same performances as the others.

rust-web-frameworks-benchmark's Issues

Add tide benchmark?

It would be good to add benchmark for tide here as well, I would like to see how well it compares with rocket. https://github.com/http-rs/tide

Warp benchmark error?

Noticed that warp was the only server with Non 2xx and 3xx responses. And not just 1 but all of them. This might have impacted benchmarking

Add benchmarks for Routerify (https://github.com/routerify/routerify)

show result sort by Latency

show result sort by latency in Conclusion section

Performance issue

Hello, first of all thank you for this benchmark comparison.

I have a question regardless to the performance I got on my machine:
- I ran the Actix server in release mode
- I'm using same wrk command
- My machine is MacBook Pro, 2.3 GHz 8 Core, Inter Core I9, 16GB DDR4

My benchmarking shows 130k/s and yours 562k/s

I'm wondering what should cause such a gap. Which OS are you using?

Thanks

Benchmark framework by let them doing the same work.

There are some obvious problem with the bench.

1. actix-web and warp return a String while others return a &'static str. This means they have additional allocation for generating payload while others don't.

2. hyper being low level does not set content-type header. And I believe all other tests do set it to "plain/text" or "text/plain; charset=utf-8". So it should set one to be on par.

3. warp by the test method mentioned in README.md does not generate meaningful response because it can not find the route you defined. This makes it return a 404 response that has no payload so it bypass the response body entirely. This is mentioned in issue #2.

Rocket v5 (async) bencmark

I try to run the benchmark against the master branch of https://github.com/SergioBenitez/Rocket (v5).
To see if there is any speed improvement. Because it's already async.

I run the benchmark to warp, to see my machine capability.

warp

❯ cd warp
❯ cargo build --release
❯ ./target/release/warp

❯ wrk --latency -t4 -c200 -d8s http://127.0.0.1:3030
Running 8s test @ http://127.0.0.1:3030
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   817.75us    1.11ms  17.93ms   93.77%
    Req/Sec    72.54k     7.78k   95.50k    68.12%
  Latency Distribution
     50%  519.00us
     75%  844.00us
     90%    1.41ms
     99%    6.21ms
  2314469 requests in 8.09s, 180.99MB read
  Non-2xx or 3xx responses: 2314469
Requests/sec: 286126.89
Transfer/sec:     22.38MB

❯ wrk --latency -t4 -c200 -d8s http://127.0.0.1:3030
Running 8s test @ http://127.0.0.1:3030
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.86ms    1.03ms  16.82ms   93.42%
    Req/Sec    65.72k     6.99k   83.28k    63.75%
  Latency Distribution
     50%  580.00us
     75%    0.93ms
     90%    1.48ms
     99%    5.55ms
  2097890 requests in 8.09s, 164.06MB read
  Non-2xx or 3xx responses: 2097890
Requests/sec: 259381.43
Transfer/sec:     20.28MB

❯ wrk --latency -t4 -c200 -d8s http://127.0.0.1:3030
Running 8s test @ http://127.0.0.1:3030
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.93ms    1.10ms  17.71ms   93.35%
    Req/Sec    60.40k     7.81k   82.04k    65.00%
  Latency Distribution
     50%  632.00us
     75%    1.01ms
     90%    1.62ms
     99%    6.10ms
  1926600 requests in 8.09s, 150.66MB read
  Non-2xx or 3xx responses: 1926600
Requests/sec: 238068.17
Transfer/sec:     18.62MB

Rocket v4

❯ cd rocket
❯ cargo +nightly build --release
❯ ./target/release/rocket

❯ wrk --latency -t4 -c200 -d8s http://127.0.0.1:8000
Running 8s test @ http://127.0.0.1:8000
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   786.81us  445.55us  14.41ms   70.86%
    Req/Sec    18.22k   734.43    22.02k    78.75%
  Latency Distribution
     50%  799.00us
     75%    1.09ms
     90%    1.28ms
     99%    1.96ms
  580046 requests in 8.05s, 80.76MB read
  Socket errors: connect 0, read 580030, write 0, timeout 0
Requests/sec:  72045.45
Transfer/sec:     10.03MB

❯ wrk --latency -t4 -c200 -d8s http://127.0.0.1:8000
Running 8s test @ http://127.0.0.1:8000
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.92ms    3.56ms 205.67ms   99.80%
    Req/Sec    17.09k     0.91k   19.49k    68.44%
  Latency Distribution
     50%  809.00us
     75%    1.16ms
     90%    1.38ms
     99%    2.08ms
  544051 requests in 8.04s, 75.75MB read
  Socket errors: connect 0, read 544036, write 0, timeout 0
Requests/sec:  67683.32
Transfer/sec:      9.42MB

❯ wrk --latency -t4 -c200 -d8s http://127.0.0.1:8000
Running 8s test @ http://127.0.0.1:8000
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.98ms    2.43ms 209.81ms   99.43%
    Req/Sec    15.55k     1.48k   19.28k    90.62%
  Latency Distribution
     50%    0.91ms
     75%    1.29ms
     90%    1.53ms
     99%    2.69ms
  495140 requests in 8.04s, 68.94MB read
  Socket errors: connect 0, read 495120, write 0, timeout 0
Requests/sec:  61560.47
Transfer/sec:      8.57MB

Rocket v5

❯ cd rocket
❯ # change Cargo.toml
❯ # change main.rs to work with v5
❯ cargo build --release
❯ ./target/release/rocket

@@ -1,5 +1,3 @@
-#![feature(proc_macro_hygiene, decl_macro)]
-
 #[macro_use]
 extern crate rocket;
 
@@ -8,6 +6,8 @@ fn index() -> &'static str {
     "Hello, world!"
 }
 
-fn main() {
-    rocket::ignite().mount("/", routes![index]).launch();
+#[launch]
+fn rocket() -> _ {
+    rocket::build()
+        .mount("/", routes![index])
 }

❯ wrk --latency -t4 -c200 -d8s http://127.0.0.1:8000
Running 8s test @ http://127.0.0.1:8000
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.30ms    1.03ms  26.44ms   86.44%
    Req/Sec    39.56k     3.79k   48.86k    67.19%
  Latency Distribution
     50%    1.08ms
     75%    1.61ms
     90%    2.33ms
     99%    5.05ms
  1261745 requests in 8.08s, 175.68MB read
Requests/sec: 156093.58
Transfer/sec:     21.73MB

❯ wrk --latency -t4 -c200 -d8s http://127.0.0.1:8000
Running 8s test @ http://127.0.0.1:8000
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.38ms    0.95ms  28.54ms   79.82%
    Req/Sec    36.52k     3.16k   45.96k    67.81%
  Latency Distribution
     50%    1.18ms
     75%    1.74ms
     90%    2.46ms
     99%    4.83ms
  1164811 requests in 8.07s, 162.18MB read
Requests/sec: 144278.36
Transfer/sec:     20.09MB

❯ wrk --latency -t4 -c200 -d8s http://127.0.0.1:8000
Running 8s test @ http://127.0.0.1:8000
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.43ms    0.99ms  21.53ms   79.25%
    Req/Sec    35.24k     3.21k   48.04k    69.38%
  Latency Distribution
     50%    1.22ms
     75%    1.81ms
     90%    2.57ms
     99%    5.05ms
  1124770 requests in 8.08s, 156.61MB read
Requests/sec: 139211.46
Transfer/sec:     19.38MB

Rocket v5 (ROCKET_ENV=prod)

❯ cd rocket
❯ # change Cargo.toml
❯ # change main.rs to work with v5
❯ cargo build --release
❯ env ROCKET_ENV=prod ./target/release/rocket

❯ wrk --latency -t4 -c200 -d8s http://127.0.0.1:8000
Running 8s test @ http://127.0.0.1:8000
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.21ms    0.92ms  21.33ms   84.57%
    Req/Sec    42.26k     3.27k   50.97k    74.38%
  Latency Distribution
     50%    1.02ms
     75%    1.50ms
     90%    2.15ms
     99%    4.61ms
  1348461 requests in 8.08s, 187.75MB read
Requests/sec: 166937.68
Transfer/sec:     23.24MB

❯ wrk --latency -t4 -c200 -d8s http://127.0.0.1:8000
Running 8s test @ http://127.0.0.1:8000
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.32ms    0.94ms  23.26ms   81.14%
    Req/Sec    38.34k     3.62k   47.96k    69.38%
  Latency Distribution
     50%    1.12ms
     75%    1.66ms
     90%    2.37ms
     99%    4.79ms
  1223178 requests in 8.08s, 170.31MB read
Requests/sec: 151402.49
Transfer/sec:     21.08MB

❯ wrk --latency -t4 -c200 -d8s http://127.0.0.1:8000
Running 8s test @ http://127.0.0.1:8000
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.45ms    1.00ms  19.32ms   79.40%
    Req/Sec    34.91k     3.04k   44.97k    71.25%
  Latency Distribution
     50%    1.24ms
     75%    1.83ms
     90%    2.59ms
     99%    5.15ms
  1113370 requests in 8.06s, 155.02MB read
Requests/sec: 138150.06
Transfer/sec:     19.24MB

Rocket v5 (witch async keyword)

❯ cd rocket
❯ # change Cargo.toml
❯ # change main.rs to work with v5
❯ cargo build --release
❯ env ROCKET_ENV=prod ./target/release/rocket

❯ wrk --latency -t4 -c200 -d8s http://127.0.0.1:8000
Running 8s test @ http://127.0.0.1:8000
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.26ms    0.94ms  18.77ms   83.92%
    Req/Sec    40.41k     3.72k   49.53k    72.50%
  Latency Distribution
     50%    1.07ms
     75%    1.57ms
     90%    2.23ms
     99%    4.80ms
  1289197 requests in 8.08s, 179.50MB read
Requests/sec: 159601.93
Transfer/sec:     22.22MB


❯ wrk --latency -t4 -c200 -d8s http://127.0.0.1:8000
Running 8s test @ http://127.0.0.1:8000
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.41ms    0.99ms  18.31ms   80.39%
    Req/Sec    35.79k     3.46k   46.59k    73.44%
  Latency Distribution
     50%    1.21ms
     75%    1.78ms
     90%    2.52ms
     99%    5.11ms
  1142153 requests in 8.08s, 159.03MB read
Requests/sec: 141319.52
Transfer/sec:     19.68MB

❯ wrk --latency -t4 -c200 -d8s http://127.0.0.1:8000
Running 8s test @ http://127.0.0.1:8000
  4 threads and 200 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     1.52ms    1.09ms  29.17ms   80.94%
    Req/Sec    33.37k     3.86k   45.75k    82.81%
  Latency Distribution
     50%    1.29ms
     75%    1.92ms
     90%    2.73ms
     99%    5.45ms
  1065163 requests in 8.07s, 148.31MB read
Requests/sec: 131921.73
Transfer/sec:     18.37MB

Summary

warp: 286_126.89, 259_381.43, 238_068.17
Rocket v4: 72_045.45, 67_683.32, 61_560.47
Rocket v5: 156_093.58, 144_278.36, 139_211.46
Rocket v5 (ROCKET_ENV=prod): 166_937.68, 151_402.49, 138_150.06
Rocket v5 (witch async keyword): 159_601.93, 141_319.52, 131_921.73

Machine

I have made my machine quite before running the benchmark.

❯ macchina
    Machine   —  LENOVO ThinkPad L390 20NSS0TA00
    Kernel    —  Linux 4.19.0-14-amd64
    Distro    —  Debian GNU/Linux
    WM        —  i3
    CPU       —  Intel® Core™ i7-8565U CPP
    Memory    —  3.3 GB/15.9 GB

Notes

IMHO, to make the command wrk --latency -t4 -c200 -d8s http://127.0.0.1:8080 consistent. We need to set each port of the web framework to 8080