Disclaim: This project is copied and adapted from https://github.com/networknt/light-java-example/tree/master/performance
Note: As suggested by the project name, this is a benchmark that only show very limited aspect about framework performance: how framework respond with a simple plain text message. For real comprehensive view on framework performance, please refer to https://www.techempower.com/benchmarks
| Framework | Max Throughput | Avg Latency | Transfer |
|---|---|---|---|
| Go-Fast-HTTP | 1888432.99 | 0.94ms | 226.92MB |
| Light Java | 1678297.39 | 1.94ms | 169.66MB |
| ActFramework | 761081.24 | 4.27ms | 119.76MB |
| Spring Boot Reactor | 384186.78 | 4.17ms | 28.21MB |
| Spark | 291954.47 | 11.14ms | 49.28MB |
| RatPack | 241775.80 | 8.44ms | 20.87MB |
| Go-HTTP | 241484.14 | 7.02ms | 29.71MB |
| JFinal 3 | 229960.41 | 8.93ms | 49.12MB |
| Play-Java | 218074.28 | 13.80ms | 26.83MB |
| AKKA-HTTP | 200481.39 | 26.26ms | 29.64MB |
| Iris (Go) | 169879.19 | 11.25ms | 20.90MB |
| Jooby/Undertow | 140645.13 | 15.91ms | 20.25MB |
| Spring Boot Undertow | 89433.13 | 18.88ms | 12.96MB |
| Bootique + Jetty/Jersey | 82373.78 | 50.39ms | 14.14MB |
| Dropwizard | 78079.06 | 57.73ms | 7.45MB |
| Spring Boot Tomcat | 68554.49 | 42.84ms | 8.25MB |
| Payra-Micro | 61703.99 | 63.32ms | 8.71MB |
| WildFly Swarm | 58179.11 | 20.08ms | 7.66MB |
| Ninjaframework | 51948.21 | 27.55ms | 14.81MB |
| Play-1 | 50176.54 | 22.42ms | 21.44MB |
| Node-express | 47429.61 | 23.26ms | 10.40MB |
| msf4j | fail | fail | fail |
The benchmark is running on my local laptop with the following configuration:
- toshiba i7-4700MQ 16G RAM SSD
- Linuxmint 18.1
- Open JDK 1.8.0_121
- go 1.6.2 linux/amd64
The benchmark tool is wrk and we are using pipeline.lua to generate more requests per second
For each testbed, we run at least twice using the command as shown below:
wrk -t4 -c128 -d30s http://localhost:8080 -s pipeline.lua --latency -- / 16
The port 8080 might be different as per project setting. Please read the README.md file inside each project to check the exact port number.
This is a simple webframework benchmark. All framework testbed accept HTTP GET request to / and respond with text Hello World!.
For comprehensive webframework benchmark, one should refer to https://www.techempower.com/benchmarks/
luog@luog-Satellite-P50-A:~/p/greenlaw110/FrameworkBenchmarks/frameworks/Java/act$ wrk -t4 -c256 -d15s http://localhost:5460 -s pipeline.lua --latency -- / 16
Running 15s test @ http://localhost:5460
4 threads and 256 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 4.27ms 4.79ms 211.96ms 93.75%
Req/Sec 191.84k 36.57k 323.23k 62.50%
Latency Distribution
50% 3.71ms
75% 5.72ms
90% 7.80ms
99% 13.46ms
11466144 requests in 15.07s, 1.76GB read
Requests/sec: 761081.24
Transfer/sec: 119.76MB
luog@luog-Satellite-P50-A:~/p/tmp/light-java-example/performance$ wrk -t4 -c128 -d30s http://localhost:8080 -s pipeline.lua --latency -- / 16
Running 30s test @ http://localhost:8080
4 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 26.26ms 128.23ms 1.83s 97.35%
Req/Sec 51.74k 11.87k 133.70k 87.02%
Latency Distribution
50% 5.65ms
75% 9.42ms
90% 17.70ms
99% 786.31ms
6034389 requests in 30.10s, 0.87GB read
Requests/sec: 200481.39
Transfer/sec: 29.64MB
luog@luog-Satellite-P50-A:~/p/greenlaw110/webframework-benchmark$ wrk -t4 -c128 -d30s http://localhost:8080 -s pipeline.lua --latency -- / 16
Running 30s test @ http://localhost:8080
4 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 50.39ms 61.17ms 730.72ms 85.85%
Req/Sec 20.72k 3.54k 38.29k 71.48%
Latency Distribution
50% 25.08ms
75% 69.40ms
90% 136.04ms
99% 273.06ms
2476508 requests in 30.06s, 425.12MB read
Requests/sec: 82373.78
Transfer/sec: 14.14MB
luog@luog-Satellite-P50-A:~/p/greenlaw110/webframework-benchmark$ wrk -t4 -c128 -d30s http://localhost:8080 -s pipeline.lua --latency -- / 16
Running 30s test @ http://localhost:8080
4 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 57.73ms 73.40ms 737.18ms 85.68%
Req/Sec 19.64k 4.22k 42.51k 71.30%
Latency Distribution
50% 25.43ms
75% 79.71ms
90% 162.27ms
99% 321.80ms
2350101 requests in 30.10s, 224.12MB read
Requests/sec: 78079.06
Transfer/sec: 7.45MB
wrk -t4 -c128 -d30s http://localhost:8080 -s pipeline.lua --latency -- / 16
Running 30s test @ http://localhost:8080
4 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 0.94ms 1.14ms 41.94ms 90.79%
Req/Sec 474.64k 56.55k 646.95k 67.17%
Latency Distribution
50% 621.00us
75% 1.14ms
90% 1.98ms
99% 5.38ms
56820016 requests in 30.09s, 6.67GB read
Requests/sec: 1888432.99
Transfer/sec: 226.92MB
wrk -t4 -c128 -d30s http://localhost:8080 -s pipeline.lua --latency -- / 16
Running 30s test @ http://localhost:8080
4 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 7.02ms 6.43ms 89.62ms 86.83%
Req/Sec 60.70k 5.54k 94.08k 72.23%
Latency Distribution
50% 5.25ms
75% 9.02ms
90% 14.87ms
99% 31.30ms
7254992 requests in 30.04s, 0.87GB read
Requests/sec: 241484.14
Transfer/sec: 29.71MB
wrk -t4 -c128 -d30s http://localhost:8080 -s pipeline.lua --latency -- / 16
Running 30s test @ http://localhost:8080
4 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 11.25ms 10.62ms 128.91ms 86.35%
Req/Sec 42.69k 3.67k 56.17k 69.57%
Latency Distribution
50% 7.83ms
75% 15.23ms
90% 25.21ms
99% 49.84ms
5104910 requests in 30.05s, 628.03MB read
Requests/sec: 169879.19
Transfer/sec: 20.90MB
luog@luog-Satellite-P50-A:~/p/tmp/light-java-example/performance$ wrk -t4 -c128 -d30s http://localhost:8080 -s pipeline.lua --latency -- / 16
Running 30s test @ http://localhost:8080
4 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 8.93ms 12.67ms 171.84ms 92.07%
Req/Sec 57.81k 9.61k 103.15k 79.67%
Latency Distribution
50% 5.20ms
75% 9.44ms
90% 18.31ms
99% 71.59ms
6921586 requests in 30.10s, 1.44GB read
Requests/sec: 229960.41
Transfer/sec: 49.12MB
luog@luog-Satellite-P50-A:~/p/tmp/light-java-example/performance$ wrk -t4 -c128 -d30s http://localhost:8080 -s pipeline.lua --latency -- / 16
Running 30s test @ http://localhost:8080
4 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 15.91ms 13.45ms 133.45ms 75.33%
Req/Sec 35.35k 3.46k 46.71k 69.42%
Latency Distribution
50% 12.45ms
75% 22.23ms
90% 34.20ms
99% 60.77ms
4221760 requests in 30.02s, 607.95MB read
Requests/sec: 140645.13
Transfer/sec: 20.25MB
luog@luog-Satellite-P50-A:~/p/tmp/light-java-example/performance$ wrk -t4 -c128 -d30s http://localhost:8080 -s pipeline.lua --latency -- / 16
Running 30s test @ http://localhost:8080
4 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 1.94ms 4.40ms 130.96ms 94.38%
Req/Sec 422.30k 88.35k 760.32k 79.40%
Latency Distribution
50% 804.00us
75% 1.90ms
90% 4.23ms
99% 17.76ms
50501984 requests in 30.09s, 4.99GB read
Requests/sec: 1678297.39
Transfer/sec: 169.66MB
luog@luog-Satellite-P50-A:~/p/tmp/light-java-example/performance$ wrk -t4 -c128 -d30s http://localhost:8080 -s pipeline.lua --latency -- / 16
Running 30s test @ http://localhost:8080
4 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 18.42ms 18.00ms 86.77ms 82.64%
Req/Sec 10.77k 3.10k 17.82k 83.33%
Latency Distribution
50% 11.81ms
75% 27.67ms
90% 45.60ms
99% 80.58ms
12875 requests in 30.05s, 1.13MB read
Socket errors: connect 0, read 270, write 0, timeout 0
Requests/sec: 428.52
Transfer/sec: 38.65KB
Note msf4j test cannot be completed due to huge number of error
2017-01-28 17:07:19 ERROR WorkerPoolDispatchingSourceHandler:127 - Error occurred inside the messaging engine
java.lang.NullPointerException
at org.wso2.msf4j.internal.MSF4JMessageProcessor.receive(MSF4JMessageProcessor.java:69)
at org.wso2.carbon.transport.http.netty.listener.WorkerPoolDispatchingSourceHandler.lambda$publishToWorkerPool$12(WorkerPoolDispatchingSourceHandler.java:125)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
While using ab to test with -c argument, it will halt. Looks like the server failed to release connection
luog@luog-Satellite-P50-A:~/p/tmp/light-java-example/performance$ wrk -t4 -c128 -d30s http://localhost:8080 -s pipeline.lua --latency -- / 16
Running 30s test @ http://localhost:8080
4 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 27.55ms 18.56ms 169.72ms 66.95%
Req/Sec 13.08k 1.13k 40.77k 85.26%
Latency Distribution
50% 24.11ms
75% 39.00ms
90% 53.58ms
99% 80.25ms
1563624 requests in 30.10s, 445.87MB read
Requests/sec: 51948.21
Transfer/sec: 14.81MB
luog@luog-Satellite-P50-A:~/p/tmp/light-java-example/performance$ wrk -t4 -c128 -d30s http://localhost:8080 -s pipeline.lua --latency -- / 16
Running 30s test @ http://localhost:8080
4 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 23.26ms 10.97ms 193.31ms 52.16%
Req/Sec 11.93k 577.06 13.40k 80.32%
Latency Distribution
50% 29.18ms
75% 39.91ms
90% 0.00us
99% 0.00us
1424427 requests in 30.03s, 312.44MB read
Requests/sec: 47429.61
Transfer/sec: 10.40MB
luog@luog-Satellite-P50-A:~/p/tmp/light-java-example/performance$ wrk -t4 -c128 -d30s http://localhost:8080 -s pipeline.lua --latency -- / 16
Running 30s test @ http://localhost:8080
4 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 63.32ms 75.53ms 971.03ms 85.91%
Req/Sec 15.51k 2.29k 23.16k 69.57%
Latency Distribution
50% 32.43ms
75% 88.55ms
90% 167.47ms
99% 333.34ms
1856843 requests in 30.09s, 262.21MB read
Requests/sec: 61703.99
Transfer/sec: 8.71MB
luog@luog-Satellite-P50-A:~/p/tmp/light-java-example/performance$ wrk -t4 -c128 -d30s http://localhost:8080 -s pipeline.lua --latency -- / 16
Running 30s test @ http://localhost:8080
4 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 13.90ms 14.89ms 186.20ms 86.63%
Req/Sec 54.89k 9.39k 121.86k 70.94%
Latency Distribution
50% 8.83ms
75% 19.19ms
90% 33.24ms
99% 67.94ms
6563656 requests in 30.10s, 807.49MB read
Requests/sec: 218074.28
Transfer/sec: 26.83MB
luog@luog-Satellite-P50-A:~/p/tmp/light-java-example/performance$ wrk -t4 -c128 -d30s http://localhost:9000 -s pipeline.lua --latency -- / 16
Running 30s test @ http://localhost:9000
4 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 22.42ms 11.35ms 70.70ms 58.37%
Req/Sec 12.61k 730.55 14.56k 75.75%
Latency Distribution
50% 22.14ms
75% 32.31ms
90% 37.47ms
99% 43.82ms
1505641 requests in 30.01s, 643.28MB read
Requests/sec: 50176.54
Transfer/sec: 21.44MB
Hello World!luog@luog-Satellite-P50-A:~/p/tmp/light-java-example/performance$ wrk -t4 -c128 -d30s http://localhost:5050 -s pipeline.lua --latency -- / 16
Running 30s test @ http://localhost:5050
4 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 8.44ms 8.48ms 99.56ms 86.67%
Req/Sec 60.85k 12.38k 115.96k 73.24%
Latency Distribution
50% 5.56ms
75% 11.37ms
90% 19.41ms
99% 40.10ms
7274698 requests in 30.09s, 631.33MB read
Requests/sec: 241775.80
Transfer/sec: 20.98MB
luog@luog-Satellite-P50-A:~/p/tmp/light-java-example/performance$ wrk -t4 -c128 -d30s http://localhost:4567 -s pipeline.lua --latency -- / 16
Running 30s test @ http://localhost:4567
4 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 11.14ms 17.60ms 264.11ms 89.63%
Req/Sec 73.42k 10.34k 118.73k 71.06%
Latency Distribution
50% 4.55ms
75% 11.29ms
90% 29.61ms
99% 88.37ms
8786831 requests in 30.10s, 1.45GB read
Requests/sec: 291954.47
Transfer/sec: 49.28MB
luog@luog-Satellite-P50-A:~/p/tmp/light-java-example/performance$ wrk -t4 -c128 -d30s http://localhost:3000 -s pipeline.lua --latency -- / 16
Running 30s test @ http://localhost:3000
4 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 4.17ms 3.99ms 57.24ms 87.97%
Req/Sec 96.64k 13.88k 185.06k 73.95%
Latency Distribution
50% 2.96ms
75% 5.22ms
90% 9.15ms
99% 19.43ms
11557468 requests in 30.08s, 848.70MB read
Requests/sec: 384186.78
Transfer/sec: 28.21MB
Running 30s test @ http://localhost:8080
4 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 18.88ms 15.52ms 149.97ms 76.72%
Req/Sec 22.47k 2.10k 28.28k 75.33%
Latency Distribution
50% 14.52ms
75% 26.44ms
90% 39.97ms
99% 70.50ms
2684368 requests in 30.02s, 389.12MB read
Requests/sec: 89433.13
Transfer/sec: 12.96MB
luog@luog-Satellite-P50-A:~/p/tmp/light-java-example/performance$ wrk -t4 -c128 -d30s http://localhost:8080 -s pipeline.lua --latency -- / 16
Running 30s test @ http://localhost:8080
4 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 42.84ms 58.42ms 794.35ms 87.61%
Req/Sec 17.23k 3.40k 25.03k 74.56%
Latency Distribution
50% 19.22ms
75% 50.19ms
90% 117.59ms
99% 272.61ms
2063347 requests in 30.10s, 248.31MB read
Requests/sec: 68554.49
Transfer/sec: 8.25MB
luog@luog-Satellite-P50-A:~/p/tmp/light-java-example/performance$ wrk -t4 -c128 -d30s http://localhost:8080 -s pipeline.lua --latency -- / 16
Running 30s test @ http://localhost:8080
4 threads and 128 connections
Thread Stats Avg Stdev Max +/- Stdev
Latency 20.08ms 11.37ms 143.50ms 64.97%
Req/Sec 14.63k 1.66k 22.76k 85.75%
Latency Distribution
50% 20.55ms
75% 29.25ms
90% 38.70ms
99% 0.00us
1749499 requests in 30.07s, 230.25MB read
Requests/sec: 58179.11
Transfer/sec: 7.66MB
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent