makemake-kbo / flood

flood is a load testing tool for benchmarking EVM nodes over RPC

🌊🌊 flood 🌊🌊

flood is a load testing tool for benchmarking EVM nodes over RPC

For each RPC method, flood measures how load affects metrics such as:

1. throughput
2. latency (mean, P50, P90, P95, P99, max)
3. error rate

flood makes it easy to compare the performance of:

• different node clients (e.g. geth vs erigon vs reth)
• different hardware/software configurations (e.g. local vs cloud vs low memory vs RAID-0)
• different RPC providers (e.g. Alchemy vs Quicknode vs Infura)

flood can generate tables, figures, and reports for easy sharing of results (example report here)

Contents

1. Installation

   1. Prerequisites
   2. Installing Flood
   3. Docker

2. Usage Guide

   1. Basic Load Tests
   2. Remote Load Tests
   3. Printing Test Results
   4. Report Generation
   5. Differential Tests
   6. From Python
   7. Performing Deep Checks

3. Contributing

Installation

Prerequisites

Install vegeta:

• on mac: brew update && brew install vegeta
• on linux: go install github.com/tsenart/vegeta/[email protected]

After installation, make sure vegeta is on your $PATH. Running vegeta -h should output a path. If it does not, you probably have not set up go to install items to your $PATH. You may need to add something like export PATH=$PATH:~/go/bin/ to your terminal config file (e.g. ~/.profile).

flood also requires python >= 3.7

Installing flood

pip install paradigm-flood

Typing flood help in your terminal should show help output. If it does not, you probably have not set up pip to install items to your $PATH. You may need to add something like export PATH=$PATH:~/.local/bin to your terminal config file (e.g. ~/.profile). Alternatively, you can avoid setting up your $PATH and just type python3 -m flood instead of flood.

Docker

Alternatively, flood can be used as a Docker image.

Usage guide

flood works by bombarding an RPC endpoint with different patterns of RPC calls. Measurements of the RPC endpoint's performance under different controlled loads are then used to paint a detailed view of the node's performance.

Every time flood runs, it saves its parameters and test results to an output directory. You can specify this output directory with the --output parameter, otherwise a temporary directory will be created. Running a test will populate the folder with the following files:

• figures/: directory containing PNG's summarizing node performance
• results.json: results of the test including performance metrics
• summary.txt: printed summary of test that was output to the console
• test.json: metadata and parameters used to create and run the test

Basic load tests

Here is an example of a basic test with flood. It will benchmark block retrieval from two different nodes. It will test at 3 different rates (10, 100, and 1000 requests per second) and it will test them for 30 seconds each.

flood eth_getBlockByNumber NODE1_NAME=NODE1_URL NODE2_NAME=NODE2_URL --rates 10 100 1000 --duration 30

To see all of the parameters available for controlling flood tests use flood --help

Remote load tests

Instead of broadcasting RPC calls from whatever machine running the flood CLI command, flood can broadcast the calls from a remote process on a remote machine. In particular, flood can broadcast the calls from the same machine that is running the EVM node in order to eliminate any noise or bottlenecks associated with networking.

This can be accomplished by installing flood on the remote machine and then providing flood with login credentials and routing details using the following syntax:

flood <test> [node_name=][username@]hostname:[test_url] ...

For example, the following command will test a reth node and an erigon node remotely:

flood eth_call [email protected]:localhost:8545 [email protected]:localhost:8545

If there are multiple remote tests, these tests will be run in parallel. After the tests are complete, flood will retrieve the results and summarize using the same methodology as a local test.

Printing test results

By default flood produces verbose output of each test as it runs. This can be disabled with the --quiet parameter. To re-print the results of an old test, use flood print <TEST_DIR>. To print a summary of multiple tests, use flood print <test_1_dir> <test_2_dir>.

Report generation

After running tests, you can generate an HTML + Jupyter report similar to this one. This is done by running flood report <TEST_DIR>. Multiple tests can be combined into one report with flood repos <TEST_DIR_1> <TEST_DIR_2> ....

Differential tests

Instead of testing the raw performance of an RPC node, flood can be used to test the correctness of a node's responses using a differential testing approach. This works by using two nodes and making sure that their responses match under a variety of RPC calls. This is done using the --equality parameter. For example:

flood all reth=91.91.91.91 erigon=92.92.92.92 --equality

From python

All of flood's functionality can be used from python instead of the CLI. Some functions:

description	python
Import flood	import flood
Run tests	flood.run(...)
Load the parameters of a test	flood.load_single_run_results_payload(output_dir)
Load the results of a test	flood.load_single_run_results_payload(output_dir)
Run a live version of a results notebook	jupyter notebook <TEST_DIR>

Performing deep checks

Under normal operation flood relies on vegeta to compute performance summaries of each test. This works well, but sometimes it is desirable to implement custom introspection not available in vegeta.

In particular, vegeta counts any status-200 response as a success, even if the contents of the response is an RPC error. Running with the --deep-check command will check every response to make sure that it returns well-formed JSON with no RPC errors. With --deep-check, flood also computes separate performance statistics successful vs failed calls.

If you want to save the timing information and raw contents of every single response from the test to the results.json output, use the --save-raw-output argument. This allows for performing own custom analyses on the raw data.

Contributing

Contributions are welcome in the form of issues, PR's, and commentary. Check out the contributor guide in CONTRIBUTING.md.