the-guild-org / gateways-benchmark

GraphQL Gateways Benchmark

This repository is a collection is different scenarios and tests performed on different implementations of GraphQL gateways.

This goals of this repo:

1. Provide a transparent, accurate and descriptive benchmark testing for different tools:

   ➡️ All tests are running in Docker containers, with the same configuration.

   ➡️ During the test, tools like cadvisor are monitoring stats.

   ➡️ Promethues and Grafana are stores results and stats for every run, and we export and embed it as PNG.

   ➡️ Every scenario have a different report, based on what it measures.

   ➡️ We are using a dedicated GitHub Actions runners, with concurrency=1 to make sure tests are running as standalone.

2. Always be up to date:

   ➡️ This allows tools to improve over time.

   ➡️ All code in the repository is open and can be changed by the community.

   ➡️ Scenarios are executed for every PR, and reports are generated automatically per scenario.

   ➡️ Renovate keeps all dependencies up-to-date.

3. Various scenarios

   ➡️ We are trying to create real-life scenarios based on our experience and our customers.

   ➡️ Each scenario has differnt setup and measure different stats.

Scenarios

fed-v1-constant-vus-over-time

Latest Results

This scenario runs the following:

1. 4 GraphQL subgraphs in dedicated services
2. A GraphQL gateway compatible with the Apollo Federation v1 spec
3. Constant rate of VUs over fixed time span

This measures the following:

1. RPS (requests per second) rate
2. Request duration (average, p95)
3. Request failures (count)
4. CPU usage during the entire execution
5. RAM usage during the entire execution
6. HTTP layer timings

This scenario uses Federation v1 spec with all gateways that supports this kind of specification (not all gateways supports v2 spec).

fed-v1-constant-vus-subgraphs-delay

Latest Results

This scenario runs the same flow as fed-v1-constant-vus-over-time but with an intentional delay on each upstream Subgraph. This creates more stress and increases memory in the server due to the more inflight requests.

fed-v1-constant-vus-subgraphs-delay-resources

Latest Results

This scenario runs the same flow as fed-v1-constant-vus-subgraphs-delay but with more resources (CPU and RAM) allocated for the gateway.

fed-v1-ramping-vus

Latest Results

This scenario runs the following:

1. 4 GraphQL subgraphs in dedicated services
2. A GraphQL gateway compatible with the Apollo Federation spec
3. Gradually ramping VUs to a high number, to demo a stress scenario

This measures the following:

1. RPS (requests per second) rate
2. Request duration (average, med, max, p95)
3. Request failures (count)
4. CPU usage during the entire execution
5. RAM usage during the entire execution
6. HTTP layer timings

This scenario uses Federation v1 spec with all gateways that supports this kind of specification (not all gateways supports v2 spec).