the-guild-org / gateways-benchmark Goto Github PK

View Code? Open in Web Editor NEW

39.0 39.0 7.0 3.59 MB

License: MIT License

JavaScript 43.47% Dockerfile 1.04% TypeScript 47.48% Shell 1.36% Rust 6.39% HCL 0.27%

gateways-benchmark's People

Contributors

Stargazers

Watchers

Forkers

wundergraph yassineldeeb fbjork locnguyenhuu finistere hasura

gateways-benchmark's Issues

Add Memory and CPU usage in the comparisons tables

add warmup phase to scenarios

confirm correct responses to ensure compatibility

Scenario: Ramping VUs until server breaks

Run:

Federation services
Federation Gateway

Query:

medium-size query

Scenario:

Ramping VUs https://k6.io/docs/using-k6/scenarios/executors/ramping-vus/

Measure:

Memory
CPU
Duration
Failures

Scenario: Copy some from stitch repo tests

https://github.com/yaacovCR/stitch

Scenario: Large Federation query

Run:

Federation services
Federation Gateway

Query:

Large query (all services, some nested and linked fields)

Scenario:

50 VU over 60s

Measure:

Memory
CPU
RPS
Duration
Failures

Benchmark gateway: Apollo Router (Rust)

Scenario: Single GraphQL endpoint wrapper

Run:

Single schema service
Gateway

Query:

Medium-size query

Scenario:

100 VU over 60s

Measure:

Memory
CPU
RPS
Duration
Failures

Benchmark gateway: Wundergraph

Latency vs Throughput benchmark

Hello!

Creating this issue to discuss what could be done to imitate what I did in the benchmark blogpost.

Currently, all benchmarks in this repository work by having a single K6 load test with a given scenario. It's unfortunately not enough to generate the latency vs throughput graphs I generated. Multiple benchmark runs are needed with a fixed arrival rate. The tricky part is that obviously, each gateway has a different maximum throughput. So to automate it, it would require a loop that does something like this:

# pseudo code

# Warmup for a few seconds
k6_run_fixed_arrival_rate(100)

# actual benchmark
arrival_rate = 100
step = 100
while true:
    results = k6_run_fixed_arrival_rate(arrival_rate)
    if (results['metrics']['iterations']['values']['rate'] - arrival_rate).abs() < 3:
        arrival_rate += step
    else:
       step = step/2
       if step <= 5:
           break
       arrival_rate -= step

That's roughly what I did. Manually 😢. The more difficult part is that I adjusted preAllocatedVUs over the benchmark runs, but it's only important when close to being CPU-bound. So for an automated benchmark, I would just set it to something high enough for all gateways and that's it. High enough means that you shouldn't have a case where the actual throughput measured by K6 doesn't match the fixed arrival rate with a gateway that wasn't CPU-bound (max cpu < CPU_LIMIT).

The last part is something that could be done with some bash. What's particularly unclear to me is how you should generate the results table/graph. In my benchmarks, I wrote the results in JSON from docker stats and k6 and processed them in a Python notebook, available here. I didn't clean it up, but if you're familiar with Python it shouldn't be too hard, I hope. :)

If you're considering adding this benchmark, I would also suggest adding a 10ms subgraph delay like I did in the end, it's more representative of a real-world case. In the current benchmarks, it's not noticeable (unless you add huge delays) because most of the latency comes from CPU contention in the first place. Adding a few dozen ms of network delay can't impact latencies in the order of seconds.

In your stead, I would also consider removing Grafana, Prometheus, and Cadvisor and instead, only use docker stats or the Docker API to retrieve the resource consumption information. I would expect it to be the most precise measurement you can have easily and it would avoid using resources for this. This is not a small change, so well. 🤷

Regarding using the host network for docker, I did see an impact without any network delay for Grafbase. I don't remember if it did for the others. So I would tend to recommend it, but I'm not sure whether this works on MacOS/Windows.

Repository problems

These problems occurred while renovating this repository. View logs.

WARN: Package lookup failures

Warning

Renovate failed to look up the following dependencies: Failed to look up docker package ghcr.io/grafbase/grafbase.

Files affected: federation-v1/gateways/grafbase/Dockerfile

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

cargo

federation-v1/subgraphs/accounts/Cargo.toml

async-graphql 6.0.11

async-graphql-axum 6.0.11

axum 0.6.20

lazy_static 1.4.0

rand 0.8.5

tokio 1.36.0

federation-v1/subgraphs/inventory/Cargo.toml

async-graphql 6.0.11

async-graphql-axum 6.0.11

axum 0.6.20

lazy_static 1.4.0

rand 0.8.5

tokio 1.36.0

federation-v1/subgraphs/products/Cargo.toml

async-graphql 6.0.11

async-graphql-axum 6.0.11

axum 0.6.20

lazy_static 1.4.0

rand 0.8.5

tokio 1.36.0

federation-v1/subgraphs/reviews/Cargo.toml

async-graphql 6.0.11

async-graphql-axum 6.0.11

axum 0.6.20

lazy_static 1.4.0

rand 0.8.5

tokio 1.36.0

docker-compose

docker-compose.metrics.yaml

prom/prometheus v2.51.0

grafana/grafana 10.4.1

gcr.io/cadvisor/cadvisor v0.49.1

federation-v1/gateways/apollo-router/docker-compose.yaml

ghcr.io/apollographql/router v1.43.0

federation-v1/gateways/apollo-server/docker-compose.yaml

federation-v1/gateways/cosmo/docker-compose.yaml

federation-v1/gateways/grafbase/docker-compose.yaml

federation-v1/gateways/mercurius/docker-compose.yaml

federation-v1/gateways/mesh-bun/docker-compose.yaml

federation-v1/gateways/mesh-supergraph-bun/docker-compose.yaml

federation-v1/gateways/mesh-supergraph/docker-compose.yaml

federation-v1/gateways/mesh/docker-compose.yaml

federation-v1/gateways/wundergraph/docker-compose.yaml

federation-v1/subgraphs/docker-compose.subgraphs.local.yaml

federation-v1/subgraphs/docker-compose.subgraphs.yaml

dockerfile

federation-v1/gateways/apollo-server/Dockerfile

node 21-alpine

federation-v1/gateways/cosmo/Dockerfile

alpine 20240329

federation-v1/gateways/grafbase/Dockerfile

ghcr.io/grafbase/grafbase 0.59.0

federation-v1/gateways/mercurius/Dockerfile

node 21-alpine

federation-v1/gateways/mesh-bun/Dockerfile

oven/bun 1.0.35

federation-v1/gateways/mesh-supergraph-bun/Dockerfile

oven/bun 1.0.35

federation-v1/gateways/mesh-supergraph/Dockerfile

node 21-alpine

federation-v1/gateways/mesh/Dockerfile

node 21-alpine

federation-v1/gateways/wundergraph/Dockerfile

node 21-alpine

federation-v1/subgraphs/Dockerfile.rust

rust 1.77.0

debian 12.5

github-actions

.github/workflows/benchmark.template.yaml

actions/checkout v4

docker/login-action v3

actions/upload-artifact v4

.github/workflows/decide-runner.template.yaml

.github/workflows/docker-bake.template.yaml

actions/checkout v4

docker/setup-qemu-action v3

docker/setup-buildx-action v3

docker/login-action v3

docker/bake-action v4

.github/workflows/report.template.yaml

actions/checkout v4

actions/download-artifact v4

actions/setup-node v4

marocchino/sticky-pull-request-comment v2

EndBug/add-and-commit v9

actions/upload-artifact v4

npm

federation-v1/gateways/apollo-server/package.json

tsx 4.7.1

typescript 5.4.3

graphql 16.8.1

@apollo/server 4.10.2

@apollo/gateway 2.7.2

@apollo/composition 2.7.2

federation-v1/gateways/grafbase/package.json

@grafbase/sdk ~0.17.0

federation-v1/gateways/mercurius/package.json

tsx 4.7.1

typescript 5.4.3

graphql 16.8.1

fastify 4.26.2

@mercuriusjs/gateway 3.0.0

federation-v1/gateways/mesh-bun/package.json

@types/node ^20.4.5

graphql ^16.7.1

@graphql-mesh/graphql ^0.97.0

@graphql-mesh/runtime ^0.98.0

@graphql-mesh/utils ^0.97.0

@graphql-mesh/cross-helpers ^0.4.0

@graphql-mesh/cache-localforage ^0.97.0

@graphql-mesh/merger-stitching ^0.97.0

@graphql-mesh/http ^0.98.0

@graphql-mesh/store ^0.97.0

@graphql-mesh/types ^0.97.0

bun-types ^1.0.0

federation-v1/gateways/mesh-supergraph-bun/package.json

@types/node ^20.4.5

graphql ^16.7.1

@graphql-mesh/supergraph ^0.3.0

@graphql-mesh/runtime ^0.98.0

@graphql-mesh/utils ^0.97.0

@graphql-mesh/cross-helpers ^0.4.0

@graphql-mesh/cache-localforage ^0.97.0

@graphql-mesh/merger-bare ^0.97.0

@graphql-mesh/http ^0.98.0

@graphql-mesh/store ^0.97.0

@graphql-mesh/types ^0.97.0

bun-types ^1.0.0

federation-v1/gateways/mesh-supergraph/package.json

graphql 16.8.1

@graphql-mesh/cli 0.89.7

@graphql-mesh/supergraph 0.3.0

federation-v1/gateways/mesh/package.json

graphql ^16.7.1

@graphql-mesh/cli ^0.89.0

@graphql-mesh/graphql ^0.97.0

federation-v1/gateways/wundergraph/package.json

@wundergraph/sdk 0.184.0

graphql ^16.7.1

@types/node ^20.4.1

typescript 5.4.3

federation-v1/scenarios/constant-vus-over-time/package.json

vega 5.28.0

vega-lite 5.17.0

tablemark 3.0.0

tsx 4.7.1

typescript 5.4.3

graphql 16.8.1

@types/node 20.11.30

federation-v1/scenarios/package.json

tsx 4.7.1

typescript 5.4.3

graphql 16.8.1

@apollo/composition 2.7.2

@types/node 20.11.30

federation-v1/scenarios/ramping-vus/package.json

vega 5.28.0

vega-lite 5.17.0

tablemark 3.0.0

tsx 4.7.1

typescript 5.4.3

graphql 16.8.1

@types/node 20.11.30

Check this box to trigger a request for Renovate to run again on this repository

Complete metrics and stats reported to GitHub

RPS
CPU
RAM
Network data overview
Failures
Duration

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.