Frequent OOM kills - Isolated to`http2` - 50 GB of RAM usage on MacBook, causing system OOM about oha

[hatoo]

Thank you. It's very helpful!

I've succeeded in reproducing. I will work on this Saturday.

[hatoo]

Thanks for your report!

I had some investigation and I found some weird huge memory consumption for oha against a specific HTTP server (such as node18's https module) backend although this isn't exact same phenomenon for this issue.

Could you tell me about your server's technology? (language, library, etc..)

[huntharo]

Could you tell me about your server's technology? (language, library, etc..)

The pwrdrvr.com domain name above is pointed at an AWS ALB which supports both HTTP2 and HTTP1.1. The ALB, in turn, is pointed at a dotnet 8 Kestrel web server using HTTP1.1, which in turn is pointed at a proxy inside of Lambda functions using HTTP2, which is finally proxying the request to a Node.js app. The Node.js app is reading from DynamoDB and returning an average of 1.6 KB in payload size through the layers above.

But essentially, the problem appears to happen when speaking to an AWS ALB with HTTP2.

The local problem I had where it used 50 GB of RAM was potentially pointed at the Node.js app using HTTP2 or at the dotnet 8 Kestrel server using HTTP1.1 (I don't have HTTP2 enabled for the Kestrel server, but I can). The details are fuzzy on this because it only happened once so far.

I could probably setup an AWS ALB route for you that just returns a constant response string and I bet the issue will happen with that.

[huntharo]

I've got a solid lead now. I was running the code under the debugger and looking heal profiles using pprof.

What I noticed from normal operations is that memory accumulates steadily. The primary usage, I think, is coming from a vector that holds all the results, so that makes sense. Maybe that can be reduced if the detailed stats are not needed until the end, but maybe it cannot.

Then I was thinking that Node.js, locally, never sends a GOAWAY message on an http2 socket while the ALB likely does send that after a number of requests, say, 10,000 or 100,000 per socket.

I realized that the problem was likely when the sockets were gracefully closed by the server. To simulate what would happen in that case I just ctrl-c'd my node.js process and, sure enough, oha started racing at 800% CPU and RAM usage in the dev container went from 3% of total to 20%, 40%, 80%, then OOM kill.

tl;dr - To Reproduce

1. For tests with server: Start node app locally with TLS cert and http2 support (probably any other stack would be fine too, just have http2)
2. Start server, start http2 test: cargo run --release -- -c 20 -p 20 -z 5m --http2 --insecure https://host.docker.internal:3001/ping
   a. Ctrl-c the http2 server
   b. Observe memory usage of oha with top - It's going to start jumping rapidly until the process is OOM killed within a few seconds to 10s of seconds
   c. CPU usage will jump to 800%, if available
   d. UI becomes unresponsive and prints no further info
   e. UI cannot be ctrl-c'd
3. Do NOT restart server, start http2 test: cargo run --release -- -c 20 -p 20 -z 5m --http2 --insecure https://host.docker.internal:3001/ping
   a. If oha is started when the server is not running, it will report 20 refused connections and immediately exit. This is not the same as what happens if the connections are established but then lost.
4. Start server, start http1.1 test: cargo run --release -- -c 200 -z 10m --insecure https://host.docker.internal:3001/ping
   a. Ctrl-C the http server
   b. Observe that oha reports refused connections and remains responsive for http1.1 when the server goes away
   c. Observe that the UI remains responsive and can be ctrl-c'd

[huntharo]

I have submitted a partial PR that operates mostly similarly to the way that this is handled for HTTP1.1: #363

I have a couple to-dos on the PR description.