Comments (5)
Thank you. It's very helpful!
I've succeeded in reproducing. I will work on this Saturday.
from oha.
Thanks for your report!
I had some investigation and I found some weird huge memory consumption for oha
against a specific HTTP server (such as node18's https module) backend although this isn't exact same phenomenon for this issue.
Could you tell me about your server's technology? (language, library, etc..)
from oha.
Could you tell me about your server's technology? (language, library, etc..)
The pwrdrvr.com
domain name above is pointed at an AWS ALB which supports both HTTP2 and HTTP1.1. The ALB, in turn, is pointed at a dotnet 8 Kestrel web server using HTTP1.1, which in turn is pointed at a proxy inside of Lambda functions using HTTP2, which is finally proxying the request to a Node.js app. The Node.js app is reading from DynamoDB and returning an average of 1.6 KB in payload size through the layers above.
But essentially, the problem appears to happen when speaking to an AWS ALB with HTTP2.
The local problem I had where it used 50 GB of RAM was potentially pointed at the Node.js app using HTTP2 or at the dotnet 8 Kestrel server using HTTP1.1 (I don't have HTTP2 enabled for the Kestrel server, but I can). The details are fuzzy on this because it only happened once so far.
I could probably setup an AWS ALB route for you that just returns a constant response string and I bet the issue will happen with that.
from oha.
I've got a solid lead now. I was running the code under the debugger and looking heal profiles using pprof.
What I noticed from normal operations is that memory accumulates steadily. The primary usage, I think, is coming from a vector that holds all the results, so that makes sense. Maybe that can be reduced if the detailed stats are not needed until the end, but maybe it cannot.
Then I was thinking that Node.js, locally, never sends a GOAWAY message on an http2 socket while the ALB likely does send that after a number of requests, say, 10,000 or 100,000 per socket.
I realized that the problem was likely when the sockets were gracefully closed by the server. To simulate what would happen in that case I just ctrl-c'd my node.js process and, sure enough, oha
started racing at 800% CPU and RAM usage in the dev container went from 3% of total to 20%, 40%, 80%, then OOM kill.
tl;dr - To Reproduce
- For tests with server: Start
node
app locally with TLS cert and http2 support (probably any other stack would be fine too, just have http2) - Start server, start http2 test:
cargo run --release -- -c 20 -p 20 -z 5m --http2 --insecure https://host.docker.internal:3001/ping
a. Ctrl-c the http2 server
b. Observe memory usage ofoha
withtop
- It's going to start jumping rapidly until the process is OOM killed within a few seconds to 10s of seconds
c. CPU usage will jump to 800%, if available
d. UI becomes unresponsive and prints no further info
e. UI cannot be ctrl-c'd - Do NOT restart server, start http2 test:
cargo run --release -- -c 20 -p 20 -z 5m --http2 --insecure https://host.docker.internal:3001/ping
a. Ifoha
is started when the server is not running, it will report 20 refused connections and immediately exit. This is not the same as what happens if the connections are established but then lost. - Start server, start http1.1 test:
cargo run --release -- -c 200 -z 10m --insecure https://host.docker.internal:3001/ping
a. Ctrl-C the http server
b. Observe thatoha
reports refused connections and remains responsive for http1.1 when the server goes away
c. Observe that the UI remains responsive and can be ctrl-c'd
from oha.
I have submitted a partial PR that operates mostly similarly to the way that this is handled for HTTP1.1: #363
I have a couple to-dos on the PR description.
from oha.
Related Issues (20)
- Display help text when passed no options HOT 1
- MacOS Crash issues HOT 7
- DNS Lookups == Number of Concurrent Connections HOT 1
- Timed out requests not reported at all in stats HOT 2
- Utilize M1 Mac GitHub actions
- Layout is broken
- Make relevant options mutually exclusive HOT 1
- Don't DNS lookup when connecting UNIX socket or VSOCK
- OHA sometimes causes the Macbook M2 to freeze and reboot. HOT 3
- About version scheme HOT 4
- Rewrite `justfile` into `bun shell` or `dax` or something
- Add HTTP pipelining support HOT 1
- Failed to install using cargo on windows HOT 2
- Support termux , DNS resolv HOT 2
- Failed to compile v1.4.3 on rustc 1.76.0 HOT 1
- oha Potential Memory Leak HOT 2
- Explain JSON format on readme
- Need to set `Accepted-Encoding` header HOT 1
- (Question) how to pass dinamic specific values? HOT 4
- How do I use post requests HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from oha.