Server leaks memory (Socket objects) about websocket.io

[rauchg]

Interesting, will take a look soon. Thanks nico

[nicokaiser]

This is an example for v8-profiler / node-inspector:

https://gist.github.com/1390119

http://cl.ly/2r0H3Z0o0K0d2T1Q0C3K

I called "node --debug server.js" (Node 0.4.12, as 0.6 seems to have problems with v8-profiler), took a "Snapshot 1" in node-inspector, ran client.js (which opens 200 concurrent connections) and took the "Snapshot 2".

You can see the leaked objects in the screenshot (Count = 200). This counts will not grow if I re-run client.js, but if I change the concurrency in client.js.

[nicokaiser]

Any news on this? Did not try to profile the server in its newest version, but when switching from node-websocket-server to websocket.io, the server has severe memory leaks...

[nicokaiser]

Don't know if this is "ws" module related. However I updated to the latest version of ws (0.4.3) and still experience memory leaks.

[rauchg]

I'll take a look at this asap, since we're switching our production deployment to websocket.io soon

[nicokaiser]

Great! Let me know if there is anything I can provide to help you.

[rauchg]

I think you already did a great job at supplying the debug/profiling information. That's all I need for now, thanks :)

[einaros]

I've slowly but surely begun dealing with resource leaks in ws -- or rather seeing if there are any there at all.

In terms of websocket.io, the only pieces of ws in use are the sender and receiver scripts. They should be fairly well time tested, seeing as much of their code has been running inside of socket.io for a few months now, but again -- this is an area of ws which still needs some work. I appreciate others looking into it as well.

[nicokaiser]

I'll try to provide more detailed profiling information during the next few days.

While experimenting with websocket.io / ws I discovered some strange things – the server uses about 200mb memory with 20k clients and node-websocket-server (pretty constant, freeing up space when clients are gone).

However with websocket.io or ws.Server memory consumption grows to about 1G per day (which is not freed). As with NWS this does not happy I suspect the modules, but I can be wrong... hard to debug. (Sidenote: 90% of the clients are hixie-76).

(nb: when using Node 0.6 instead of 0.4, even NWS' memory usage grows, which makes the whole thing even more frustrating...)

[nicokaiser]

While trying to track down my still existing memory leak problem with WebSocket.IO (and WS as a WebSocket server module as well), I created this simple server:

https://gist.github.com/2031620

Then, I put a simple WebSocket connect call to a website (in the background).

In about 1 hour, 4.000 connections were made, about 500 concurrent. The RSS went from 13 MB (heapUsed: 4 MB) to about 50 MB (heapUsed: 15 MB).

Then I removed the script from the website and disconnected the remaining clients by calling socket.close() (for every socket in server.clients[]).

With 0 remaining clients, the server still consumes 43 MB RSS (heapUsed: 10 MB).

Looking at the v8-profiler's snapshots, I can only see many closures (about 28.000) staying there:

After startup: http://cl.ly/4245061P0y2i0w382Z3I
With 500 concurrend clients: http://cl.ly/470e2N0e0W0y0q2M1v19
After closing all clients (and waiting a few minutes for the GC): http://cl.ly/0n040w2m3u2w111y0201

[einaros]

Which version of ws is running under that setup?

[nicokaiser]

WebSocket.IO from GitHub master, ws 0.4.8.

[einaros]

And the connections you're testing with in this case are all hybi?

[nicokaiser]

No, they are mixed, about 50/50%.

But the leaks also happened on the live server (which I cannot debug) when it hat 100% hixie connections (hybi protocols disabled in websocket.io)

[einaros]

Would you be able to do a hybi-only test? I've had trouble reproducing this for my tests (which have targetted hybi).

In either case I will spend time diving into this come next week. I'm currently knee-deep in another project, which I'll have to finish before taking on anything else.

[nicokaiser]

I can try to run this test again (it's difficult as I'm operating on the live server and using one of our websites as botnet ;-)) with only hybi connections (disabling the "default" transport in websocket.io should do it?).

[einaros]

To isolate the issue to ws, it'd actually be best if this test ran through ws directly, rather than through websocket.io->ws. Is this doable for you? Otherwise, any insight would help, but as I've said I'll put some effort into this early next week. If it's in ws I'll find it quickly.

[nicokaiser]

Yes, I'll change the server to use ws directly. Until you're ready I can try to provide you access to the server maybe.

I experienced the leaks also while using ws directly for the "big leaking live server process" ;-), however also with mostly hixie connections.

Thanks in advance for your help!

[einaros]

I wouldn't be awfully surprised to learn that ws' hixie parsers were leaking somewhat. The hybi bits, however, I do expect to behave pretty well.

[nicokaiser]

I'm analyzing the RSS problem over here: websockets/ws#43

[tskimmett]

I know this was a while ago, but this is a pretty serious issue. Has any progress been made in solving it? Based on my experience, ws.io does not seem to cleanup WebSockets that have been closed. Though a manual destroy() inside the 'close' handler for the WebSocket might do the trick (I'm still testing this).

[rauchg]

We switched to ws as it started supporting the older drafts as well