Tool Confusion about node-memwatch

[lloyd]

Tough one. Afaik certain objects allocate external memory not managed on nodes heap. I think buffers work this way, but if you were leaking buffers you would see a small leak reported in heap size growth, and a larger one when watching rss size.

I would recommend putting a critical eye on any binary components you are using.

If you look through v8 headers you can find v8 function calls that "register" external memory.

This sounds like a fun debugging adventure.

-- lloyd (thumb-typing)

On Feb 2, 2013, at 5:22 PM, Evan Tahler [email protected] wrote:

Hi!
I have a node app with a memory leak. I know this because my OS thinks that the 'node' application keeps consuming more and mor ram. I've been trying in vain to find the source of this for some time now with no luck. Using node-memwatch, I am consistently reported:

after:
{ nodes: 93563,
time: Sat Feb 02 2013 16:20:25 GMT-0800 (PST),
size_bytes: 21457040,
size: '20.46 mb' },
from a new memwatch.HeapDiff() which I run periodically. The size fluctuates between 20 and 30 MB, which is fine for my application. However, the OS thinks that the real memory consumption of the app keeps growing, all the way up to 1GB.

What might the cause of the differing reports be? I've tried the same app with node-memwatch on both OSX and Solaris/Joyent-smartmachine

—
Reply to this email directly or view it on GitHub.

[evantahler]

Heh, it's a fun one!

Your note led me to this StackOverflow article, which seems REMARKABLY relevant. Thanks fot the tip! I'll close this issue.

http://stackoverflow.com/questions/14065611/nodejs-how-to-free-buffers-that-get-allocated-outside-of-the-v8-memory-heap

[lloyd]

Nice! Glad you have a lead, curious to hear what the issue was when you resolve it.

Maybe grepping your heapdiff for 'Buffer' could give you a quick confirmation? (Objects themselves are tiny, but can reference arbitrarily large chunks of memory)

-- lloyd (thumb-typing)

On Feb 2, 2013, at 5:56 PM, Evan Tahler [email protected] wrote:

Heh, it's a fun one!

Your note led me to this StackOverflow article, which seems REMARKABLY relevant. Thanks fot the tip! I'll close this issue.

http://stackoverflow.com/questions/14065611/nodejs-how-to-free-buffers-that-get-allocated-outside-of-the-v8-memory-heap

—
Reply to this email directly or view it on GitHub.

[evantahler]

I'm working on the task system for http://actionherojs.com

From the link above, it looks like the way the taskProcessors have been working do, in fact, recurse upon themselves (work, sleep, work, sleep, ... ) via internal recursion (self.work(self.sleep(self.work))). It looks like we were clearing self.currentTask properly, but another reference (task) to object being worked off was staying around in limbo. Just throwing down a task = null; at the end seems to be the fix and clear up any references the task may have made (which often include buffers to send email, etc)!

Initial tests look positive, but I'm going to let this thing run overnight and see if my laptop catches on fire. Refactoring the taskProcessor to not use recursion is a whole other problem :D

Check out https://github.com/evantahler/actionHero/blob/master/initializers/taskProcessor.js