elm-live slow on linux and leaks memory about elm-live

[ashishnegi]

seems to be fixed.. with latest release.. :) 👍

[mdevlamynck]

@tomekwi Using elm-live 2.7.5 node 6.11.4 and npm 3.10.10 I still get this issue. Each time I edit a file the memory consumption and cpu usage of the node process and the reload time grows. It happens with any editor (even a simple touch on a file).

[ChristophP]

When it crashes this it the console output.

Success! Compiled 4 modules.                                        
Successfully generated public/elm.js
[4567] 6955ms        0B GET    304 /public/normalize.css

<--- Last few GCs --->

 4594325 ms: Mark-sweep 1346.7 (1438.0) -> 1346.3 (1438.0) MB, 1754.5 / 2.3 ms [allocation failure] [GC in old space requested].
 4595831 ms: Mark-sweep 1346.3 (1438.0) -> 1346.3 (1438.0) MB, 1506.1 / 2.6 ms [allocation failure] [GC in old space requested].
 4597545 ms: Mark-sweep 1346.3 (1438.0) -> 1353.1 (1415.0) MB, 1713.8 / 2.7 ms [last resort gc].
 4599204 ms: Mark-sweep 1353.1 (1415.0) -> 1360.0 (1415.0) MB, 1658.7 / 3.1 ms [last resort gc].


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x3aff575cfb51 <JS Object>
    1: tickDone(aka tickDone) [internal/process/next_tick.js:~30] [pc=0x13ca3f943] (this=0x3aff57504381 <undefined>)
    2: _tickCallback [internal/process/next_tick.js:102] [pc=0x13c89b4d8] (this=0x3aff575e1959 <a process with map 0x35f11e411671>)

==== Details ================================================

[1]: tickDone(aka tickDone) [internal/process/next_tick.js:~30] [pc=0x13ca3f943] (th...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [node]
 2: 0x10cec6c [node]
 3: v8::Utils::ReportApiFailure(char const*, char const*) [node]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [node]
 5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [node]
 6: v8::internal::DeoptimizationInputData::New(v8::internal::Isolate*, int, v8::internal::PretenureFlag) [node]
 7: v8::internal::LCodeGenBase::PopulateDeoptimizationData(v8::internal::Handle<v8::internal::Code>) [node]
 8: v8::internal::LChunk::Codegen() [node]
 9: v8::internal::OptimizedCompileJob::GenerateCode() [node]
10: v8::internal::Compiler::FinalizeOptimizedCompileJob(v8::internal::OptimizedCompileJob*) [node]
11: v8::internal::OptimizingCompileDispatcher::InstallOptimizedFunctions() [node]
12: v8::internal::StackGuard::HandleInterrupts() [node]
13: v8::internal::Runtime_StackGuard(int, v8::internal::Object**, v8::internal::Isolate*) [node]
14: 0x13c3092a7
Aborted (core dumped)

[ChristophP]

More Details: I am run this command elm-live Main.elm --open --pushstate --output=public/elm.js -- --debug through npm start. Node Version 6.9.2, npm version 3.10.9 any help appreciated.

[tomek-he-him]

Hey man, I’m sorry you’re experiencing this. I used elm-live on Fedora a lot back in the day – and nothing worrying happened.

One thing comes to mind though. IIRC, we used gaze for file watching before we switched to chokidar. Don’t remember what the issue was back then, but I’ll go ahead and find the conversation for you.

[tomek-he-him]

Found the PR: #16. The problem it tackled though was very different and I doubt it’s related to the problem you’ve posted.

Just one more question: did you build node yourself, install it from apt repos, or are you using official builds through something like nvm or n?

[ChristophP]

Hey thanks for your reply. I checked the other thread but yeah i am also not sure that it's a related problem. I installed node via apt repos and am not using nvm.
I have a hard time reproducing the behavior consistently though, but i happens a lot of times when I just code away. Just saving the same file mutliple times doesn't seem to be a problem.
Just recently I had a similar issue with a non elm project that used a the watch package for file watching. It worked on the mac and crashed on ubuntu. I figured out that for some reason ubuntu couldn't handle the fact that node_modules where also being watched. After I specified an ignore regex the problem dissappeared. Unfortunately elm-live doesn't have such an option yet. Is this something that is planned in the future?
Also I have another question about what exactly elm-live is watching. I know it watches

1. static assets
2. and elm files
   But which directories are being watched and are they watched deeply(recursively)? Can they be different directories?
   To find out if it's the node_modules that are causing the problem I changed my project structure from this

project
- Main.elm
- node_modules
- package.json
- index.html

To this:

project
- src
  - Main.elm
- public
  - index.html
- node_modules
- package.json

Then I try to run elm-live from the src directory only so it will not watch node_modules but also want it to watch the public directory (with the --dir argument) for reloading when assets change.
Any help would be appreciated.

[tomek-he-him]

I remember having weird problems when a friend compiled node from source on gentoo (or perhaps arch linux, can’t remember anymore). Turns out he was missing an obscure compilation flag, which disabled some sort of optimization.

I think people managing apt repos compile programs from source as well. Might be a similar problem perhaps?

Anyway, to answer your questions:

What exactly is elm-live watching?

elm-live itself is only watching **/*.elm files in the current working directory, including any inside elm-stuff and node_modules.

But, elm-live also spawns budo as a static livereload server, and AFAIK it watches *.html, *.css and *.js files inside --dir and triggers the livereload on change. Possibly, only files actually requested by the browser are watched.

Is specifying an ignore regex planned in the future?

Sure, if folks actually need it 🙂 I’d love elm-live to have as few configuration options as possible, covering most use cases out of the box. If we don’t find a no-config solution, this will be the way to go.

Any help would be appreciated.

Did you try running elm-live from src with --dir=../public?

[ChristophP]

Cool, thank you for the clarification. I tried running it from the src directory but when I do that it does not find the elm packages because elm-stuff is in the parent folder. Then it I am being prompted if I want to install the required packages in the current one. Which I don't :-/

[ChristophP]

So I have done all of the previous stuff with a globally installed version of elm-live. I just installed a local one and haven't encountered the issue so far. I will test a bit more but maybe it something to do with that.

[ChristophP]

It is weird however, when I start elm-live it reloads and compiles multiple times right away. Here's the console output. It reloads 3 times because stuff in my node_modules changed. I don't really change anything in there, all I do is use some symlinks from my public dir to my build dir, where the compiled files(elm -> js, less -> css) are.

elm-live:
  You’ve added `public/index.html`. Rebuilding!

Success! Compiled 1 module.                                         
Successfully generated build/elm.js

elm-live:
  You’ve added `public/style.css`. Rebuilding!

Success! Compiled 1 module.                                         
Successfully generated build/elm.js
[0002] 31ms       394KB GET    200 /elm.js

elm-live:
  You’ve added `node_modules/.bin/strip-indent`. Rebuilding!

Success! Compiled 1 module.                                         
Successfully generated build/elm.js

elm-live:
  You’ve added `node_modules/.bin/which`. Rebuilding!

Success! Compiled 1 module.                                         
Successfully generated build/elm.js
[0003] 9ms           0B GET    304 /
[0003] 41ms          0B GET    304 /style.css
[0003] 61ms       394KB GET    200 /elm.js
[0003] 15ms          0B GET    304 /img/ballistic-logo.png

elm-live:
  You’ve added `node_modules/elm/node_modules/.bin/uuid`. Rebuilding!

Success! Compiled 1 module.                                         
Successfully generated build/elm.js

elm-live:
  You’ve added `node_modules/watchify/node_modules/.bin/browserify`. Rebuilding!

Success! Compiled 1 module.                                         
Successfully generated build/elm.js
[0004] 6ms           0B GET    304 /
[0004] 15ms          0B GET    304 /style.css
[0004] 26ms       394KB GET    200 /elm.js
[0005] 0ms           0B GET    304 /img/ballistic-logo.png

What do you say to that? I think it really shouldn't listen to node_modules changes. But I am also not quite sure why I have so many changes in my node_modules in the first place :-) It isn't keeping me from using elm-live I just noticed more reloading than I was expecting(I still love elm-live :D )

[tomek-he-him]

Yeah, it’s really weird – shouldn’t be picking up those changes. We had had problems with symlinks before, but I’ve been sure we solved them back then. Can you look at #74 and see if it seems similar?

Also, which version of elm-live are you using?

[ChristophP]

Hey, sure. So the issue #74 seems similal in the sense that there seems to be more rebuilding and reloading going on than needed, but different because it isn't an endless loop for me. Looking at the output that I am getting these lines seem to be triggering that.

You’ve added `public/index.html`. Rebuilding!
You’ve added `public/style.css`. Rebuilding!
You’ve added `node_modules/.bin/strip-indent`. Rebuilding!
You’ve added `node_modules/.bin/which`. Rebuilding!
You’ve added `node_modules/elm/node_modules/.bin/uuid`. Rebuilding!
You’ve added `node_modules/watchify/node_modules/.bin/browserify`. Rebuilding!

The first two are my files, but the others seem to be dependecies of my dependencies.
I am using elm-live 2.7.4 so I already have the symlink patch.

[tomek-he-him]

Hey @ChristophP, apologies for the delay – this must have slipped through inbox.

This is very weird – even the two first files shouldn’t really be picked up by the watcher. Let alone the others. Is the project open source so we can try and reproduce this on other computers?

[ooesili]

Linux archbook 4.9.11-1-ARCH #1 SMP PREEMPT Sun Feb 19 13:45:52 UTC 2017 x86_64 GNU/Linux

I'm getting a issue on Linux as well. Elm live starts up great, but after a handful of saves, it starts slowing down. Starting at 14ms to load / and eventually going into the 100s and 200s.

@ChristophP Are you by any chance using vim? It does some weird stuff when saving a file by default, and I'm wondering if it's contributing to this issue.

[ChristophP]

Hey guys, sorry for the late response. @tomekwi unfortunately it isn't open source, and not something I can make available to you. I'll see if I can strip it down so I can show it to you.
@ooesili Yeah I am using vim. You think that could be related?

[ooesili]

@ChristophP I think that Vim might be making it worse, but probably not the underlying issue. The reason I got suspicious is because the page reloads significantly faster when I just touch src/Main.elm than saving the file in Vim. I found this interesting stackoverflow question about what happens when a file is save in vim.

I'm playing around with a file watching library in Go right now (I'm trying to understand the domain a little more and I'm most comfortable in Go) and I see this when I save a file in Vim:

"src/4913": REMOVE
"src/Main.elm": RENAME
"src/Main.elm~": CREATE
"src/Main.elm": CREATE
"src/Main.elm": WRITE
"src/Main.elm": CHMOD
"src/Main.elm": CHMOD
"src/Main.elm~": REMOVE

My initial thought is that chokidar or budo might be leaking watches, or budo and chokidar are fighting over the same files since they both seem to do watching.

I'm not super savvy with Node, so I'm kind of terrified to go trudging through hundreds of dependencies to track town a performance issue, but I'm going to give it my best shot. I'm going to look over this article this weekend and see if I can find something.

Side note, I usually run the elm tools inside of a docker container with the current directory mounted into the container, and when I do that, the slow down becomes much worse (into the thousands of milliseconds). I don't know if that's useful information, but it might spark some ideas, so I thought I'd share.

[tomek-he-him]

Hey @ooesili, thanks so much for the help with this! Please keep us updated and let me know if I can help in any way. Also, if you can help me reproduce this issue for example by posting a docker image I can repro this on my mac with, I’d really appreciate it.

[ooesili]

https://hub.docker.com/r/codesimple/elm/

[tomek-he-him]

@ooesili Thanks! An image with code inside and steps to reproduce the problem would be of great help.

[tomek-he-him]

For the record: thanks to @Natim, we’ll soon have one half of the problem worked around. Changes to symlinks inside node_modules should no longer be picked up once we merge that PR. Feel free to fork [email protected]:Natim/elm-live and check out fix-elm-watch to see if it makes things easier to reproduce.

[tomek-he-him]

#104

[ooesili]

I'm having a really hard time packaging up a reproducible example. Filesystem watching is a very OS-specific thing, so it's a not really straight forward thing to reproduce, especially since you're on a Mac. I'm experiencing this behavior running on Arch Linux with node v.7.7.3, elm 0.18, and elm-live 2.7.4. I tried creating an ubuntu based Vagrant box with a tiny elm app, but I don't experience the slowdown there.

[tomek-he-him]

@ooesili If the issue you’re having is related to a large number of watched files, the solution we’re currently considering at #104 (comment) might help. Gimme a tick for a quick test.

[tomek-he-him]

@ooesili Could you please npm install --global tomekwi/elm-live@39acd54 and check if that fixes the file watching issue?

[n1k0]

I'm hitting the same issue as described above, I'm using Ubuntu 16.10, node v7.8.0, npm v4.5.0 and elm-live v2.7.4.

I tried to install the version mentionned in the previous comment with npm install --global tomekwi/elm-live@39acd54, but it failed with this error:

npm ERR! addLocal Could not install /home/niko/work/tooty/tomekwi/elm-live@39acd54
npm ERR! path /home/niko/work/tooty/tomekwi/elm-live@39acd54
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall open
npm ERR! enoent ENOENT: no such file or directory, open '/home/niko/work/tooty/tomekwi/elm-live@39acd54'

I'd really like to help with this bug, starting with providing feedback with possible fixes. How can I help?

[tomek-he-him]

Yayyy! Seems like vim was actually using symlinks there then. The biggest change in the latest release was fixing the way elm-live handles symlinks. More info in #104.

@ChristophP can you confirm this is now fixed?

[abradley2]

Yeah, having this issue as well. At first I thought it might be elm-format making my editor slow but I noticed the compile times on save were matching my slowing speeds everywhere else.

I'm using elm-live 2.7.5, node v 9.4, Ubuntu 16.04

Let me know if there's any other information I could provide to help

[tomek-he-him]

Hey hey @mdevlamynck @abradley2, I’m terribly sorry for the radio silence on my end! Totally my bad, I’ve been super busy recently. If you’re watching me on twitter or linkedin, you might have noticed that alongside a super intensive full time job, I’m also starting a new business. Not to mention my lovely little family that deserves my time more than anything else 😊

I’ll be looking for new maintainers of elm-live – so if any of you guys know anyone that’s keen to take over the project, let me know! 🚀

Did you manage to work around the issue?

[wking-io]

We recently launched 0.19 update and updated a couple architecture decisions. Would love to know if these issues still exist!

[mdevlamynck]

@wking-io I can't reproduce the slow down and cpu usage anymore. Memory usage can rise slowly if you spam changes but when the changes are sparse enough (as in a user makes the changes and not a script made to spam changes) the GC does its job and keep memory usage pretty low.

So I'd say this release fixed the issue, at least for me. Thanks!

[wking-io]

Okay closing for now! If this issue remains please open a new issue referencing this one and we will go from there. 👌