rebuilding everything every time is slow about ember-app-kit

[manoharank]

@stefanpenner Precompiling all template files is also slow. grunt-ember-templates now supports precompile option, we can use it in build:debug, but i am unable to split that task between build:debug and build:dist. Any idea?

[thomasboyt]

if we do that, we'll also need to use handlebars.js in debug and handlebars.runtime.js in dist. shouldn't be too difficult to set up with the dom munger task

[thomasboyt]

also, I'm trying it now and not seeing a noticeable performance boost, though to be fair I'm not using very many templates. worried it may not be worth the extra build complexity

[neojp]

Would this help? https://github.com/tschaub/grunt-newer

[stefanpenner]

@neojp it might be worth looking again, but last time i did it still suffered from more not removing files that have already removed.

[MajorBreakfast]

In the long run taking a look at grunt-newer might be an option. I don't think it's needed just now because the first step is to only rebuild the stylesheets when a stylesheet was changed etc. That should give us a 4x performance boost.

[MajorBreakfast]

#205 makes it faster

[stefanpenner]

the current thought is to investigate a proper pipeline, such as what brunch has to offer. Brunch currently does not support es6, which is something we will have to modify. Implementing our own pipeline via grunt (incremental or not) is quite painful and error prone.

[ahx]

I don't want to move back to rake-pipeline, but i think it solved this problem pretty good.

[stefanpenner]

@ahx now that we have settled many other problems, this is a high priority. As my largest apps cannot use EAK yet either.

http://brunch.io/'s pipeline is quite appealing.

@thomasboyt got some thoughts?

[thomasboyt]

not sure what the best way out here is.

brunch is neat, but has a smaller ecosystem and certainly cannot just replace our grunt configuration. brunch isn't a "runner" at all like grunt is, purely a pipeline - there could be no brunch test or brunch deploy, as far as i can tell.

brunch+grunt also sounds like its own special hell, and i'd like to avoid it if possible.

the fantasy has been a new pipeline that could use existing grunt tasks and reconfigure them at runtime (i.e. i changed app/foo.js -> re-run the transpile task with {src: "app/foo.js"}) but it sounds like that might be impossible within grunt; apparently other spikes in this direction have failed.

i'll look into a brunch plugin for es6 modules (even if we don't end up using it, it's something i think should exist anyways), but can't promise anything.

[pichfl]

I recently discovered that grunt-contrib-watch let's you be quite specific when it comes to running tasks - See grunt-contrib-watch - Compiling Files As Needed, maybe this could be a starting point.

I'm still pretty new to Grunt and EAK, so I'm not much of a help here yet.

[simonexmachina]

Can I just clarify, what was the issue with @mbixby's changes in #144? Also has anyone investigated grunt-newer? It looks promising to me but I don't (yet) have the grunt chops to make it happen. @MajorBreakfast seems to have some strong ideas and work in progress in #172 and #205, but #205 hasn't been merged yet.

Regarding "the fantasy" that @thomasboyt refers to, this seems to be exactly what grunt-newer attempts to do, and what @mxbixby was doing in #144.

I'd like to get an picture of where we're at with all these efforts so that I can help out.

[mbixby]

#144 reactively constructs the pipeline and correctly responds to created / deleted files, like what @thomasboyt is describing. I'm using it in-house, but it's out of scope of this project and probably even Grunt. You'll run into weird bugs with Grunt events.

From tasks/watch.js (note the deletion behavior and inferred paths):

// Subscribing task to file modifications events means that every
// time a relevant file changes, that task will be run. (Relevant
// file is a file amongst source (src) files for that task.)
// 
// If a file is deleted, the destination (dest) for the task will be
// automatically removed before the task is executed.
// 
// Futhermore, if you set `task.options.shouldFilterModifiedFile`
// to true, the scope of the task will be narrowed down only 
// to the modified file (with grunt filter function).
// 
// Initial file modifications events are fired by grunt-contrib-watch.
// Succeeding events are fired after a particular task modifies its
// destination path. This results in reactive chaining of tasks
// inferred from their source / destination paths.

[simonexmachina]

Yes I tried merging #144 with the current version of EAK from master and
ran into issues. I forward-ported the changes but didn't end up with
something workable.

I personally think it would be good for a branch or a fork to provide this.
Maybe @mxbixby's fork is it, but I'd prefer a branch that tracked along
with master.

Anybody have any thoughts about grunt-newer?

[MajorBreakfast]

I think it's always a good idea to look at the other solutions that are out there.

• Grunt offers great flexibility if you have to do something custom in some project. (i.e. import an excel file? => Write a quick task for that. No problem.)
• Also grunt has a very big community. The new stuff always comes to grunt first.
• And grunt ist very configureable because you can exactly define what runs when. Example: If you want to generate a sprite sheet this has to happen before you compile your SCSS because you need the SCSS file that's being generated in there. And only after all CSS files are generated something like autoprefixer can run.

These are grunt's strengths. Its weakness is that wiring everything up is very tedious and we're far from done, yet. We need someone who worked with brunch before and can tell us more. I'm not against moving away from grunt. But we should make damn sure that it's the right thing to do before we do it.

@aexmachina I have some things cooking, that's right. Next thing I'm going to contribute is a commit that eliminates some unnecessary file copying like copying the vendor directory. #169 The debug server can serve files from the real vendor directory directly just as well.

@pichfl I agree. We have to listen more closely to what our watch task has to say. I'm confident that we can improve on that as we go along.

[mbixby]

@aexmachina If you forward-ported #144, please make it public, I'll merge in my fixes.

On Tuesday 22 October 2013 at 07:06, MajorBreakfast wrote:

I think it's always a good idea to look at the other solutions that are out there.
Grunt offers great flexibility if you have to do something custom in some project. (i.e. import an excel file? => Write a quick task for that. No problem.)
Also grunt has a very big community. The new stuff always comes to grunt first.
And grunt ist very configureable because you can exactly define what runs when. Example: If you want to generate a sprite sheet this has to happen before you compile your SCSS because you need the SCSS file that's being generated in there. And only after all CSS files are generated something like autoprefixer can run.

These are grunt's strengths. Its weakness is that wiring everything up is very tedious and we're far from done, yet. We need someone who worked with brunch before and can tell us more. I'm not against moving away from grunt. But we should make damn sure that it's the right thing to do before we do it.
@aexmachina (https://github.com/aexmachina) I have some things cooking, that's right. Next thing I'm going to contribute is a commit that eliminates some unnecessary file copying like copying the vendor directory. #169 (#169) The debug server can serve files from the real vendor directory directly just as well.
@pichfl (https://github.com/pichfl) I agree. We have to listen more closely to what our watch task has to say. I'm confident that we can improve on that as we go along.

—
Reply to this email directly or view it on GitHub (#30 (comment)).

[stefanpenner]

@mbixby I've seen some related fixes go into gaze, which may resolve some of the earlier concerns/issues with your PR. This may get us closer to the goal of a proper pipeline.

[simonexmachina]

@MajorBreakfast I've just pulled in your changes and they've made a big difference, thanks and great work.

I've used Brunch before (last year) and ultimately we switched to the Rails asset pipeline because of problems/limitations. I think that Grunt's definitely the right tool for this project.

@mbixby, I'm afraid I threw it away - anyway it didn't work. It was only about 15 mins to handle the conflicts. Let me know and I can dig it out of my Git history, but that'd probably take as long as the merge did :)

[stefanpenner]

@MajorBreakfast has made some good progress here.

We are likely pushing our naive grunt based build system to its maximum. The faster we make it the more brittle it becomes.

The path forward will be utilizing broccoli or similar.