Speed up with multi-threading?

[danfinlay]

Hi there, I've got a pretty large app I'm working on, and I'm getting pretty long build times (~7000ms at times). I was wondering if you thought it might be practical to use child_process.fork() or the cluster module to split the build process up among a computer's full processor.

This blog post is a pretty good introduction to the methods, I'm not adverse to trying to implement this myself, but I'd like some help grasping the broccoli architecture to help me do it.

Where, for example, is the individual file processing assigned?

[stefanpenner]

Unfortunately, only workloads that are actually CPU intensive would be helped. Currently the vast majority of time (unless you are using ruby-sass or similar) is in pure IO. This is due to a data-loss quickfix for osx, and is on the roadmap to be solved.

Additionally, as of right now there is no way (without introducing some native extention) to utilize multiple user-land threads in a single node process. I suspect you use the word multithreading as a mistake, but instead meant multi-process

[danfinlay]

What about using a source map to only edit portions of the output that are updated? (server-mode only)

[stefanpenner]

I believe your idea isn't bad, but it ultimately wouldn't help enough right now. Or would require a different approach then the current architecture. The biggest problem IO sensitivity, has a known solution: see:

6b0a9d3

Honestly this accounts for more the 90% of all time spent in broccoli, once restored broccoli will again be incredibly fast.

If you have energy to help with that effort, it would yield fantastic returns.

[danfinlay]

Do you have a link to a description of the data loss quickfix for osx? I'm not familiar with the problem.

Also, if it's being worked on, I can't wait! Is that branch already somewhat working? Worth trying out in development?

[stefanpenner]

@FlySwatter I believe noone has time currently, but @rjackson and @joliss hope to work on it when they have time. I am sure they would be happy if someone can drive it to completion soon.

Additionally, once that is resolved, I suspect your class of optimization will be once against important.

[stefanpenner]

I would also like to push broccoli to a background process, I would love if broccoli did this automatically.

[danfinlay]

For the sake of others who might be willing to pitch in, could you explain / link to the problem in more detail?

[chnn]

I'm also curious to learn more about this and would like to help out if at all possible.

[stefanpenner]

quick GH search reveals: #88

[chnn]

thanks

[joliss]

Do you have a link to a description of the data loss quickfix for osx? I'm not familiar with the problem.

We had to stop using hardlinks; it's described here: https://github.com/broccolijs/broccoli/blob/master/docs/hardlink-issue.md

We're most likely going to switch to using symlinks, as the commit Stef linked alludes to, Coming soon.

[joliss]

Re the original question, I wrote down some thoughts on the problems with parallelizing stuff http://www.solitr.com/blog/2014/02/broccoli-first-release/, section "No Parallelism".

[joliss]

I'm getting pretty long build times (~7000ms at times)

For what it's worth, this seems generally acceptable for an initial build, but obviously unacceptable for incremental rebuilds. (I'm not clear on which one it is for you.)

We clearly want to keep working on performance, this is super important. Parallelizing seems like a workaround (with limited gains) - I really want to solve all the fundamental problems instead.

[rwjblue]

FYI - I spoke with @FlySwatter in IRC and was able to reduce rebuild times from 7s to 1s by removing unused folders from vendor/.

[rwjblue]

Problem

Currently, for mergeTrees we are copying all files for every input tree into the destination directory. This breaks down fairly quickly with either a large number of files or a small number of very large files.

@joliss and I (with help from many others) have been working on a good solution to this fundamental problem.

Large Files

I am actively working to move from copying to symlinking which by itself will remove the large file speed impact (since making a symlink is constant time regardless of source file size), but does not directly address the slowdown for large numbers of files (although it would likely be a bit better).

Large Numbers of Files

The way we plan to handle the large file count issue is by symlinking the root directories directly if the directory does not already exist in the destination directory. This will reduce the need to stat all files in each input tree, and allow symlinking a much smaller number of directories since the vast majority of merges are not using the same directories (especially true for vendor/ in Ember CLI).

Timeline

This is a hard one, but I'm hoping to have the rough changes made and ready for detailed review by @joliss early next week.

Plugin Impact

Changing mergeTrees to use symlinks has fairly large implications for other plugins. We have to change node-walk-sync to recurse into symlinks. Thankfully, the majority of plugins that inherit from broccoli-filter can simply update their version dependency, but many other plugins that may be doing their own directory traversal will have to ensure that they also traverse symlinks.

There are likely other considerations (as far as symlinking impact), feel free to discuss other impact here.

[kumavis]

if unwatched vendor trees were causing such long build times, this PR may deserve more attention

[stefanpenner]

more updates: broccolijs/broccoli-merge-trees#11

lots of great perf!

[stefanpenner]

I suspect this can be closed for now. In most apps the bottleneck is in slow filters themselves, and rarely an issue on incremental builds.

Some future work can be done to maybe "fast-boot" in perfect conditions, but this seems like a lower priority them improve some slow filters like esnext.

[stefanpenner]

I suspect once those concepts are thoroughly exhausted multi-process might need to be investigated.

[insidewhy]

@FlySwatter sighjs is a build system much like broccoli that can also delegate tasks to multiple CPUs.

[danfinlay]

Nice to know!