Parallel Suites

[aearly]

This feature allows you to specify that a suite's child suites will be run in parallel. Here is a code example:

describe("parallel suite parent", function () {
  this.parallel(true);

  it("this test will be called in series before the child suites");

  describe("parallel suite 1", function () {
    it("will be run in parallel to suite 2", function(done) {
      someAsyncFunc("foo", function (err, res) {
        //some assertion
        done();
      });
    });
  });

  describe("parallel suite 2", function () {
    it("will be run in parallel to suite 1", function(done) {
      someAsyncFunc2("bar", function (err, res) {
        //some assertion
        done();
      });
    });
  });

  after(function () {
    // this will be called after the child suites complete
  });
})

I realize that this might be a bit out of scope for mocha, as it goes against some of the design decisions made in the past. But, nonetheless I had a use case for parallelism, I didn't want to abandon the other features of mocha, and there is some demand for parallelism from others, so I patched it in. This is related to issues #3, #19, #81, #98, #235, #309

Doing it at the Suite level seemed to be the most logical way to do it. Tests within a suite are still run in series. Here's a high level overview of the changes:

• Change Runners to wrap only a single suite, create child Runners for child Suites, inheriting the proper properties along the way (globals, bailing, etc....)
• Add a parallel property to Suite
• If the current Runner's Suite is parallel, run child Runners in parallel, otherwise run in series
• Have the parent Runner proxy events from child Runners, re-emit them immediately if running in series, or batching them if in parallel, replaying the sequence at the end. This prevents jumbled and interleaved reporting
• Add a --parallel CLI flag to run all root-level suites in parallel, if you want.

This also encompasses my previous pull request, #836.

[nbrownus]

Any exceptions that are thrown in an asynchronous test are assigned to the very first runnable item, this change would force everyone to catch their assertion errors and pass them to done manually.

[aearly]

You are right, this is an oversight on my part -- it didn't show up in my testing. This can be solved by using domains, but then mocha will not run properly in the browser. My fork of mocha would be Node-only, or require explicit passing of the error to done().

[nbrownus]

Also just ran into an issue where mocha no longer exits with a non zero on test failure. Seems to be related to nested Runners but haven't dug very deep just yet.

[aearly]

I will try to fix these additional issues before the end of tomorrow. I think I will have suites always run in series in the browser regardless of the flag, since there is no way to add domain-like behavior.

[nbrownus]

The more I think about this the more I feel like defining a suite as parallel should impact how it is run with its siblings and not how its children are run. Going that way would also fix a bug where if you set a suite without children suites as parallel the whole thing fails to output anything of use and hangs.

[aearly]

What do you propose the UI should look like exactly? Either way, the parent suite will have to know what to run in parallel and what in series.

[tj]

sorry :( I dont want to go down that road with mocha, "expresso" was fully parallel before and it causes nothing but issues, and in many cases it's not any faster, in most of my experience it usually results in slower tests (depending on case of course) due to thrashing of various kinds. Consistency and predictability is definitely preferred here over slight performance gains. Some people might be interested in a fork though I'm not sure

[aearly]

OK, that is fine. I will probably continue to maintain the fork, as I'm using it pretty heavily on a project.

[aearly]

FYI, I am finding that domains are not adequate to handle all asynchronous errors. See nodejs/node-v0.x-archive#3908

It's frustrating, because I cannot create a simple test case that reproduces the error I get when running a complicated suite. All the node core methods are fairly fastidious about binding their callbacks to the current domain. I'm going to investigate using separate processes, but in the meantime, parallelism is better achieved outside of mocha itself.

[markstos]

@aearly, there is a "Parallel Mocha" project, https://github.com/atsuya/parallel-mocha I have not tried it yet, but I'm also interested in the concept.

There's a test runner for Perl called prove which has options to run tests in parallel, and the option to exec other commands to run an individual test file which would produce TAP output. Mocha provides a "TAP reporter" option. It would not matter that prove is written in Perl. You would just be using it as a binary. I have not tried this approach yet... I just thought it. There would be some extra overhead from forking and loading node and common 'requires' for each test, so you'd have to do some benchmarking to see if it's actually a win.

[nbrownus]

I ended with doing a mocha compatible project that supports both promises and parallelization if anyone is interested. There are some concurrency controls and neat reporters to show how things are scheduled and ran when you run into issues.

https://github.com/nbrownus/ppunit

[aearly]

Interesting, I'll have to check it out. I see you're using domains for
error handling -- I wonder if the issue I had last year is resolved. It's
been on my todo list to write a proper multi-process BDD test runner, but
if yours works off the shelf...

Anything is likely better than the multi-process frankenstein monster I
made by wrapping mocha.

On Mon, Apr 14, 2014 at 2:29 PM, Nathan Brown [email protected]:

I ended with doing a mocha compatible project that supports both promises
and parallelization if anyone is interested. There are some concurrency
controls and neat reporters to show how things are scheduled and ran when
you run into issues.

https://github.com/nbrownus/ppunit

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/849#issuecomment-40420635
.

[tj]

probably not modularizing your projects much if the tests are that slow :p the original expresso test runner was parallel but it just causes far more issues than you gain, but it's easy to parallelize via modularity and CI

[aearly]

Well, if each suite launches a functional test on a Sauce Labs browser,
it's going to be slow no matter what. As I said, frankenstein monster. :)

On Mon, Apr 14, 2014 at 3:17 PM, TJ Holowaychuk [email protected]:

probably not modularizing your projects much if the tests are that slow :p
the original expresso test runner was parallel but it just causes far more
issues than you gain, but it's easy to parallelize via modularity and CI

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/849#issuecomment-40425067
.

[nbrownus]

Certainly agree for unit testing @visionmedia. Our issues with serialized tests were based around full suite functional testing. Only a few tests needed to run in isolation while the rest could be parallelized. Using CI to solve it would have caused us to engineer mutexes outside of the test runner. Fun experiment either way.

[rprieto]

I agree as well, all our unit & integration tests run in matters of seconds. We do have one specific test suite though that would benefit from parallel tests:

This test suite crawls our API using hypermedia & makes assertion on data integrity. This means making HTTP calls, discovering other related links to crawl, and creating new describes and its dynamically. This works very well, but all HTTP calls are made in sequence. 5000 links * 100ms is close to 10mins. Being able to run 10 tests concurrently for example would definitely help.

I understand however that it would add complexity / instability to Mocha, so we might look at other ways. It's probably going to be tricky since the tests are created dynamically, so we'd need the actual test runner to keep track of which ones have been run & which ones are left in the queue.

Maybe if Mocha allows using a custom tests runner (similar to custom reporters) we might be able to look into providing a 3rd party runner?