Run specs or suites in Parallel

[ashleyfrieze]

I intend to build some Acceptance tests for a batch processing system. These generally have the form of:

Given we drop file X into the input
When the system has a result in the output
Then it is a good one

As you might imagine, each test uploading files or waiting for propagation time is very slow in terms of elapsed time, even though the system under test could receive all inputs at once and run them all in parallel. While we could write the whole test as a tangle of parallel concerns:

Given we drop file X into the input
And we drop file Y into the input
When the system has results in the output for X AND Y
Then X is a good one
And Y is a good one

The best option would be to write the tests to run in parallel. Spectrum style it might look like this:

feature("unconflicting use cases that can share the system", with(parallelRunning(), () -> {
   scenario("the X files", () -> {
       given("we drop the file into the input", () -> {
          system.drop("X.file");
       });
      when("the system has results", () -> {
          system.waitFor("X.result");
      });
      then("the result is good", () -> {
          validator.check("X.result", "X.expectedResult");
      });
   }); 
   
   // this scenario is expected to run in parallel with the other
   scenario("the y files", () -> {
       given(...);
       when(...);
       then(...);
   });
});

This will play havoc with the output to Eclipse or IntelliJ's runner, though potentially a RunNotifier proxy could be put in the middle to avoid that.

[ashleyfrieze]

@greghaskins if I distil the above into the following question, how would you react?

What if execute all specs in parallel could be done by a PreCondition like parallelRunning or runInParallel? Would that be in keeping with your design expectations?

[ashleyfrieze]

I'm looking at implementing this feature again. The recent #115 change has somewhat damaged my original intentions... but it was a necessary change.

Here are the three possible ways of making thread safety possible within tests.

1. Make the specs that run in parallel HAVE to receive all dependent objects, like let etc through injection - i.e. they become a Consumer of their dependencies, allowing the multi-threaded ecosystem to pass them a thread isolated value
2. Make the let objects give different objects depending on the thread using them - we've already had this trouble, but perhaps there are other ways of solving the problem
3. Have parallel execution somehow create a new instance of the test class and each isolated instance is somehow used in a separate thread.

I think any other solution would fall into one of the above categories. I think the right answer may be to do 2, but to do something automagic with thread groups and make it a rule that you have to pass objects across thread boundaries in multithreaded tests, using something more robust.

I've been worrying a fair bit about how parallelisation would work, especially with thread pools and possible deadlocks, especially if parallelisation can be automatic down the hierarchy. I've decided to keep it simple.

Let's say we have a Configuration called parallelExecution which takes, as a parameter, a maximum concurrency (or none if you don't care).

So let's say you have a spec like this:

describe("Some stuff", () -> {
   it("is awesome", () -> {});
   it("is brilliant", () -> {});
   it("is genius", () -> {});
}); 

and you want to parallelise the execution. You'd just go

describe("Some stuff", with(parallelExecution(2), () -> {
   it("is awesome", () -> {});
   it("is brilliant", () -> {});
   it("is genius", () -> {});
})); 

Meaning that the describe block would run the its in parallel up to a maximum of 2 concurrent.

For nested describes, you could choose to parallelise them or not:

describe("complex suite", with(parallelExecution(), () -> {
   describe("suite 1", () -> {
        it("is not run in parallel", () -> {});
        it("is not run in parallel also", () -> {});
   });

   describe("suite 1", with(parallelExecution(10), () -> {
        it("is run in parallel", () -> {});
        it("is run in parallel also", () -> {});
        it("is run in parallel also", () -> {});
        it("is etc...", () -> {});
   }));
}));

This could lead to a thread EXPLOSION, but that's where I think the idea of turnstiles will come along. A turnstile can be used to limit the number of threads allowed through a critical section.

it("is part of a multithreaded test", () -> {
    // do something that we don't care about thread control for

    turnstile.execute(() -> {
        // do something that we don't want to be TOO concurrent
    });

    // do something else we don't care about thread control
});

A turnstile would be created like this:

Turnstile turnstile = new Turnstile(25); // create turnstile with max capacity of 25

it would have functions a bit like this in it:

public void execute(Runnable runnable) {
    execute(() -> { runnable.run(); return null; });
}

public <T> T execute(ThrowingSupplier<T> supplier) {
    // will execute this supplier and return its results when there's a spare slot in the turnstile
}