Run tests in parallel

[dan-ryan]

Sorry if this has been brought up before, I couldn't find this in the issues tab.
Would be great if we could run tests in parallel, that would definitely speed up our tests if that was possible.

[jbrwilkinson]

What is the cause of the slowness at the moment? Lots of tests or some take a long time?

[chvillanuevap]

Is this different from using MPI_* functions in the tests? Actually, can we use MPI_* functions with Catch at all?

[dan-ryan]

"What is the cause of the slowness at the moment?"
For us (CricHQ), lots of tests. With a number of unused cores waiting to be used :)

[BatmanAoD]

@zammbi What are you using to run your tests? Can you just have your runner start multiple test processes at once? ctest, for instance, provides this functionality.

[dan-ryan]

"What are you using to run your tests?"
Well that is up to the developer. I just run mine in the IDE which is Xcode or Appcode. We are looking at getting the tests to also run on CircleCI soon.

"Can you just have your runner start multiple test processes at once? "
That is a possibility. But would be easier if the library automagically handled that all for you and made sure the logs still made sense, etc :)

It isn't a big deal atm, just a nice to have. Catch is much better than what we were working with previously (gtest).

[movitto]

I suspect this is one of "those issues" for this project are there is this and #681 and a few others wrt thread safety which is sort of along these lines.

In any case, I wrote a bash script to parallelize by process, it first runs the specified test runner w/ "-l" to get the tags, and then splits those up into the specified number of groups to be run as bash backround jobs (the script waits until everything is done before returning).

Terminal output from failed jobs may be a bit intermangled (maybe not, depends on timing), perhaps these can be captured seperately and output serially once everything is done (just saying there is alot of room for improvement!)

Come get it!

[ScottHutchinson]

If we had a version of Catch2 that didn't need compatibility with C++98, then could it be modified to run tests in parallel? The work-arounds are not getting the job done in our case. We're using C++17, and could use the PPL to run tests asynchronously and in parallel. We have about 1,000 tests, nearly all of which are file I/O intensive. They can take over 30 minutes to complete. We add several more such tests every week. Running them in separate processes has not shortened the elapsed time.

[bowie7070]

In case it helps someone else I have found the GNU parallel program very helpful complement to catch2 because it solves two problems I currently have. First, running jobs in parallel. Second, having some kind of indicator of progress. Here is my script for running my tests in parallel during the day. Note that:

• during the day I want to stop at first failure (similar to -a behaviour) so I use --halt now,fail=1,
• I want a progress bar so I use --bar,
• I have capped the number of processes at 20 with -j because that is appropriate for the hardware I am using and the average thread use of our processes but please set appropriately for you,
• I like to run the tests in random order so I use --order rand --rng-seed time,
• the "$@" allows me to pass a tag or other filter so that I can run a subset of tests.

#!/bin/bash

cd directory-for-testing

mkdir "tmp$$"

exe=../release/catch-program

$exe --list-test-names-only --order rand --rng-seed time "$@" |
    parallel --bar --halt now,fail=1 -j 20 $exe "{}" ">" "tmp$$/{}.out"

Thanks to the team behind catch2 - I can't imagine how many hours having a good testing framework has saved us by finding bugs sooner rather than later.

[ScottHutchinson]

@bowie7070 What is the parallel program? Where can we find it? Thanks!

[bowie7070]

The parallel I referred to is gnu parallel. More information at:

https://www.gnu.org/software/parallel/

I am using Ubuntu so I installed it using apt-get. Apologies for any confusion @ScottHutchinson . Let me know if any further clarification is needed.

[horenmar]

For the record, the official answer to both this and #681 is that multiple threads interacting with assertions might become supported*, but running multiple test cases won't be. This means that if you want to run your test suite in parallel, you should use an external runner and something like tags to split it up. Catch2, and my other projects use CTest, GNU parallel can also be used, I expect bunch of other tools can be used too.

(I do not recommend running each test case in its own process for a variety of reasons, especially once your binaries grow to thousands of test cases)

* There are two obstacles: time, and the fact that it is hard to come up with reasonable semantics for how multiple threads should interact with SECTIONs...

[bowie7070]

If you have time, I would really appreciate it if you could elaborate on "I do not recommend running each test case in its own process for a variety of reasons, especially once your binaries grow to thousands of test cases."

This is the exactly the situation we are faced with at work. We have for one library about 4,500 tests with an average run time of about 5 seconds (per test). I've just started using GNU parallel to farm them out into individual processes and we are getting a 10x or more speedup. It has made it viable to run all of our tests again during the day while making changes.

Running one test per process makes two things very easy. First, providing a good diagnostic when a test fails. Second, finding the diagnostic output from only the failing test.

The main issue I am worried about is that we have lost the ability to check for hidden state in our code. That is, run test A then test B and everything passes but run test B then test A and one or more of them fails. However, we mitigate that by still running our nightly tests in one process so we should pick that up at the end of the day.

Like I said, we have only just embarked on this approach so if there are traps we are about to fall into, I would appreciate hearing about them now rather than finding out the hard way later.

[horenmar]

The main issue I am worried about is that we have lost the ability to check for hidden state in our code.

This is one of them. The other one is the process startup overhead can add significant time costs, but if this

We have for one library about 4,500 tests with an average run time of about 5 seconds (per test).

is either median or from a non-heavy tailed distribution, it doesn't apply to you. In most codebases I worked on, the runtime of 99% of test cases is much lower than the overhead of starting up a process and registering the test cases. On Windows, this is true even for batches of tests.

[bouweandela]

To parallelize with CTest, I used the following:

catch_discover_tests(target)
foreach(test_name ${target_TESTS})
  add_test(NAME target::${test_name} COMMAND target ${test_name})
endforeach()

Does that look like a good approach?

[Motyak]

Yet another way to run catch2 tests in parallel, without gnu parallel :
https://gist.github.com/Motyak/1bb66de8cda660143aece81e67377f03