Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Performance issue: batch test unit execution for test framework plugins where it is possible #1223

Open
Vampire opened this issue May 30, 2023 · 0 comments

Comments

@Vampire
Copy link
Contributor

Vampire commented May 30, 2023

I ported the tests of a project from Spock 1.3 (JUnit 4 based) to Spock 2.3 (JUnit 5 engine).
With Spock 1.3 the PIT execution needed about 1 hour.
With Spock 2.3 but the exact same tests, the PIT execution needed about 3 hours.

The ultimate reason was a tremendous amount of unnecessary test iterations.
By removing those, I got back down at least to 1.5 hours for the PIT execution.

But it shows a general performance issue.
There were some mutants that were covered by 6.5 thousand test iterations, and some even by 14 thousand.
Don't ask about those numbers, as I said, those are fixed now and down to 250 iterations.

The difference was, that with Spock 1.3 the smallest unit was a feature with all its iterations.
With Spock 2.3, the smallest unit is one actual iteration.
When PIT executed the 6.5 or 14 thousand iterations by simply kicking off the whole feature, the tests were able to finish within seconds.
But with Spock 2.3 PIT kicks off every iteration individually. That means for each iteration the whole JUnit 5 engine ramp up, discovery and so on needs to be done and that caused the 6.5 thousand iterations to need 20 minutes to kill a mutant if it was one of the later iterations that actually kills it.

There could be a significant performance improvement if test units could be batched if possible.

What I envision is e.g. some method in the PIT test plugin configuration that can specify that the plugin supports test unit batching.
If batching is supported, it would then somehow not execute each test unit individually, but give a list of test units to a batch method together with a boolean whether fullMutationMatrix is requested or not.
The test plugin could then kick off the whole list of test units in one run, aborting as soon as a killing test was executed if requested.

I would have draft an implementation already, but the problem is, that the early-out is currently not properly possible with JUnit 5.
The implementation of junit-team/junit5#1880 is a prerequisite for this improvement, at least for the JUnit 5 plugin, so that the execution can be aborted as soon as a killing test was found.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant