You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I ported the tests of a project from Spock 1.3 (JUnit 4 based) to Spock 2.3 (JUnit 5 engine).
With Spock 1.3 the PIT execution needed about 1 hour.
With Spock 2.3 but the exact same tests, the PIT execution needed about 3 hours.
The ultimate reason was a tremendous amount of unnecessary test iterations.
By removing those, I got back down at least to 1.5 hours for the PIT execution.
But it shows a general performance issue.
There were some mutants that were covered by 6.5 thousand test iterations, and some even by 14 thousand.
Don't ask about those numbers, as I said, those are fixed now and down to 250 iterations.
The difference was, that with Spock 1.3 the smallest unit was a feature with all its iterations.
With Spock 2.3, the smallest unit is one actual iteration.
When PIT executed the 6.5 or 14 thousand iterations by simply kicking off the whole feature, the tests were able to finish within seconds.
But with Spock 2.3 PIT kicks off every iteration individually. That means for each iteration the whole JUnit 5 engine ramp up, discovery and so on needs to be done and that caused the 6.5 thousand iterations to need 20 minutes to kill a mutant if it was one of the later iterations that actually kills it.
There could be a significant performance improvement if test units could be batched if possible.
What I envision is e.g. some method in the PIT test plugin configuration that can specify that the plugin supports test unit batching.
If batching is supported, it would then somehow not execute each test unit individually, but give a list of test units to a batch method together with a boolean whether fullMutationMatrix is requested or not.
The test plugin could then kick off the whole list of test units in one run, aborting as soon as a killing test was executed if requested.
I would have draft an implementation already, but the problem is, that the early-out is currently not properly possible with JUnit 5.
The implementation of junit-team/junit5#1880 is a prerequisite for this improvement, at least for the JUnit 5 plugin, so that the execution can be aborted as soon as a killing test was found.
The text was updated successfully, but these errors were encountered:
I ported the tests of a project from Spock 1.3 (JUnit 4 based) to Spock 2.3 (JUnit 5 engine).
With Spock 1.3 the PIT execution needed about 1 hour.
With Spock 2.3 but the exact same tests, the PIT execution needed about 3 hours.
The ultimate reason was a tremendous amount of unnecessary test iterations.
By removing those, I got back down at least to 1.5 hours for the PIT execution.
But it shows a general performance issue.
There were some mutants that were covered by 6.5 thousand test iterations, and some even by 14 thousand.
Don't ask about those numbers, as I said, those are fixed now and down to 250 iterations.
The difference was, that with Spock 1.3 the smallest unit was a feature with all its iterations.
With Spock 2.3, the smallest unit is one actual iteration.
When PIT executed the 6.5 or 14 thousand iterations by simply kicking off the whole feature, the tests were able to finish within seconds.
But with Spock 2.3 PIT kicks off every iteration individually. That means for each iteration the whole JUnit 5 engine ramp up, discovery and so on needs to be done and that caused the 6.5 thousand iterations to need 20 minutes to kill a mutant if it was one of the later iterations that actually kills it.
There could be a significant performance improvement if test units could be batched if possible.
What I envision is e.g. some method in the PIT test plugin configuration that can specify that the plugin supports test unit batching.
If batching is supported, it would then somehow not execute each test unit individually, but give a list of test units to a batch method together with a boolean whether
fullMutationMatrix
is requested or not.The test plugin could then kick off the whole list of test units in one run, aborting as soon as a killing test was executed if requested.
I would have draft an implementation already, but the problem is, that the early-out is currently not properly possible with JUnit 5.
The implementation of junit-team/junit5#1880 is a prerequisite for this improvement, at least for the JUnit 5 plugin, so that the execution can be aborted as soon as a killing test was found.
The text was updated successfully, but these errors were encountered: