Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support test execution order for JUnit 5 #161

Closed
anatoliy-balakirev opened this issue Nov 26, 2019 · 8 comments
Closed

Support test execution order for JUnit 5 #161

anatoliy-balakirev opened this issue Nov 26, 2019 · 8 comments
Labels
question

Comments

@anatoliy-balakirev
Copy link

Test execution order was added to Junit 5 some time ago:
https://junit.org/junit5/docs/current/user-guide/#writing-tests-test-execution-order

However, currently it looks like PIT is just ignoring them and executes tests in some not clear order.
@szpak
Copy link
Owner

szpak commented Nov 26, 2019 •
edited
Loading

Have you consulted the README file about the required configuration? If yes, please provide a sample project reproducing the problem (or add/modify the existing JUnit 5 functional test to detect that).

@szpak szpak added the question label Nov 26, 2019
@szpak
Copy link
Owner

szpak commented Dec 13, 2019 •
edited
Loading

@anatoliy-balakirev I've just realized that your are not asking about JUnit 5 tests in general, but about the execution order 🤦‍♂️

I haven't seen that feature in PIT itself, therefore I propose you to create a ticket for PIT core directly.

@szpak szpak closed this as completed Dec 13, 2019
@anatoliy-balakirev
Copy link
Author

Hi @szpak . Sorry for the late reply. Couldn't find time to set up a sample project. Ordered Junit 4 tests are already supported by the pitest plugin. Problem is only with Junit 5 ordered tests, which were introduced recently.

Here is a sample repo: https://github.com/anatoliy-balakirev/pitest_ordered_tests
There you can find two tests:

  • pitest.sample.controller.SampleControllerTest - junit 5
  • pitest.sample.controller.SampleControllerJunit4Test - junit 4

If you switch to junit4 branch there (or just remove testPlugin = "junit5" from the build.gradle) and run pitest - tests are executed properly, everything is green. If you try to execute pitest with Junit 5 (default in master) - tests are failed with:

14:10:30 PIT >> INFO : MINION : 14:10:30 PIT >> SEVERE : Description [testClass=pitest.sample.controller.SampleControllerTest, name=validate_2_internalError()]
org.opentest4j.AssertionFailedError: 
Expecting:
 <204>
to be equal to:
 <500>
but was not.
	at java.base/jdk.internal.reflect.N
14:10:30 PIT >> INFO : MINION : ativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(De
14:10:30 PIT >> INFO : MINION : legatingConstructorAccessorImpl.java:45)
	at pitest.sample.controller.SampleControllerTest.validate_2_internalError(SampleControllerTest.java:59)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.
14:10:30 PIT >> INFO : MINION : reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.ju
14:10:30 PIT >> INFO : MINION : nit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:515)
	at org.junit.jupiter.engine.execution.ExecutableInvoker.invoke(ExecutableInvoker.java:115)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestM
14:10:30 PIT >> INFO : MINION : ethod$6(TestMethodTestDescriptor.java:171)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.
14:10:30 PIT >> INFO : MINION : java:167)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:114)
	at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:59)
	at org.junit.platform.engine.s
14:10:30 PIT >> INFO : MINION : upport.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:105)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.ex
14:10:30 PIT >> INFO : MINION : ecuteRecursively(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:71)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
	at org.junit.platform.engine.support.hierarchical.SameThrea
14:10:30 PIT >> INFO : MINION : dHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:110)
	at org.junit.platform.engine.support.hierarchi
14:10:30 PIT >> INFO : MINION : cal.ThrowableCollector.execute(ThrowableCollector.java:72)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:71
14:10:30 PIT >> INFO : MINION : )
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1540)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:38)
	at org.junit.platform.engine.support
14:10:30 PIT >> INFO : MINION : .hierarchical.NodeTestTask.lambda$executeRecursively$5(NodeTestTask.java:110)
	at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:72)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeR
14:10:30 PIT >> INFO : MINION : ecursively(NodeTestTask.java:95)
	at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:71)
	at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExe
14:10:30 PIT >> INFO : MINION : cutorService.java:32)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
	at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:51)
	a
14:10:30 PIT >> INFO : MINION : t org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:220)
	at org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:188)
	at org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams
14:10:30 PIT >> INFO : MINION : (DefaultLauncher.java:202)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:181)
	at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
	at org.pitest.junit5.JUnit5TestUnit.execute(JUnit5Test
14:10:30 PIT >> INFO : MINION : Unit.java:95)
	at org.pitest.coverage.execute.CoverageDecorator.execute(CoverageDecorator.java:50)
	at org.pitest.testapi.execute.containers.UnContainer.execute(UnContainer.java:31)
	at org.pitest.testapi.execute.Pitest.executeTests(Pitest.java:57)
	at org
14:10:30 PIT >> INFO : MINION : .pitest.testapi.execute.Pitest.run(Pitest.java:48)
	at org.pitest.coverage.execute.CoverageWorker.run(CoverageWorker.java:51)
	at org.pitest.coverage.execute.CoverageMinion.main(CoverageMinion.java:90)

14:10:30 PIT >> SEVERE : Description [testClass=pitest.sample.controller.SampleControllerTest, name=validate_2_internalError()] did not pass without mutation.
14:10:30 PIT >> FINE : Coverage generator Minion exited ok
14:10:30 PIT >> INFO : Calculated coverage in 2 seconds.
14:10:30 PIT >> SEVERE : Tests failing without mutation: 
Description [testClass=pitest.sample.controller.SampleControllerTest, name=validate_2_internalError()]
Exception in thread "main" org.pitest.help.PitHelpError: 1 tests did not pass without mutation when calculating line coverage. Mutation testing requires a green suite.
See http://pitest.org for more details.
	at org.pitest.coverage.execute.DefaultCoverageGenerator.verifyBuildSuitableForMutationTesting(DefaultCoverageGenerator.java:114)
	at org.pitest.coverage.execute.DefaultCoverageGenerator.calculateCoverage(DefaultCoverageGenerator.java:96)
	at org.pitest.coverage.execute.DefaultCoverageGenerator.calculateCoverage(DefaultCoverageGenerator.java:51)
	at org.pitest.mutationtest.tooling.MutationCoverage.runReport(MutationCoverage.java:115)
	at org.pitest.mutationtest.tooling.EntryPoint.execute(EntryPoint.java:120)
	at org.pitest.mutationtest.tooling.EntryPoint.execute(EntryPoint.java:50)
	at org.pitest.mutationtest.commandline.MutationCoverageReport.runReport(MutationCoverageReport.java:87)
	at org.pitest.mutationtest.commandline.MutationCoverageReport.main(MutationCoverageReport.java:45)

FAILURE: Build failed with an exception.

@anatoliy-balakirev
Copy link
Author

Do you want to reopen this task or rather create a new one?

@szpak
Copy link
Owner

szpak commented Jan 15, 2020

Thanks for the sample project @anatoliy-balakirev!

I haven't found any information about @TestMethodOrder in pitest-junit5-plugin. @hcoles, @tobiasstadler is it supported?

@szpak szpak reopened this Jan 15, 2020
@anatoliy-balakirev
Copy link
Author

Just to be on the same page here:

  • @FixMethodOrder(MethodSorters.NAME_ASCENDING) - Junit 4
  • @TestMethodOrder(MethodOrderer.Alphanumeric.class) - Junit 5

@anatoliy-balakirev
Copy link
Author

Actually, I just realized that it might be better to create an issue for the pitest-junit5-plugin (which I thought I actually did). Will open a new one there. This one might be closed, I think.

@anatoliy-balakirev
Copy link
Author

pitest/pitest-junit5-plugin#46

@szpak szpak closed this as completed Jan 21, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@szpak @anatoliy-balakirev