Introduce junit-platform-suite-api module

[smoyer64]

Overview

Move the following Java types to a new junit-platform-suite-api module. These types will be useful to many engine developers and, other than the references to @RunWith(JUnitPlatform.class) in the javadocs, should not be tied to the vintage Runner.

• ExcludeClassNamePatterns.java
• ExcludeEngines.java
• ExcludePackages.java
• ExcludeTags.java
• IncludeClassNamePatterns.java
• IncludeEngines.java
• IncludePackages.java
• IncludeTags.java
• SelectClasses.java
• SelectPackages.java
• UseTechnicalNames.java

Feature request

I've extended the JupiterTestEngine to create the JupiterSuiteEngine which supports an arbitrary hierarchy of test containers between the engine and test classes. At each level of the suite hierarchy, @BeforeSuite and @AfterSuite are supported, so you can also nest the hierarchy of setUp and tearDown. An example suite looks something like this:

@Suite
@IncludeClassNamePatterns("**/*JmqITs")
public class RequiresJmqSuite {
	
	@BeforeSuite
	void setUpJmqConnection() { ... }
	
	@AfterSuite
	void tearDownJmqConnection() { ... }

}

The JupiterSuiteEngine is currently hard-coded to scan for classes that end with "Suite" but it's my intention to have the engine consider the class name patterns and selections passed into the engine (once I make sure there won't be collisions with the JupiterTestEngine. You can obviously have more than one suite discovered by the engine in which case they will be peers in the test hierarchy.

In any case, when importing junit-platform-runner into my project, I also get the actual JUnit 4 Runner classes which I don't need. Worse yet, this implicit dependencies to my project which I don't need. My current work-around is to exclude these classes from the dependency graph as follows:

<dependency>
	<groupId>org.junit.platform</groupId>
	<artifactId>junit-platform-runner</artifactId>
	<version>${version.junit.platform}</version>
	<exclusions>
		<exclusion>
			<groupId>junit</groupId>
			<artifactId>junit</artifactId>
		</exclusion>
		...
	</exclusions>
</dependency>

This obviously isn't the cleanest way to manage dependencies and it's my belief that the types listed above will be useful to many engine developers. As an aside, providing maintained interfaces to the utils/*Utils classes in junit-platform-commons will also help this project tremendously

Deliverables

• Move each of the types listed above from junit-platform-runner to junit-platform-suite-api.

[gaganis]

@smoyer64 I have the impression that if we did #691, migrating to the java library plugin, it would solve the implicit dependencies problems that you are mentioning. I suspect the junit dependency of the junit-platform-runner project would be an implementation dependency.

Even if the dep problem would be solved by #691, introducing a new project might still be valuable as it could it could clearly denote what the API is for building engines.

[marcphilipp]

I think it has to be an api dependency because JUnitPlatform's public API contains references to JUnit 4 classes.

[sbrannen]

This is something I've been wanting to do ever since we introduced the suite support for JUnit 4 on the platform.

I recommend moving the annotations to a module named junit-platform-suite-api.

And I'm slating this for inclusion before 5.0 GA so that the change is less disruptive for consumers of the API.

[sbrannen]

The proposed JupiterSuiteEngine should then be addressed in a separate issue.

[sbrannen]

Actually, I'd like to move the annotations in M4: the earlier, the better.

@junit-team/junit-lambda any objections?

[sormuras]

Actually, I'd like to move the annotations in M4: the earlier, the better.

Sure. Go ahead!

[sbrannen]

There are, however, some issues with this approach...

Since the suite annotations are open for use by anyone (i.e., any TestEngine), how will a given test engine know that it's supposed to actually honor the suite annotations and generate a suite in the test plan?

With the Vintage engine, it's straightforward since @RunWith(JUnitPlatform.class) must be present. But... with the Jupiter engine, what would be the trigger that signals to the Jupiter engine that it must create a suite?

Should the JUnit Platform itself actually honor the suite annotations instead of test engines taking over that job? For example, those annotations actually have a one-to-one correlation to selectors and filters already supported by the launcher APIs. So, the launcher could perhaps automatically detect those annotations and modify the request dynamically, before passing the request to any registered test engines.

Has anyone put any thought into these types of issues?

[sbrannen]

@smoyer64, I just noticed your proposed @Suite annotation -- which I assume would be part of the junit-jupiter-api module -- right?

[smoyer64]

@sbrannen I've already got the JupiterSuiteEngine running and was going to migrate it to junit-pioneer. And since it only uses the junit-engine-api, it's also capable of wrapping an engine that's not junit-jupiter-engine, I was planning to add support to use @IncludeEngines and rename the suite class to indicate it's for more than just the Jupiter engine. This proposal wasn't to add the suite functionality into JUnit5 at all but rather to simply eliminate the dependency on JUnit (vintage) that my suite implementation ended up with.

If you're saying this feature will be added into JUnit 5 itself, it seems that I should probably stop now - or at least coordinate adding this feature into the JUnit 5 codebase instead?

If you look at #687 you'll also find that this function could be accomplished as an Extension instead of an engine. I think in the long run this solution is even better ... other extensions might want to create test containers too? Right now I've also got @BeforeSuite/@AfterSuite but I'm leaning towards reusing @BeforeAll/@AfterAll so that the "all" is context dependent.

[sbrannen]

@smoyer64, I honestly have not put a great deal of thought into this topic (at least not on the implementation level).

IMHO, the JUnit Platform would ideally provide support for declarative suites (e.g., configured via annotations) regardless of which test engines are registered.

That's about all I can contribute at the moment in terms of a vision.

Sooooo, my goal would be to make that vision become reality. But... I have yet to determine the feasibility of such a vision.

[sbrannen]

in progress

[sbrannen]

@smoyer64, would you mind opening a new issue to address the JupiterSuiteEngine and your additional ideas in that regard?

[sbrannen]

Addressed in 248943f.

[smoyer64]

I'd be happy to open another issue to (help) specify the declarative suite behavior and I'm more than happy to see it end up in the JUnit 5 itself. Assuming the suite's execution could hook into JUnit 5 in a non-public way, I'll leave it neutral with respect to #687 but that may still have some benefits.

This also seems like it's going to supercede junit-pioneer/junit-pioneer#35.

[sbrannen]

Thanks, @smoyer64.

Please let me know once you're created the new issue (in case I miss it).