JUnit5 tests annotated with @Tag are not executed in suite that has @IncludeTags, unless combined with @SelectPackages

[carlspring]

Bug description

A test suite with @IncludeTags({ "foo" }) will not pick up test annotated with @Tag("foo"), unless the suite also explicitly has the @SelectPackages({ "org.foo" )} annotation. It should be possible to not have to define all the packages.

Steps to reproduce

• Test class:

@Tag("foo")
class SomeIT
{

   @Test
   public void testSomeStuff()
   {
       ...
   }

}

• Suite class:

@RunWith(JUnitPlatform.class)
@IncludeTags({"foo"})
//@SelectPackages("org.foo")
public class SomeITSuite
{
}

When you don't have the @SelectPackages("org.foo"), the SomeIT will not be automatically discovered and executed.

• Minimal pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">

    <modelVersion>4.0.0</modelVersion>

    <groupId>org.foo</groupId>
    <artifactId>bar</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>


    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

        <version.slf4j>1.7.30</version.slf4j>
        <version.junit>5.7.0</version.junit>
        <version.junit.platform.runner>1.7.0</version.junit.platform.runner>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.8.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <encoding>UTF-8</encoding>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-surefire-plugin</artifactId>
                <version>3.0.0-M5</version>
            </plugin>
        </plugins>
    </build>

    <dependencies>
        <!-- for testing -->
        <dependency>
            <groupId>org.junit.jupiter</groupId>
            <artifactId>junit-jupiter</artifactId>
            <version>${version.junit}</version>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.junit.platform</groupId>
            <artifactId>junit-platform-runner</artifactId>
            <version>${version.junit.platform.runner}</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <profiles>
        <profile>
            <id>integration-tests</id>

            <build>
                <plugins>
                    <plugin>
                        <groupId>org.apache.maven.plugins</groupId>
                        <artifactId>maven-failsafe-plugin</artifactId>
                        <version>3.0.0-M5</version>
                        <configuration>
                            <includes>
                                <include>**/*ITSuite.java</include>
                            </includes>
                        </configuration>
                        <executions>
                            <execution>
                                <id>integration-test</id>
                                <phase>integration-test</phase>
                                <goals>
                                    <goal>integration-test</goal>
                                </goals>
                            </execution>
                            <execution>
                                <id>verify</id>
                                <phase>verify</phase>
                                <goals>
                                    <goal>verify</goal>
                                </goals>
                            </execution>
                        </executions>
                    </plugin>
                </plugins>
            </build>
        </profile>
    </profiles>
</project>

This executes the suite with 0 tests in both Idea and a console (using mvn clean install -Pintegration-tests).

If I restore the commented out @SelectPackages("org.foo"), it will run all the tests, regardless of whether they're tagged, or not. It doesn't even care, if they're in those packages. What am I missing here? Is this a bug?

Associated Stackoverflow question.

Context

• Used versions (Jupiter/Vintage/Platform):
  • JUnit: 5.7.0
  • JUnit Platform Runner: 1.7.0
• Build Tool/IDE:
  • Maven 3.6.3, IntelliJ Idea, XTerm

Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
Maven home: /java/apache/maven-3.6.3
Java version: 1.8.0_241, vendor: Oracle Corporation, runtime: /java/jdk1.8.0_241/jre
Default locale: en_GB, platform encoding: UTF-8
OS name: "linux", version: "5.4.0-56-generic", arch: "amd64", family: "unix"

Deliverables

• It should be possible to run the tests in the suite without having a @SelectPackages("org.foo"), if there is a @IncludeTags.

[marcphilipp]

@IncludeTags is a filter, not a selector, i.e. it will only ever reduce the number of tests otherwise selected, but never add any.

[carlspring]

@marcphilipp ,

Perhaps I'm misusing this, as you're saying, but it's actually not reducing the tests, it's running all of them, regardless of whether they have the tag.

Would you mind showing me what the correct way to do this is?

Many thanks in advance!

[marcphilipp]

If you use @SelectPackages and @IncludeTags it should only run tests with the correct tag in the selected packages.

[carlspring]

@marcphilipp ,

But this is what I've also tried and it runs all the tests in the packages, regardless of whether they have the @Tag("foo"). Would you mind re-testing it with the simple example I've provided above? For me, this is not working.

[marcphilipp]

In the example above, SomeIT has tag "foo" and hence runs. I'm happy to dig deeper if you provide a complete minimal example project. 🙂

[sormuras]

Perhaps, SF/FS's way of passing tags and/or applying own filters interferes here?

See section "Filtering by Tags" of https://maven.apache.org/surefire/maven-surefire-plugin/examples/junit-platform.html

[carlspring]

It appears that the integration tests that we had were all named FooBarIT and the IT suffix was confusing it. I renamed these to FooBarITTest and things started working. Thanks to @steve-todorov for spotting this!

[marcphilipp]

Glad you figured it out! For the record that's because of the default class name pattern as documented in the runner's Javadoc:

junit5/junit-platform-runner/src/main/java/org/junit/platform/runner/JUnitPlatform.java

Lines 87 to 93 in 3f7fed6

	* <p>When used on a class that serves as a test suite and the
	* {@link IncludeClassNamePatterns @IncludeClassNamePatterns} annotation is not
	* present, the default include pattern
	* {@value org.junit.platform.engine.discovery.ClassNameFilter#STANDARD_INCLUDE_PATTERN}
	* will be used in order to avoid loading classes unnecessarily (see {@link
	* org.junit.platform.engine.discovery.ClassNameFilter#STANDARD_INCLUDE_PATTERN
	* ClassNameFilter#STANDARD_INCLUDE_PATTERN}).

The default class name pattern is ^(Test.*|.+[.$]Test.*|.*Tests?)$. Thus, if you'd like to stick with the IT suffix, you should add a @IncludeClassNamePatterns("^.*IT$") annotation to the suite class. Please also see Failsafe's default includes if you want to make it more complete.