Templated test contexts do not see transformed test classes (classloader/ordering issues in tests)

[holly-cummins]

Describe the bug

I think something is not right in classloading for test classes. If an extension transforms a test class, what gets executed is the transformed class (as you'd expect) - except that if the test is parameterised or uses a TestTemplate, the template gets passed a context which has the untransformed class.

For example, in this test, MyContextProvider gets handed a context which has the test class in it, loaded by the QuarkusClassLoader, but un-transformed.

    @TestTemplate
    @ExtendWith(MyContextProvider.class)
    void verificationTestTemplate(ExtensionContext context) {
        ClassLoader loader = this.getClass().getClassLoader();
        Annotation[] contextAnnotations = context.getRequiredTestClass().getAnnotations();
        Annotation[] myAnnotations = this.getClass().getAnnotations();

        Assertions.assertEquals(myAnnotations.length, contextAnnotations.length, "The test template sees a different version of the class than the test execution" + Arrays.toString(myAnnotations) + " vs " + Arrays.toString(contextAnnotations));
    }

Expected behavior

I would expect the view of the test class would be consistent across the whole test execution, including in the ExtensionContext.

Actual behavior

The test above fails; the ExtensionContext has a version of the class which does not have transformations on it.

How to Reproduce?

Reproducer: https://github.com/holly-cummins/parameterized-test-reproducer/blob/main/README.md

What’s the reproducer doing? An extension defines some byte code transformations which apply to a test class.

The test uses a Junit 5 @testtemplate for tests. At runtime, the version of the test that runs is the version which was transformed by the extension. However, the ContextProvider for the test template is initialised with the non-transformed version of the test class (this is the problem).

Note that QuarkusDevModeTest may not seem to be enough to show this issue, it may need a proper integration testing running in dev mode. (I had another issue which definitely did not show up in the QuarkusDevModeTest but did with mvn Quarkus:dev.)

2022-09-08 21:59:10,425 INFO  [io.qua.test] (Test runner thread) Running 1/0. Running: sample.house.ParameterizedTest#verificationTestTemplate(ExtensionContext)
Test class is class sample.house.ParameterizedTest
Test class annotations is [@io.quarkus.test.junit.QuarkusTest()]
Test class classloader is QuarkusClassLoader:Deployment Class Loader: TEST@7881b3d8
2022-09-08 21:59:10,428 INFO  [io.qua.test] (Test runner thread) Running 1/0. Running: sample.house.ParameterizedTest#[@io.quarkus.test.junit.QuarkusTest()]
This tests's loader is QuarkusClassLoader:Quarkus Runtime ClassLoader: TEST restart no:1@576a1438
This tests's annotations is [@quarkus.io.pact.runtime.AnnotationAddedByExtension(), @io.quarkus.test.junit.QuarkusTest()]

To see the failure,

rm -rf integration-tests/src/test/resources-filtered/projects/bff/target && mvn clean  install verify  -DskipTests=false -DskipITs=false

To play with the failing test in more detail,

cd integration-tests/src/test/resources-filtered/projects/bff
mvn quarkus:dev

mvn verify also fails.

Output of uname -a or ver

No response

Output of java -version

openjdk version "17.0.2" 2022-01-18 OpenJDK Runtime Environment Temurin-17.0.2+8 (build 17.0.2+8) OpenJDK 64-Bit Server VM Temurin-17.0.2+8 (build 17.0.2+8, mixed mode)

GraalVM version (if different from Java)

No response

Quarkus version or git rev

<quarkus.version>2.12.0.Final</quarkus.version>

Build tool (ie. output of mvnw --version or gradlew --version)

Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)

Additional information

No response

[holly-cummins]

I've updated the reproducer to remove some confusing package names.

[holly-cummins]

I've made the tests more comprehensive, which has given me some more data points.

• Behaviour is different with and without @QuarkusTest ... but bad in both cases
• With @QuarkusTest and parameterization, the executed test sees it's been bytecode-transformed, but the parameterising context does not see the transformations
• Without @QuarkusTest, neither the executed test or the parameterising context see the bytecode-transformations
• With no parameterisations, the executed test sees its bytecode transformations

Sad case:

2022-12-08 16:49:16,775 ERROR [io.qua.test] (Test runner thread) >>>>>>>>>>>>>>>>>>>> Summary: <<<<<<<<<<<<<<<<<<<<
sample.house.ParameterizedTest#classloaderIntrospectionTestTemplate(ParameterizedTest.java:42) ParameterizedTest#[@io.quarkus.test.junit.QuarkusTest()] The test template is using a different classloader to the actual test. ==> expected: <QuarkusClassLoader:Quarkus Runtime ClassLoader: TEST restart no:0@142e10d6> but was: <QuarkusClassLoader:Deployment Class Loader: TEST@283a6e34>
sample.house.ParameterizedTest#verificationTestTemplate(ParameterizedTest.java:32) ParameterizedTest#[@io.quarkus.test.junit.QuarkusTest()] The test template sees a different version of the class than the test execution[@io.quarkiverse.acme.runtime.AnnotationAddedByExtension(), @io.quarkus.test.junit.QuarkusTest()] vs [@io.quarkus.test.junit.QuarkusTest()] ==> expected: <2> but was: <1>

Happy classloader: QuarkusClassLoader:Quarkus Runtime ClassLoader: TEST restart no:1@44b7fc54
Sad classloader: QuarkusClassLoader:Deployment Class Loader: TEST@6ab866c2

[holly-cummins]

@stuartwdouglas reports that this issue is blocked on junit-team/junit5#2579.

[holly-cummins]

It looks like we may be able to resolve this in JUnit 5.10.

If we didn't want to wait, a possible approach to handling this kind of problem (and various other problems related to JUnit seeing the un-transformed classes and XStream serialization not working with Java 17) is illustrated in Junit #1146. What they do is re-transform the classes in the JUnit extension. I don't think we want to do this, but it's nice to know the option there as a backup.

What we get in JUnit 5.10 is junit-team/junit5#3091 (documentation). The docs say "A typical use case is to create a custom replace the ClassLoader used by the JUnit Platform to load test classes and engine implementations." (Hurray!)

Further out is junit-team/junit5#3028, which allows richer classloader customisation. I think we won't need that, but if the LauncherInterceptor doesn't do what we need, hopefully it will.