3.16: @WithTestResource starts all test resources (regression)

[snazy]

Describe the bug

Since Quarkus 3.16 (including 3.16.1, 3.15.1 is fine) our integration test classes start literally test-resources from all literally all @WithTestResource annotations, even for those that are not (yet) executed.

Before 3.16(.1), only the test-resource for the currently running integration-test-class were started.

The major change in behavior that I could identify so far is in io.quarkus.test.common.TestResourceManager#restrictToAnnotatedClass, which now delegates to the TestResourceClassEntryHandler implementations (for @QuarkusTestResource and @WithTestResource). (Admittedly, restrictToAnnotatedClass could never yield true before 3.16.)

Since restrictToAnnotatedClass() checks for == RESTRICTED_TO_CLASS on all annotations "everywhere", it initialized and starts all test resources. This can lead to e.g. 4 Keycloaks, 2 Minios and a bunch of other containers 🤷

Our integration-tests are annotated using @WithTestResource(value = xyz.class), leaving the default scope MATCHING_RESOURCES to leverage the ... Quarkus will not restart when running consecutive tests that use the same set of resources. behavior.

The current workaround is to set the scope to RESTRICTED_TO_CLASS.

/cc @geoand

Expected behavior

• Only have the test-resources for the current test class running
• Group tests by test-resources, considering "initArgs"
• Don't start the "whole world" ;)

Actual behavior

No response

How to Reproduce?

I do have a reproducer, but it's still a (local) WIP to bump Quarkus to 3.16.1 in Nessie, including the workaround mentioned above.

Output of uname -a or ver

No response

Output of java -version

No response

Quarkus version or git rev

No response

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[geoand]

Thanks a lot for reporting.

Any chance you can add a reproducer so I can easily see the difference between 3.15 and 3.16 in action?

[hemanth-web-dev]

In Quarkus 3.16+, @WithTestResource annotations are unexpectedly starting all defined test resources for every test class, not just those needed for the current test, which worked as expected in previous versions. This change, due to TestResourceManager#restrictToAnnotatedClass now checking RESTRICTED_TO_CLASS by default, results in redundant containers starting up (e.g., multiple Keycloak and Minio instances) and increased resource consumption. While setting scope = RESTRICTED_TO_CLASS on each @WithTestResource annotation provides a temporary workaround, we need a fix that respects the original resource-scoping behavior, ensuring only necessary resources start per test class, as in Quarkus <= 3.15.

[geoand]

Please provide us with a sample application were we cansee the difference in action thus making said fix much more likely

[snazy]

Here's a reproducer: #44279
The added tests verify that only the expected test-resources are started (uses TestInjector for this), but fails because all test resources are started.

There are three resources - all with scope MATCHING_RESOURCES.
One for "test one", one for "test two" and one that's shared but with different init-args. The shared test-resource is also started twice, but associates the wrong test-resource to the test (wrong value injected into the test class).

Sample output:

[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.301 s <<< FAILURE! -- in io.quarkus.it.extension.testresources.WithResourcesPoliciesFirstTest
[ERROR] io.quarkus.it.extension.testresources.WithResourcesPoliciesFirstTest.checkOnlyResource1started -- Time elapsed: 0.141 s <<< FAILURE!
org.opentest4j.AssertionFailedError: 

expected: ["resource1", null, "test-one"]
 but was: ["SomeResource1", "SomeResource2", "test-two"]
	at io.quarkus.it.extension.testresources.WithResourcesPoliciesFirstTest.checkOnlyResource1started(WithResourcesPoliciesFirstTest.java:28)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at io.quarkus.test.junit.QuarkusTestExtension.runExtensionMethod(QuarkusTestExtension.java:971)
	at io.quarkus.test.junit.QuarkusTestExtension.interceptTestMethod(QuarkusTestExtension.java:821)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

[INFO] Running io.quarkus.it.extension.testresources.WithResourcesPoliciesSecondTest
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.013 s <<< FAILURE! -- in io.quarkus.it.extension.testresources.WithResourcesPoliciesSecondTest
[ERROR] io.quarkus.it.extension.testresources.WithResourcesPoliciesSecondTest.checkOnlyResource1started -- Time elapsed: 0.001 s <<< FAILURE!
org.opentest4j.AssertionFailedError: 

expected: [null, "resource2", "test-two"]
 but was: ["SomeResource1", "SomeResource2", "test-two"]
	at io.quarkus.it.extension.testresources.WithResourcesPoliciesSecondTest.checkOnlyResource1started(WithResourcesPoliciesSecondTest.java:28)
	at java.base/java.lang.reflect.Method.invoke(Method.java:569)
	at io.quarkus.test.junit.QuarkusTestExtension.runExtensionMethod(QuarkusTestExtension.java:971)
	at io.quarkus.test.junit.QuarkusTestExtension.interceptTestMethod(QuarkusTestExtension.java:821)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)

[geoand]

Thanks @snazy!

[snazy]

Maybe a bit clearer, when just running one of the added tests:

./mvnw -pl integration-tests/test-extension/tests install -Dtest=WithResourcesPoliciesFirstTest



...
SomeResource1 start
SharedResource start with arg 'test-one'
SomeResource2 start
SharedResource start with arg 'test-two'
...
[ERROR] io.quarkus.it.extension.testresources.WithResourcesPoliciesFirstTest.checkOnlyResource1started -- Time elapsed: 0.143 s <<< FAILURE!
org.opentest4j.AssertionFailedError: 

expected: ["resource1", null, "test-one"]
 but was: ["SomeResource1", "SomeResource2", "test-two"]
	at io.quarkus.it.extension.testresources.WithResourcesPoliciesFirstTest.checkOnlyResource1started(WithResourcesPoliciesFirstTest.java:28)

...

SomeResource1 stop
SharedResource stop
SomeResource2 stop
SharedResource stop

Starting SomeResource2 and one of the SharedResource is wrong here.

[geoand]

@snazy @hemanth-web-dev can you please describe to me what your Quarkus 3.15 test resource setup was like (the case of the reproducer however indeed quite clear)?

The reason I am asking here is that it is not clear at all to me what you expect the semantic of MATCHING_RESOURCES to be when used with resources that are not part of a class.

[geoand]

In any case, can you give the updated version of #44279 a try? It should fix the problem

[snazy]

@snazy @hemanth-web-dev can you please describe to me what your Quarkus 3.15 test resource setup was like (the case of the reproducer however indeed quite clear)?

What's in the reproducer (minus cosmetic things like omitting default annotation values).

[geoand]

Thanks @snazy