Adds argLine to instrument Integration Tests #17677

bvarner · 2021-06-03T16:44:34Z

Fix Quarkus code coverage with failsafe #7536 by adding the ability to specify an argLine property when running tests.

For Example, in a project I'm building:

            <plugin>
                <artifactId>maven-failsafe-plugin</artifactId>
                <version>${surefire-plugin.version}</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>integration-test</goal>
                            <goal>verify</goal>
                        </goals>
                        <configuration>
                            <systemPropertyVariables>
                                <java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager>
                                <maven.home>${maven.home}</maven.home>
                                <quarkus.test.native-image-profile>it</quarkus.test.native-image-profile>
                                <quarkus.test.argLine>${argLine}</quarkus.test.argLine>
                            </systemPropertyVariables>
                        </configuration>
                    </execution>
                </executions>
            </plugin>

Fixes: #17760

bvarner · 2021-06-03T18:06:16Z

With the code from this PR, I've been able to define my plugins as this:

And get a coverage report containing the results of both the surefire and failsafe coverage.
This is highly helpful, especially when enforcing quality gates.

    <build>
        <plugins>
            <plugin>
                <groupId>org.jacoco</groupId>
                <artifactId>jacoco-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <id>default-prepare-agent-integration</id>
                        <goals>
                            <goal>prepare-agent-integration</goal>
                        </goals>
                        <configuration>
                            <destFile>${project.build.directory}/jacoco-quarkus.exec</destFile>
                            <append>true</append>
                            <exclClassLoaders>*QuarkusClassLoader</exclClassLoaders>
                        </configuration>
                    </execution>
                    <execution>
                        <id>report-it</id>
                        <phase>post-integration-test</phase>
                        <goals>
                            <goal>report</goal>
                        </goals>
                        <configuration>
                            <dataFile>${project.build.directory}/jacoco-quarkus.exec</dataFile>
                            <outputDirectory>${project.build.directory}/jacoco-report</outputDirectory>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <artifactId>maven-failsafe-plugin</artifactId>
                <version>${surefire-plugin.version}</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>integration-test</goal>
                            <goal>verify</goal>
                        </goals>
                        <configuration>
                            <systemPropertyVariables>
                                <java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager>
                                <maven.home>${maven.home}</maven.home>
                                <quarkus.test.native-image-profile>it</quarkus.test.native-image-profile>
                                <quarkus.test.argLine>${argLine}</quarkus.test.argLine>
                            </systemPropertyVariables>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            ...
       </plugins>
   </build>

bvarner · 2021-06-03T21:47:40Z

@geoand and @glefloch New PR from main, closed #17649.

stuartwdouglas · 2021-06-03T23:45:45Z

test-framework/common/src/main/java/io/quarkus/test/common/JarLauncher.java

@@ -39,6 +40,7 @@ private JarLauncher(Path jarPath, Config config) {
                config.getValue("quarkus.http.test-port", OptionalInt.class).orElse(DEFAULT_PORT),
                config.getValue("quarkus.http.test-ssl-port", OptionalInt.class).orElse(DEFAULT_HTTPS_PORT),
                config.getValue("quarkus.test.jar-wait-time", OptionalLong.class).orElse(DEFAULT_JAR_WAIT_TIME),
+                config.getOptionalValue("quarkus.test.argLine", String.class).orElse(null),


This needs to be added to io.quarkus.deployment.dev.testing.TestConfig so it ends up in the 'all properties' documentation.

It would also be good to include this in the testing documentation, as we don't have any info at the moment on doing code coverage with @QuarkusIntegrationTest.

Alright. So wrt the io.quarkus.deployment.dev.testing.TestConfig I'm a bit confused.
I see in that class

/** * The profile to use when testing the native image */ @ConfigItem(defaultValue = "prod") String nativeImageProfile;

Which seems a bit misleading, since it's not prod explicitly as a default -- but rather null.

So, while I patterned the implementation of this OptionalValue off that, I feel like I probably should not define a defaultValue for the @ConfigItem, does that sound right?

wrt the testing documentation -- where would I find that, and I'm not convinced I'm your guy for documenting that.

I've found the testing documentation to be a bit ... lacking... especially in the realm of Integration testing, and what-not.

Yea, the default value should be null. Also the property name should be something line quarkus.test.integration-jvm-arg-line to fit in with our naming convention, and also to make it clear where this applies so you have something like:

/** * JVM parameters that are used to launch jar based integration tests. */ @ConfigItem String integrationJvmArgLine;

The documentation is at: https://github.com/quarkusio/quarkus/blob/main/docs/src/main/asciidoc/getting-started-testing.adoc#16-using-quarkusintegrationtest

Documentation and TestConfig updated. Let me know what else I can do, or any other feedback you have.

@stuartwdouglas I'm not sure why this is failing, it's also failing locally.
It looks like I needed to add this ConfigItem to other classes as well, but I can't quite seem to put my finger on it.

Oh wow. How'd I miss that it needs to be Optional<String>? 🤦

quarkus-bot · 2021-06-04T09:59:02Z

This workflow status is outdated as a new workflow run has been triggered.

Failing Jobs - Building `6d121e4`

Status	Name	Step	Test failures	Logs	Raw logs
✖	Maven Tests - JDK 11	`Build`	Test failures	Logs	Raw logs

Full information is available in the Build summary check run.

Test Failures

⚙️ Maven Tests - JDK 11 #

📦 integration-tests/maven

✖ io.quarkus.maven.it.DevMojoIT.testThatNewResourcesAreServed line 849 - More details - Source on GitHub

geoand

LGTM, thanks a lot!

quarkus-bot · 2021-06-04T14:40:37Z

This workflow status is outdated as a new workflow run has been triggered.

Failing Jobs - Building `122ee10`

Status	Name	Step	Test failures	Logs	Raw logs
✖	Initial JDK 11 Build	`Build`	⚠️ Check →	Logs	Raw logs

geoand · 2021-06-04T19:24:27Z

Please squash the commits when you are done

stuartwdouglas

LGTM, thanks a lot

stuartwdouglas · 2021-06-06T23:43:50Z

docs/src/main/asciidoc/getting-started-testing.adoc

@@ -1088,6 +1088,89 @@ Finally, if a container image was created during the build (by using including t

 As is the case with `@NativeImageTest`, this is a black box test that supports the same set features and has the same limitations.

+=== Code Coverage for Integration Tests


Actually can you move this to https://github.com/quarkusio/quarkus/blob/c28a2276d9fbdc5c438c19b39b5b357daee58424/docs/src/main/asciidoc/tests-with-coverage.adoc

Otherwise looks good.

Change pushed.

quarkus-bot · 2021-06-08T06:12:38Z

This workflow status is outdated as a new workflow run has been triggered.

Failing Jobs - Building `c66481c`

Status	Name	Step	Test failures	Logs	Raw logs
✖	Initial JDK 11 Build	`Build`	⚠️ Check →	Logs	Raw logs

geoand · 2021-06-08T06:14:25Z

Seems like the imports are not supported. Can you please run mvn process-sources, commit and squash the changes?

Thanks

… documentation, and testing documentation on how to configure. * Fix quarkusio#7536 by adding the ability to specify an argLine property when running tests. For Example, in a project I'm building: ``` <plugin> <artifactId>maven-failsafe-plugin</artifactId> <version>${surefire-plugin.version}</version> <executions> <execution> <goals> <goal>integration-test</goal> <goal>verify</goal> </goals> <configuration> <systemPropertyVariables> <java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager> <maven.home>${maven.home}</maven.home> <quarkus.test.native-image-profile>it</quarkus.test.native-image-profile> <argLine>${argLine}</argLine> </systemPropertyVariables> </configuration> </execution> </executions> </plugin> ```

quarkus-bot bot added the area/testing label Jun 3, 2021

stuartwdouglas requested changes Jun 3, 2021

View reviewed changes

quarkus-bot bot added area/core area/documentation labels Jun 4, 2021

geoand approved these changes Jun 4, 2021

View reviewed changes

geoand requested a review from stuartwdouglas June 4, 2021 14:26

bvarner force-pushed the feature/jar-launcher-argline branch from b03843b to 887897e Compare June 4, 2021 19:33

stuartwdouglas approved these changes Jun 6, 2021

View reviewed changes

stuartwdouglas approved these changes Jun 8, 2021

View reviewed changes

geoand added triage/waiting-for-ci Ready to merge when CI successfully finishes triage/backport? labels Jun 8, 2021

geoand mentioned this pull request Jun 8, 2021

QuarkusIntegrationTest on java jar, should support Jacoco #17760

Closed

bvarner force-pushed the feature/jar-launcher-argline branch from c66481c to e6cd85e Compare June 8, 2021 14:09

stuartwdouglas merged commit de502dc into quarkusio:main Jun 9, 2021

quarkus-bot bot added this to the 2.1 - main milestone Jun 9, 2021

quarkus-bot bot removed the triage/waiting-for-ci Ready to merge when CI successfully finishes label Jun 9, 2021

gsmet modified the milestones: 2.1 - main, 2.0.0.Final Jun 10, 2021

gsmet removed the triage/backport? label Jun 10, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds argLine to instrument Integration Tests #17677

Adds argLine to instrument Integration Tests #17677

bvarner commented Jun 3, 2021 •

edited by geoand

Loading

bvarner commented Jun 3, 2021 •

edited

Loading

bvarner commented Jun 3, 2021 •

edited

Loading

stuartwdouglas Jun 3, 2021

bvarner Jun 4, 2021

bvarner Jun 4, 2021

stuartwdouglas Jun 4, 2021

geoand Jun 4, 2021

bvarner Jun 4, 2021

bvarner Jun 4, 2021

bvarner Jun 4, 2021

quarkus-bot bot commented Jun 4, 2021 •

edited

Loading

Failing Jobs - Building `6d121e4`

Test Failures

⚙️ Maven Tests - JDK 11 #

📦 integration-tests/maven

geoand left a comment •

edited

Loading

quarkus-bot bot commented Jun 4, 2021 •

edited

Loading

Failing Jobs - Building `122ee10`

geoand commented Jun 4, 2021

stuartwdouglas left a comment

stuartwdouglas Jun 6, 2021

bvarner Jun 7, 2021

quarkus-bot bot commented Jun 8, 2021 •

edited

Loading

Failing Jobs - Building `c66481c`

geoand commented Jun 8, 2021

		@@ -1088,6 +1088,89 @@ Finally, if a container image was created during the build (by using including t

		As is the case with `@NativeImageTest`, this is a black box test that supports the same set features and has the same limitations.

		=== Code Coverage for Integration Tests

Adds argLine to instrument Integration Tests #17677

Adds argLine to instrument Integration Tests #17677

Conversation

bvarner commented Jun 3, 2021 • edited by geoand Loading

bvarner commented Jun 3, 2021 • edited Loading

bvarner commented Jun 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

quarkus-bot bot commented Jun 4, 2021 • edited Loading

Failing Jobs - Building 6d121e4

Test Failures

⚙️ Maven Tests - JDK 11 #

📦 integration-tests/maven

geoand left a comment • edited Loading

Choose a reason for hiding this comment

quarkus-bot bot commented Jun 4, 2021 • edited Loading

Failing Jobs - Building 122ee10

geoand commented Jun 4, 2021

stuartwdouglas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

quarkus-bot bot commented Jun 8, 2021 • edited Loading

Failing Jobs - Building c66481c

geoand commented Jun 8, 2021

bvarner commented Jun 3, 2021 •

edited by geoand

Loading

bvarner commented Jun 3, 2021 •

edited

Loading

bvarner commented Jun 3, 2021 •

edited

Loading

quarkus-bot bot commented Jun 4, 2021 •

edited

Loading

Failing Jobs - Building `6d121e4`

geoand left a comment •

edited

Loading

quarkus-bot bot commented Jun 4, 2021 •

edited

Loading

Failing Jobs - Building `122ee10`

quarkus-bot bot commented Jun 8, 2021 •

edited

Loading

Failing Jobs - Building `c66481c`