Some tests are skipped on CI

[felixbarny]

Looking at the test report from the latest build on master, it seems not all tests are executed: https://apm-ci.elastic.co/job/apm-agent-java/job/apm-agent-java-mbp/job/master/432/testReport/

These are the packages that contain executed tests:

co.elastic.apm.agent.asynchttpclient
co.elastic.apm.agent.concurrent
co.elastic.apm.agent.configuration
co.elastic.apm.agent.es.restclient.v5_6
co.elastic.apm.agent.es.restclient.v6_4
co.elastic.apm.agent.es.restclient.v7_1
co.elastic.apm.agent.httpclient
co.elastic.apm.agent.jaxrs
co.elastic.apm.agent.jdbc
co.elastic.apm.agent.jms
co.elastic.apm.agent.jms.spring
co.elastic.apm.agent.kafka
co.elastic.apm.agent.log.shipper
co.elastic.apm.agent.mongoclient
co.elastic.apm.agent.okhttp
co.elastic.apm.agent.redis.lettuce
co.elastic.apm.agent.report
co.elastic.apm.agent.resttemplate
co.elastic.apm.agent.scala.concurrent
co.elastic.apm.agent.spring.webmvc.exception
co.elastic.apm.agent.urlconnection
co.elastic.apm.agent.util
co.elastic.apm.api
co.elastic.apm.servlet
co.elastic.apm.spring.boot

For example, the Dubbo tests and co.elastic.apm.agent.error.logging are missing here.

Looking at the logs, the Dubbo tests are skipped for some reason.

[2020-06-30T12:06:03.536Z] [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ apm-dubbo-plugin ---
[2020-06-30T12:06:03.536Z] [INFO] Tests are skipped.

Another snippet from the logs for the apm-error-logging-plugin

[2020-06-30T12:04:08.302Z] [INFO] --------------< co.elastic.apm:apm-error-logging-plugin >---------------
[2020-06-30T12:04:08.302Z] [INFO] Building co.elastic.apm:apm-error-logging-plugin 1.17.1-SNAPSHOT  [9/77]
[2020-06-30T12:04:08.302Z] [INFO] --------------------------------[ jar ]---------------------------------
[2020-06-30T12:04:08.302Z] [INFO] 
[2020-06-30T12:04:08.302Z] [INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ apm-error-logging-plugin ---
[2020-06-30T12:04:08.302Z] [INFO] 
[2020-06-30T12:04:08.302Z] [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-java) @ apm-error-logging-plugin ---
[2020-06-30T12:04:08.302Z] [INFO] 
[2020-06-30T12:04:08.302Z] [INFO] --- jacoco-maven-plugin:0.8.5:prepare-agent (default) @ apm-error-logging-plugin ---
[2020-06-30T12:04:08.302Z] [INFO] argLine set to -javaagent:/var/lib/jenkins/workspace/t-java_apm-agent-java-mbp_master/src/github.com/elastic/apm-agent-java/.m2/org/jacoco/org.jacoco.agent/0.8.5/org.jacoco.agent-0.8.5-runtime.jar=destfile=/var/lib/jenkins/workspace/t-java_apm-agent-java-mbp_master/src/github.com/elastic/apm-agent-java/apm-agent-plugins/apm-error-logging-plugin/target/jacoco.exec
[2020-06-30T12:04:08.302Z] [INFO] 
[2020-06-30T12:04:08.302Z] [INFO] --- license-maven-plugin:1.19:update-file-header (first) @ apm-error-logging-plugin ---
[2020-06-30T12:04:08.302Z] [WARNING] The failOnMissingHeader has no effect if the property dryRun is not set.
[2020-06-30T12:04:08.302Z] [WARNING] The failOnNotUptodateHeader has no effect if the property dryRun is not set.
[2020-06-30T12:04:08.302Z] [INFO] adding extra resolver file:////var/lib/jenkins/workspace/t-java_apm-agent-java-mbp_master/src/github.com/elastic/apm-agent-java/apm-agent-plugins/apm-error-logging-plugin/../../licenses
[2020-06-30T12:04:08.302Z] [INFO] Will search files to update from root /var/lib/jenkins/workspace/t-java_apm-agent-java-mbp_master/src/github.com/elastic/apm-agent-java/apm-agent-plugins/apm-error-logging-plugin/src/main/java
[2020-06-30T12:04:08.302Z] [INFO] Will search files to update from root /var/lib/jenkins/workspace/t-java_apm-agent-java-mbp_master/src/github.com/elastic/apm-agent-java/apm-agent-plugins/apm-error-logging-plugin/src/test/java
[2020-06-30T12:04:08.302Z] [INFO] Scan 7 files header done in 4.589ms.
[2020-06-30T12:04:08.302Z] [INFO] All files are up-to-date.
[2020-06-30T12:04:08.302Z] [INFO] 
[2020-06-30T12:04:08.302Z] [INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ apm-error-logging-plugin ---
[2020-06-30T12:04:08.302Z] [INFO] Using 'UTF-8' encoding to copy filtered resources.
[2020-06-30T12:04:08.302Z] [INFO] Copying 1 resource
[2020-06-30T12:04:08.303Z] [INFO] 
[2020-06-30T12:04:08.303Z] [INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ apm-error-logging-plugin ---
[2020-06-30T12:04:08.303Z] [INFO] Changes detected - recompiling the module!
[2020-06-30T12:04:08.303Z] [INFO] Compiling 4 source files to /var/lib/jenkins/workspace/t-java_apm-agent-java-mbp_master/src/github.com/elastic/apm-agent-java/apm-agent-plugins/apm-error-logging-plugin/target/classes
[2020-06-30T12:04:08.564Z] [WARNING] bootstrap class path not set in conjunction with -source 7
[2020-06-30T12:04:08.564Z] [INFO] 
[2020-06-30T12:04:08.564Z] [INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ apm-error-logging-plugin ---
[2020-06-30T12:04:08.564Z] [INFO] Using 'UTF-8' encoding to copy filtered resources.
[2020-06-30T12:04:08.564Z] [INFO] skip non existing resourceDirectory /var/lib/jenkins/workspace/t-java_apm-agent-java-mbp_master/src/github.com/elastic/apm-agent-java/apm-agent-plugins/apm-error-logging-plugin/src/test/resources
[2020-06-30T12:04:08.564Z] [INFO] 
[2020-06-30T12:04:08.564Z] [INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ apm-error-logging-plugin ---
[2020-06-30T12:04:08.564Z] [INFO] Changes detected - recompiling the module!
[2020-06-30T12:04:08.564Z] [INFO] Compiling 3 source files to /var/lib/jenkins/workspace/t-java_apm-agent-java-mbp_master/src/github.com/elastic/apm-agent-java/apm-agent-plugins/apm-error-logging-plugin/target/test-classes
[2020-06-30T12:04:08.564Z] [WARNING] bootstrap class path not set in conjunction with -source 9
[2020-06-30T12:04:08.564Z] [INFO] 
[2020-06-30T12:04:08.564Z] [INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ apm-error-logging-plugin ---
[2020-06-30T12:04:08.564Z] [INFO] Tests are skipped.
[2020-06-30T12:04:08.564Z] [INFO] 
[2020-06-30T12:04:08.564Z] [INFO] --- jacoco-maven-plugin:0.8.5:report (report) @ apm-error-logging-plugin ---
[2020-06-30T12:04:08.564Z] [INFO] Skipping JaCoCo execution due to missing execution data file.
[2020-06-30T12:04:08.564Z] [INFO] 
[2020-06-30T12:04:08.564Z] [INFO] --- maven-jar-plugin:3.2.0:jar (default-jar) @ apm-error-logging-plugin ---
[2020-06-30T12:04:08.564Z] [INFO] Building jar: /var/lib/jenkins/workspace/t-java_apm-agent-java-mbp_master/src/github.com/elastic/apm-agent-java/apm-agent-plugins/apm-error-logging-plugin/target/apm-error-logging-plugin-1.17.1-SNAPSHOT.jar
[2020-06-30T12:04:08.564Z] [INFO] 
[2020-06-30T12:04:08.564Z] [INFO] --- maven-javadoc-plugin:3.1.1:jar (default) @ apm-error-logging-plugin ---
[2020-06-30T12:04:08.564Z] [INFO] Skipping javadoc generation
[2020-06-30T12:04:08.564Z] [INFO] 
[2020-06-30T12:04:08.564Z] [INFO] --- maven-failsafe-plugin:2.19.1:integration-test (default) @ apm-error-logging-plugin ---
[2020-06-30T12:04:08.564Z] [INFO] Tests are skipped.
[2020-06-30T12:04:08.564Z] [INFO] 
[2020-06-30T12:04:08.564Z] [INFO] --- animal-sniffer-maven-plugin:1.17:check (signature-check) @ apm-error-logging-plugin ---
[2020-06-30T12:04:08.564Z] [INFO] Checking unresolved references to org.codehaus.mojo.signature:java17:1.0
[2020-06-30T12:04:09.136Z] [INFO] 
[2020-06-30T12:04:09.136Z] [INFO] --- maven-failsafe-plugin:2.19.1:verify (default) @ apm-error-logging-plugin ---
[2020-06-30T12:04:09.136Z] [INFO] Tests are skipped.
[2020-06-30T12:04:09.136Z] [INFO] 
[2020-06-30T12:04:09.136Z] [INFO] --- license-maven-plugin:1.19:check-file-header (license-header-check) @ apm-error-logging-plugin ---
[2020-06-30T12:04:09.136Z] [INFO] adding extra resolver file:////var/lib/jenkins/workspace/t-java_apm-agent-java-mbp_master/src/github.com/elastic/apm-agent-java/apm-agent-plugins/apm-error-logging-plugin/../../licenses
[2020-06-30T12:04:09.136Z] [INFO] Will search files to update from root /var/lib/jenkins/workspace/t-java_apm-agent-java-mbp_master/src/github.com/elastic/apm-agent-java/apm-agent-plugins/apm-error-logging-plugin/src/main/java
[2020-06-30T12:04:09.136Z] [INFO] Will search files to update from root /var/lib/jenkins/workspace/t-java_apm-agent-java-mbp_master/src/github.com/elastic/apm-agent-java/apm-agent-plugins/apm-error-logging-plugin/src/test/java
[2020-06-30T12:04:09.136Z] [INFO] Scan 7 files header done in 4.135ms.
[2020-06-30T12:04:09.137Z] [INFO] All files are up-to-date.
[2020-06-30T12:04:09.137Z] [INFO] 
[2020-06-30T12:04:09.137Z] [INFO] --- maven-install-plugin:3.0.0-M1:install (default-install) @ apm-error-logging-plugin ---
[2020-06-30T12:04:09.137Z] [INFO] Installing /var/lib/jenkins/workspace/t-java_apm-agent-java-mbp_master/src/github.com/elastic/apm-agent-java/apm-agent-plugins/apm-error-logging-plugin/target/apm-error-logging-plugin-1.17.1-SNAPSHOT.jar to /var/lib/jenkins/workspace/t-java_apm-agent-java-mbp_master/src/github.com/elastic/apm-agent-java/.m2/co/elastic/apm/apm-error-logging-plugin/1.17.1-SNAPSHOT/apm-error-logging-plugin-1.17.1-SNAPSHOT.jar
[2020-06-30T12:04:09.137Z] [INFO] Installing /var/lib/jenkins/workspace/t-java_apm-agent-java-mbp_master/src/github.com/elastic/apm-agent-java/apm-agent-plugins/apm-error-logging-plugin/pom.xml to /var/lib/jenkins/workspace/t-java_apm-agent-java-mbp_master/src/github.com/elastic/apm-agent-java/.m2/co/elastic/apm/apm-error-logging-plugin/1.17.1-SNAPSHOT/apm-error-logging-plugin-1.17.1-SNAPSHOT.pom

[felixbarny]

Seems like #1263 didn't quite do the trick. Maybe it's the update of the surefire plugin from #1048 that caused this. See also the test trend chart: https://apm-ci.elastic.co/job/apm-agent-java/job/apm-agent-java-mbp/job/master/

[felixbarny]

As of version 2.22.0 of the surefire plugin, there's a new way to declare dependencies to the test runners: https://junit.org/junit5/docs/current/user-guide/#running-tests-build-maven