Enabling the Post-Integration-test phase to run on test failure

[nab0310]

Hello,

I want to use the scalatest-maven-plugin to run dockerized integration tests. Similarly to #45 I was stuck because a failing test would fail the build immediately. One solution that I came up with for remedying this situation was to make a flag, runVerifyOnFailure that would cause the setup for a verify stage to be preformed within the test phase if there was a failure. Then you would have to include the verify stage on your plugin config and that is where it would detect the failures. Because the verfiy stage runs after the post-integration-test phase in maven, the docker cleanup would have already happened, remedying the issue.

I'm just seeking feedback on if this would even be something feasible that the maintainers would want in the project. Heres a rough draft of what I was thinking on my fork: nab0310@65913b2

[axiopisty]

@nab0310 I'm experiencing the same problem.

Will you provide an example of your configuration of the scalatest-maven-plugin that enabled you to work around this?

---

My scenario is as follows:

I have a multi-module maven project. The main module produces a docker image as an artifact. Another module (the last one to run in my build) is used to build and run end-to-end tests using a combination of the fabric8/docker-maven-plugin to start the container built from my main artifact along with all the other containers it requires as dependencies, and the scalatest-maven-plugin to execute my end-to-end integration tests.

Looking at the project build from the perspective of maven phases results in the following simplified view:

Maven Phase -> Plugin:goal

• package -> docker-maven-plugin:docker.build
• pre-integration-tests -> docker-maven-plugin:docker.start
• integration-test -> scalatest-maven-plugin:test
• post-integration-tests -> docker-maven-plugin:docker.stop
• verify -> (this is where the maven-failsafe-plugin fails the build if tests failed)

Using the scalatest-maven-plugin, if tests fail, the docker containers don't get cleaned up in the post-integration-tests phase.

This issue gets a big +1 vote from me. Fixing it would make this plugin work more like the maven-failsafe-plugin (which could be used as a reference for how to implement this feature request) rather than the maven-surefire-plugin.

[axiopisty]

I did a little research on the maven-failsafe-plugin to see how it delays reporting the failed build status until the verify phase. Basically, the maven-failsafe-plugin is a simple plugin closely related to the maven-surefire-plugin, they are maintained within the same code base. So how does it work?

The maven-failsafe-plugin has only 2 goals (mojos), the IntegrationTestMojo bound to the integration-test maven phase. This mojo runs the tests and creates a report summary file called failsafe-summary.xml which is simply a file which reports the status of the tests that were executed.

It contains details such as how many tests:

• completed
• had errors
• had failures
• were skipped

Based on configuration there could be many summary reports of this style. All these are created by the IntegrationTestMojo during the integration-test phase. So, simply put, that Mojo does not fail the build by throwing MojoFailureException if there are any problems (tests which either had errors or failed). It just writes the results to a file.

Then there is the VerifyMojo. That goal is bound to the verify phase. All it does (over simplification here) is to read the test summary files that were generated by the IntegrationTestMojo and check to see if any of the tests would have caused the build to fail. If any did, then it will fail the build by throwing MojoFailureException.

Failing the build in this manner allows the post-integration-tests phase to execute and clean up any resources that were allocated in the pre-integration-tests phase.

The scalatest-maven-plugin could easily implement a similar approach. All it would need to do is create a summary report as output for the TestMojo and ReporterMojo. Those mojos should never throw MojoFailureException. Then a VerifyMojo should be created to read the summary reports and throw MojoFailureException if the build needs to be failed.

[nab0310]

@axiopisty Using my fork, my maven config is as follows. All I did in my fork was to write out to a file the word "failure" and then read it in the verify stage. This was just a rough prototype because I wanted to gage interest from the maintainers if they even wanted something like this as a feature.

In order to make this a viable solution there would have to be some more work done in understanding how the maven-failsafe-plugin writes the reports it reads from and just implement a solution that is similar in the scalatest-maven-plugin. The one thing I don't know is if this "fail-fast" behavior (of throwing a MojoFailureException whenever a failure occurs) was implemented for a reason and that changing it would go against someone's vision for the project. However, I think a change like this would only increase compatibility and given how prevalent docker is today, I think it would be a good design decision that would help what I presume to be a lot of people.

<plugin>
  <groupId>org.scalatest</groupId>
  <artifactId>scalatest-maven-plugin</artifactId>
  <version>3.0.0</version>
    <configuration>
      <runVerifyOnFailure>true</runVerifyOnFailure>
      <systemProperties>
        <java.awt.headless>true</java.awt.headless>
        <mysql.port>${mysql.port}</mysql.port>
      </systemProperties>
    </configuration>
    <executions>
      <execution>
        <id>test</id>
        <phase>integration-test</phase>
        <goals>
          <goal>test</goal>
        </goals>
      </execution>
      <execution>
        <id>verify</id>
        <goals>
          <goal>verify</goal>
        </goals>
      </execution>
    </executions>
</plugin>

[axiopisty]

@nab0310 I think it would be helpful if you were to open a PR with your changes so we can review the changes you made on your branch. If you do, please reference this issue in your PR.

[nab0310]

I opened the PR

[rondefreitas]

I'm running into a similar issue with docker tests. It would be useful to have this functionality

[cheeseng]

@nab0310 @axiopisty I wonder if it is possible if we can produce the same failsure-summary.xml and let maven-failsafe-plugin's VerifyMojo to do its task? The first to achieve that perhaps is that we should not require to disable surefire?

[axiopisty]

@cheeseng Perhaps this plugin could create the xml file and delegate the verification to the failsafe plugin. But it seems to me that it might be easier to just create the functionality directly within this plugin rather than trying to couple everything together with the failsafe plugin. I doubt the maintainers of the surefire and failsafe plugins (which are both maintained from the same source repository) would be considering the xml files they generate and read to be part of some public API. So I tend to think it would be easier to just replicate the idea and implement a solution directly within the scalatest-maven-plugin.

[axiopisty]

@cheeseng I doubt that maintainers of the scalatest-maven-plugin would want to start receiving issues being reported by people who might be using both the scalatest-maven-plugin and the surefire/failsafe plugins. To quote Offspring: "You gotta keep 'em separated!"

[cheeseng]

@axiopisty That's true, looking at @nab0310 's it appears to be very near we can have it already. The PR looks good just I don't have a good way to test it.

[axiopisty]

I wonder if there might be examples of test cases in the failsafe repository.

[cheeseng]

@axiopisty I added a comment to the PR here:

https://github.com/scalatest/scalatest-maven-plugin/pull/60/files#r911599853

I don't quite get how verify stage work yet, by any chance do you mind to check if that's intended behavior to always throw the exception?

Cheers.

[axiopisty]

We stopped using this plugin in our project for a variety of reasons. Our project has been completely rewritten and doesn't even use scala anymore. So I don't have any easy way to contribute anything other than what I already have in comments. Sorry.

[cheeseng]

@axiopisty No worry, thanks for your comments so far, they are useful! :)