Flaky Tests rerun report overrides/overrule the actual full test run report.

[naveedriay]

Summary

Flaky Tests rerun report overrides / takes precedence over the actual full test run report.
Under the rerun Flaky test runner implementation, when my tests actually encounters a flaky test, it goes into that flaky test's execution at the end of build, but test report (based on cucumber.json) only showing flaky tests results and totally ignores/ exclude the actual full test run report.

Expected Behavior

When under the flaky test's rerun mechanism, the flaky tests gets executed at the end of the build, its report should not override / overrule the actual full test report. Ideally it should combine the both original test run report & flaky test run report in cucumber.json

Steps to Reproduce (for bugs)

For detailed steps, please visit (https://stackoverflow.com/questions/50115819/cucumber-json-report-getting-overwritten-by-rerun-scenario-report)
I originally raised it there, thinking that I might be doing something wrong somewhere, but eventually it comes out to be a bug/issue with cucumber-jvm own report generation problem.

1. I have cucumber-jvm, junit, maven project testing UI website. For pom.xml view, visit above url. Pom file using following plugins and dependencies.

maven-surefire-plugin version=2.21.0 with rerunFailingTestCount=1
maven-compiler-plugin version= 3.7.0
dependencies includes, cucumber-java, cucumber-junit, cucumber-jvm, cucumber-spring all based on version number 2.4.0

2. I have used a conventional runner mechanism in place with following code.

@RunWith(Cucumber.class)
@ContextConfiguration(locations = {"file:/src/test/resources/spring-config.xml"})
@CucumberOptions(
glue = "com.test.uitests",
features = "classpath:cucumber",
tags = {"~@ignore", "@ui_tests"},
monochrome = true,
plugin = {"pretty", "html:target/cucumber-reports",
"json:target/cucumber-reports/cucumber.json",
"rerun:target/rerun.txt"} //Creates a text file with failed scenarios
)
public class AllTestsRunner {
}

3. In my feature file, i have 5 scenarios. If all of them pass in 1st run. It successfully generates the report with cucumber.json report showing all 5 scenarios. But, if (say) 2 out of 5 scenarios fails, and those gets executed automatically in a rerun mechanism, the cucumber.json report file only recording results of those two scenarios and not the all 5 scenarios. Overall build PASSES if those 2 scenarios passes in rerun or FAILs if those 2 scenarios fails. That's correct, but my problem is with the cucumber.json not recording the actual full test run results. I have tried to use the maven-cucumber-reporting plugin v_3.16.0 but it actually reads the cucumber.json file itself and hence don't resolve my problem.

Context & Motivation

Your Environment

OS used: Windows 7
Tool: IntelliJ IDEA 2016.2

[mpkorstanje]

The good news is you're not doing anything wrong!

The bad news is that the results you're observing are entirely as expected. This is a natural consequence of chaining different tools (Surefire --> JUnit --> Cucumber) that are otherwise unaware of each other. From the perspective of Cucumber it would appear that the rerun is an entirely new execution so it will happily overwrite the old reports. Only at the start of the chain it is possible to create accurate reports.

As such your options from least to most effort and worst to best quality are:

1. Use the reports generated by Surefire.
2. Use the cucumber-jvm-parallel-plugin create a individual unit test for each scenario. This will still overwrite the failed report, but only for that scenario. cucumber-reporting can combine these, but it's not as pretty.
3. Write your own plugin that appends results rather then overwriting them.
4. Get involved with the Cucumber project and help resolve this fundamentally.

[naveedriay]

Thanks for your reply on this issue.
I will try to get the solution via Option-3 above but if not successful, then i will take the Option-2 to work for me. Thanks, you can close this issue.

[Abhishek29github]

Is this issue resolved

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.