Corrupted Cucumber.xml when using surefire.rerunFailingTestsCount parameter #2709

carmenloghin · 2023-03-20T10:09:08Z

👓 What did you see?

Using Cucumber 7.11.0 or 7.11.1 and surefire.rerunFailingTestsCount parameter in order to run the UI tests pack, I noticed the report is no longer generated on Azure DevOps and the following warning message is displayed:

##[warning]Failed to read /home/vsts/work/1/s/ui/target/cucumber-xml-reports/cucumber.xml. Error : Unexpected end of file while parsing Name has occurred. Line 44, position 10..

✅ What did you expect to see?

With other version of Cucumber (tested with versions from 7.5 to 7.10), I saw the report generated like

📦 Which tool/library version are you using?

Cucumber 7.11.0/7.11.1, Selenium 4.8, Java 11, Maven compiler 3.10, Maven Surefire Plugin 3.0.0-M7, Azure DevOps

🔬 How could we reproduce it?

Steps to reproduce the behavior:

Install 'Cucumber' version '7.11.0 or 7.11.1', Java version 11, Selenium version 4.8, Maven compiler version 3.10, Maven Surefire Plugin version 3.0.0-M7
Have a feature file (with 4 scenarios, 2 of them need to fail in order to be rerun again and the cucumber report to be generated with the new results ) which run in an Azure DevOps pipeline, on a Docker machine, with the maven command: '-Dcucumber.filter.tags="@smoke" -Dsurefire.rerunFailingTestsCount=1' and test results file being created at '$(Build.SourcesDirectory)/ui/target/cucumber-xml-reports/cucumber.xml'
Run the pipeline and inspect the results
The build is green, even if there are failed tests. When looking at the results in the logs, I could see:
[WARNING] ForkStarter IOException: Element name cannot be empty
Element name cannot be empty. See the dump file /home/vsts/work/1/s/ui/target/surefire-reports/2023-03-20T06-39-40_033-jvmRun1.dumpstream
See warning
##[warning]Failed to read xxxxxx/ui/target/cucumber-xml-reports/cucumber.xml. Error : Unexpected end of file while parsing Name has occurred. Line 44, position 10..

This defect is reproducing on my local machine as well:

Have a feature file with tag @test(with 4 scenarios, 2 of them need to fail in order to be rerun again and the cucumber report to be generated with the new results ) and a TestRunner class
@suite
@IncludeEngines("cucumber")
@SelectDirectories("src/test/java/features")
@ConfigurationParameter(key = GLUE_PROPERTY_NAME, value = "steps")
@SuiteDisplayName("UI tests")
@IncludeTags("Test")
public class RunCucumberTest {

}
2. Run in maven terminal the command: mvn test -Dcucumber.filter.tags="@test" -Dsurefire.rerunFailingTestsCount=1
3. Open the cucumber.xml, the file contains only the first test which failed, and after the tag is closed, another tag for test case is not fully open, showing like this <testcase

📚 Any additional context?

From what I could see, this problem appears only when tests fail.
The content of the cucumber.xml file generated after running the tests on my local machine looks like this

<testsuite name="Cucumber" time="11.255" tests="2" skipped="0" failures="2" errors="0">
<testcase classname="Test" name="xxx" time="4.858">
<failure type="org.opentest4j.AssertionFailedError" message="Number of connections is not correct ==&gt; expected: &lt;20&gt; but was: &lt;0&gt;">
<![CDATA[org.opentest4j.AssertionFailedError: Number of connections is not correct ==> expected: <20> but was: <0>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
	at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:575)
	at steps
	at ✽.xxx(file:///Users/xxx/ui/src/test/java/features/Test.feature:65)
]]>
</failure>
<system-out><![CDATA[
Given xxxx..passed
And xxx......passed
And xxx......passed
And xxx......failed
And xx.......skipped
Then xx......skipped
]]></system-out>
</testcase>
<test</testcase>
<testcasecase

The text was updated successfully, but these errors were encountered:

mpkorstanje · 2023-03-20T12:03:57Z

Do you have the full stacktrace for:

[WARNING] ForkStarter IOException: Element name cannot be empty
Element name cannot be empty. See the dump file /home/vsts/work/1/s/ui/target/surefire-reports/2023-03-20T06-39-40_033-jvmRun1.dumpstream

It should be in the dumpstream file.

carmenloghin · 2023-03-21T07:00:16Z

There is no dumpstream file in the /home/vsts/work/1/s/ui/target/surefire-reports/ folder, only the TEST-RunCucumberTest.xml file.

mpkorstanje · 2023-03-21T12:03:24Z

I suppose you didn't configure your pipeline to capture it. Is the dump file also absent if you reproduce the problem locally? It would help a lot because it would provide the exact line that makes the mistake.

mpkorstanje · 2023-03-21T21:54:41Z

Okay. I think I found it.

You most likely have a scenario or feature without a name. For example:

Feature: 
  Scenario:
    Given a calculator I just turned on
    When I add 4 and 7
    Then the result is 2

As a work around you can give your scenarios and features a name.

mpkorstanje · 2023-03-21T22:38:14Z

Nope. That's not actually it.

Weird thing is, I can get a stack trace, but it's nothing like yours:

java.lang.NullPointerException: Cannot invoke "Object.hashCode()" because "key" is null
        at java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
        at io.cucumber.junitxmlformatter.XmlReportData.getFeatureName(XmlReportData.java:207)
        at io.cucumber.junitxmlformatter.XmlReportWriter.writeTestCaseAttributes(XmlReportWriter.java:86)
        at io.cucumber.junitxmlformatter.XmlReportWriter.writeTestcase(XmlReportWriter.java:77)
        at io.cucumber.junitxmlformatter.XmlReportWriter.writeTestsuite(XmlReportWriter.java:45)
        at io.cucumber.junitxmlformatter.XmlReportWriter.writeXmlReport(XmlReportWriter.java:34)
        at io.cucumber.junitxmlformatter.MessagesToJunitXmlWriter.close(MessagesToJunitXmlWriter.java:60)

And I do need two failing scenarios in a feature to trigger the problem. Otherwise nothing bad happens at all.

carmenloghin · 2023-03-22T15:59:03Z

I'm not able to see any dump files, neither on Azure nor on the local machine. What I can say is that the issue is reproducing when having at least 2 failing tests (I use 4 tests, 2 which pass and 2 which fails) and the command
mvn test -Dcucumber.filter.tags="@test" -Dsurefire.rerunFailingTestsCount=1.

My feature file looks like this:

@Test
Feature: Test

  Scenario: Successfully test
    Given that I open the page "https://github.com/"
    Then the page title is "GitHub: Let’s build from here · GitHub"

  Scenario: First unsuccessfully test
    Given that I open the page "https://github.com/"
    Then the page title is "Google"

  Scenario: Second unsuccessfully test
    Given that I open the page "https://github.com/"
    Then the page title is "Youtube"

When I check the TEST-RunCucumberTest.xml I saw, at the end of the file, the same error you've mentioned before:

<testcase name="Cucumber" classname="UI tests" time="5.974">
    <error type="java.lang.NullPointerException"><![CDATA[java.lang.NullPointerException
	at java.base/java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936)
	at io.cucumber.junitxmlformatter.XmlReportData.getFeatureName(XmlReportData.java:207)
	at io.cucumber.junitxmlformatter.XmlReportWriter.writeTestCaseAttributes(XmlReportWriter.java:86)
	at io.cucumber.junitxmlformatter.XmlReportWriter.writeTestcase(XmlReportWriter.java:77)
	at io.cucumber.junitxmlformatter.XmlReportWriter.writeTestsuite(XmlReportWriter.java:45)
	at io.cucumber.junitxmlformatter.XmlReportWriter.writeXmlReport(XmlReportWriter.java:34)
	at io.cucumber.junitxmlformatter.MessagesToJunitXmlWriter.close(MessagesToJunitXmlWriter.java:60)
	at io.cucumber.core.plugin.JUnitFormatter.write(JUnitFormatter.java:35)
	at io.cucumber.core.eventbus.AbstractEventPublisher.send(AbstractEventPublisher.java:51)
	at io.cucumber.core.eventbus.AbstractEventBus.send(AbstractEventBus.java:12)
	at io.cucumber.core.runtime.SynchronizedEventBus.send(SynchronizedEventBus.java:47)
	at io.cucumber.core.runtime.CucumberExecutionContext.emitTestRunFinished(CucumberExecutionContext.java:118)
	at io.cucumber.core.runtime.CucumberExecutionContext.finishTestRun(CucumberExecutionContext.java:98)
	at io.cucumber.junit.platform.engine.CucumberEngineExecutionContext.finishTestRun(CucumberEngineExecutionContext.java:126)
	at io.cucumber.junit.platform.engine.CucumberEngineDescriptor.cleanUp(CucumberEngineDescriptor.java:60)
	at io.cucumber.junit.platform.engine.CucumberEngineDescriptor.cleanUp(CucumberEngineDescriptor.java:10)

The error is displayed when I run the maven command from terminal as well.

The generated cucumber.xml is

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="Cucumber" time="3.606" tests="2" skipped="0" failures="2" errors="0">
<testcase classname="Test" name="Second unsuccessfully test" time="1.635">
<failure type="org.opentest4j.AssertionFailedError" message="expected: &lt;Youtube&gt; but was: &lt;GitHub: Let’s build from here · GitHub&gt;">
<![CDATA[org.opentest4j.AssertionFailedError: expected: <Youtube> but was: <GitHub: Let’s build from here · GitHub>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
	at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1141)
	at steps.TestStepDefinition.thePageTitleIs(TestStepDefinition.java:85)
	at ✽.the page title is "Youtube"(file:///Users/xxxx/ui/src/test/java/features/Test.feature:14)
]]>
</failure>
<system-out><![CDATA[
Given that I open the page "https://github.com/"............................passed
Then the page title is "Youtube"............................................failed
]]></system-out>
</testcase>
<testcase

Notice the testcase tag is not closed.

When I run with the following command: mvn test -Dcucumber.filter.tags="@test" then the error is no longer displayed and the cucumber.xml report looks like this:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="Cucumber" time="11.511" tests="3" skipped="0" failures="2" errors="0">
<testcase classname="Test" name="Successfully test" time="7.783">
<system-out><![CDATA[
Given that I open the page "https://github.com/"............................passed
Then the page title is "GitHub: Let’s build from here · GitHub".............passed
]]></system-out>
</testcase>
<testcase classname="Test" name="First unsuccessfully test" time="1.728">
<failure type="org.opentest4j.AssertionFailedError" message="expected: &lt;Google&gt; but was: &lt;GitHub: Let’s build from here · GitHub&gt;">
<![CDATA[org.opentest4j.AssertionFailedError: expected: <Google> but was: <GitHub: Let’s build from here · GitHub>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
	at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1141)
	at steps.TestStepDefinition.thePageTitleIs(TestStepDefinition.java:85)
	at ✽.the page title is "Google"(file:///Users/xxxx/ui/src/test/java/features/Test.feature:10)
]]>
</failure>
<system-out><![CDATA[
Given that I open the page "https://github.com/"............................passed
Then the page title is "Google".............................................failed
]]></system-out>
</testcase>
<testcase classname="Test" name="Second unsuccessfully test" time="1.736">
<failure type="org.opentest4j.AssertionFailedError" message="expected: &lt;Youtube&gt; but was: &lt;GitHub: Let’s build from here · GitHub&gt;">
<![CDATA[org.opentest4j.AssertionFailedError: expected: <Youtube> but was: <GitHub: Let’s build from here · GitHub>
	at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
	at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
	at org.junit.jupiter.api.AssertEquals.failNotEqual(AssertEquals.java:197)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)
	at org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)
	at org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1141)
	at steps.TestStepDefinition.thePageTitleIs(TestStepDefinition.java:85)
	at ✽.the page title is "Youtube"(file:///Users/xxxxx/ui/src/test/java/features/Test.feature:14)
]]>
</failure>
<system-out><![CDATA[
Given that I open the page "https://github.com/"............................passed
Then the page title is "Youtube"............................................failed
]]></system-out>
</testcase>
</testsuite>

mpkorstanje · 2023-03-22T16:39:22Z

Cheers. Then I think I've got it.

The JUnit Platform allows Scenarios and Examples to be selected by their unique id. This id is stable between test executions and can be used to rerun failing Scenarios and Examples. For practical reasons each unique id is processed individually[+] and results in a feature file being parsed. The parsed feature is then compiled into pickles. Both are mapped to a hierarchy of JUnit 5 test descriptors. These are then merged. The merge process will discard any duplicate nodes. Each time a feature file is parsed the parser will assign unique identifiers to all ast nodes and pickles. As a result the merged hierarchy of junit test descriptors contained pickles that belonged to a different feature file. This poses a problem for tools such as the junit-xml-formatter that depend on the identifiers in pickles and features to stitch everything back together. By caching the parsed feature files we ensure that within a single execution the internal identifiers do not change. + : It would actually matter little if we did process all unique ids at once, the problem would persist when uri selectors are used, either alone or in combination with unique id selectors. Fixes: #2709

mpkorstanje mentioned this issue Mar 21, 2023

[JUnit Platform Engine] Cache parsed features #2711

Merged

7 tasks

mpkorstanje closed this as completed in #2711 Mar 23, 2023

dunja132015 mentioned this issue Aug 7, 2024

Issue with test execution report when multiple scenario outlines and surefirererun is set #2911

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Corrupted Cucumber.xml when using surefire.rerunFailingTestsCount parameter #2709

Corrupted Cucumber.xml when using surefire.rerunFailingTestsCount parameter #2709

carmenloghin commented Mar 20, 2023 •

edited by mpkorstanje

Loading

mpkorstanje commented Mar 20, 2023

carmenloghin commented Mar 21, 2023

mpkorstanje commented Mar 21, 2023

mpkorstanje commented Mar 21, 2023 •

edited

Loading

mpkorstanje commented Mar 21, 2023 •

edited

Loading

carmenloghin commented Mar 22, 2023

mpkorstanje commented Mar 22, 2023

Corrupted Cucumber.xml when using surefire.rerunFailingTestsCount parameter #2709

Corrupted Cucumber.xml when using surefire.rerunFailingTestsCount parameter #2709

Comments

carmenloghin commented Mar 20, 2023 • edited by mpkorstanje Loading

👓 What did you see?

✅ What did you expect to see?

📦 Which tool/library version are you using?

🔬 How could we reproduce it?

📚 Any additional context?

mpkorstanje commented Mar 20, 2023

carmenloghin commented Mar 21, 2023

mpkorstanje commented Mar 21, 2023

mpkorstanje commented Mar 21, 2023 • edited Loading

mpkorstanje commented Mar 21, 2023 • edited Loading

carmenloghin commented Mar 22, 2023

mpkorstanje commented Mar 22, 2023

carmenloghin commented Mar 20, 2023 •

edited by mpkorstanje

Loading

mpkorstanje commented Mar 21, 2023 •

edited

Loading

mpkorstanje commented Mar 21, 2023 •

edited

Loading