flaky-test-attempts matrix outcome always failure

[andreschab90]

usecase:

• precondition: flaky test-attempt is enabled
• 1st run failed
• 2nd run is successful
  -> test suite is reported as unstable, which will be marked as failed for flank

[bootstraponline]

[andreschab90]

Ok then it's the flaky case...I just noticed it today

[bootstraponline]

Thanks for the report, I'm looking into how to best support this new outcome in Flank.

[andreschab90]

I just noticed that following behaviour:

• Re-run failed -> result is failed
• Re-run passed -> result is unstable (flaky)

[andreschab90]

I assume the easiest solution would be to parse flaky as passed as well

[bootstraponline]

yep that's what I plan to do.

[bootstraponline]

I'll hopefully have time to look at this tomorrow or early next week.

[andreschab90]

Thanks

[bootstraponline]

I tracked this down:

{
  "matrix-1hjqookdik7px": {
    "matrixId": "matrix-1hjqookdik7px",
    "state": "FINISHED",
    "gcsPath": "test-lab-jwwvtzdh5d20w-yyju9fnudnpts/2019-05-16_16-57-51.123000_SCCQ/shard_0",
    "webLink": "https://console.firebase.google.com/project/delta-essence-114723/testlab/histories/bh.ec6be5d5d906fd8e/matrices/6106938843968825941",
    "downloaded": true,
    "billableVirtualMinutes": 3,
    "billablePhysicalMinutes": 0,
    "outcome": "failure",
    "outcomeDetails": ""
  }
}

Given a JUnit report with all tests passing:

<?xml version='1.0' encoding='UTF-8' ?>
<testsuites>
  <testsuite name="NexusLowRes-28-en-portrait" tests="4" failures="0" errors="0" skipped="0" time="4.220" timestamp="2019-05-16T17:11:09" hostname="localhost">
    <testcase name="testIsFlaky2" classname="com.example.test_app.ExampleInstrumentedTest" time="1.162">
      <webLink>https://console.firebase.google.com/project/delta-essence-114723/testlab/histories/bh.ec6be5d5d906fd8e/matrices/6106938843968825941</webLink>
    </testcase>
    <testcase name="testIsFlaky3" classname="com.example.test_app.ExampleInstrumentedTest" time="1.031">
      <webLink>https://console.firebase.google.com/project/delta-essence-114723/testlab/histories/bh.ec6be5d5d906fd8e/matrices/6106938843968825941</webLink>
    </testcase>
    <testcase name="testPasses" classname="com.example.test_app.ExampleInstrumentedTest" time="0.962">
      <webLink>https://console.firebase.google.com/project/delta-essence-114723/testlab/histories/bh.ec6be5d5d906fd8e/matrices/6106938843968825941</webLink>
    </testcase>
    <testcase name="testIsFlaky" classname="com.example.test_app.ExampleInstrumentedTest" time="1.065">
      <webLink>https://console.firebase.google.com/project/delta-essence-114723/testlab/histories/bh.ec6be5d5d906fd8e/matrices/6106938843968825941</webLink>
    </testcase>
  </testsuite>
</testsuites>

Firebase sets the matrix outcome to failure. Flank then checks the matrix outcome and reports a failure. This is a FTL bug. When using flaky-test-attempts: 2 the matrix outcome should not be failure when a test passes in at least one attempt.

Attempt 1:

<?xml version='1.0' encoding='UTF-8' ?>
<testsuite name="" tests="4" failures="2" errors="0" skipped="0" time="15.737" timestamp="2019-05-16T17:07:04" hostname="localhost">
  <properties />
  <testcase name="testIsFlaky2" classname="com.example.test_app.ExampleInstrumentedTest" time="1.162" />
  <testcase name="testIsFlaky3" classname="com.example.test_app.ExampleInstrumentedTest" time="1.919">
        <failure>java.lang.AssertionError</failure>
  </testcase>
  <testcase name="testPasses" classname="com.example.test_app.ExampleInstrumentedTest" time="0.962" />
  <testcase name="testIsFlaky" classname="com.example.test_app.ExampleInstrumentedTest" time="1.707">
        <failure>java.lang.AssertionError</failure>
  </testcase>
</testsuite>

Attempt 2:

<?xml version='1.0' encoding='UTF-8' ?>
<testsuite name="" tests="4" failures="2" errors="0" skipped="0" time="16.243" timestamp="2019-05-16T17:11:09" hostname="localhost">
  <properties />
  <testcase name="testIsFlaky2" classname="com.example.test_app.ExampleInstrumentedTest" time="1.961">
        <failure>java.lang.AssertionError</failure>
  </testcase>
  <testcase name="testIsFlaky3" classname="com.example.test_app.ExampleInstrumentedTest" time="1.844">
        <failure>java.lang.AssertionError</failure>
  </testcase>
  <testcase name="testPasses" classname="com.example.test_app.ExampleInstrumentedTest" time="1.035" />
  <testcase name="testIsFlaky" classname="com.example.test_app.ExampleInstrumentedTest" time="1.065" />
</testsuite>

Attempt 3:

<?xml version='1.0' encoding='UTF-8' ?>
<testsuite name="" tests="4" failures="1" errors="0" skipped="0" time="14.08" timestamp="2019-05-16T17:11:09" hostname="localhost">
  <properties />
  <testcase name="testIsFlaky2" classname="com.example.test_app.ExampleInstrumentedTest" time="1.157" />
  <testcase name="testIsFlaky3" classname="com.example.test_app.ExampleInstrumentedTest" time="1.031" />
  <testcase name="testPasses" classname="com.example.test_app.ExampleInstrumentedTest" time="0.979" />
  <testcase name="testIsFlaky" classname="com.example.test_app.ExampleInstrumentedTest" time="1.888">
    <failure>java.lang.AssertionError</failure>
  </testcase>
</testsuite>

[bootstraponline]

I added support for the flaky outcome, however that doesn't resolve the issue with the run being marked as failure due to the FTL bug described above:

358346c#diff-0863ffce1d9b2a9c25d2c95e3d4c6022R103

I could work around this by parsing the merged XML and basing success/failure on that. The retry feature seems broken on FTL though so I'm not sure that's worth the effort? I had a similar work around implemented in the past.

#545 will solve this once the issues in the API are worked out.

[andreschab90]

thanks for the update

[bootstraponline]

#572 has been released in flank v6.2.3, that should help with this issue. The exit code when using flaky test attempts is now based on the JUnit XML instead of FTL's matrix outcome.

[andreschab90]

@bootstraponline thanks. Will try it after my vacation 😎

[andreschab90]

it works @bootstraponline

[bootstraponline]

Awesome. Thanks for confirming.