Flank snapshot smart result report differs from current 8.1.0 breaking backward compatibilty

[runningcode]

The current 8.1.0 smart result reports stores information in the gcs bucket with a hostname.
Example:
<testsuite name="Nexus5X-23-en-portrait#" tests="50" failures="0" errors="0" skipped="0" time="1388.586" timestamp="2020-04-28T20:38:55" hostname="localhost">
The current flank snapshot release does not store information in the gcs bucket with a hostname.
Example:
<testsuite name="Nexus5X-23-en-portrait" tests="8" failures="0" flakes="0" errors="0" skipped="0" time="150.446" timestamp="2020-04-25T09:45:37">

When testing the snapshot and then reverting back to flank 8.1.0, flank 8.1.0 will fail to parse the information stored in the bucket:

com.fasterxml.jackson.module.kotlin.MissingKotlinParameterException: Instantiation of [simple type, class ftl.reports.xml.model.JUnitTestSuite] value failed for JSON property hostname due to missing (therefore NULL) value for creator parameter hostname which is a non-nullable type
 at [Source: (byte[])"<?xml version='1.0' encoding='UTF-8' ?>
<testsuites>
  <testsuite name="Nexus5-23-en-portrait" tests="1" failures="0" flakes="0" errors="0" skipped="0" time="2.594" timestamp="2020-04-23T20:40:34">
    <testcase name="seeView" classname="com.osacky.flank.gradle.sample.ExampleInstrumentedTest" time="2.594"/>
  </testsuite>
  <testsuite name="Pixel2-26-en-portrait" tests="1" failures="0" flakes="0" errors="0" skipped="0" time="2.712" timestamp="2020-04-23T20:40:14">
    <testcase name="seeView" cl"[truncated 109 bytes]; line: 5, column: 3] (through reference chain: ftl.reports.xml.model.JUnitTestResult["testsuite"]->java.util.ArrayList[0]->ftl.reports.xml.model.JUnitTestSuite["hostname"])

This poses a problem for teams testing out the snapshot because it will break all the other builds in the pipeline which are using the same gcs results bucket.

[jan-goral]

Hi @runningcode, thanks for reporting issue. You are right, API parsing is not backward compatible with legacy XML parsing and was not designed to be. That's because of structural changes and some improvements. But also is not possible to obtain hostname value from api, which in XML parsing probably always is localhost.

In description of legacy-junit-result option is mentioned that can generate slightly different output format but maybe it's insufficient info.

Flank provides two ways for parsing junit xml results.
New way uses google api instead of merging xml files, but can generate slightly different output format.
This flag allows fallback for legacy xml junit results parsing
Currently available for android, iOS still uses only legacy way.
legacy-junit-result: false

Simply, currently is not possible to use API generated report to calculate shards for XML generated report. It possible to adjust implementation to be backward compatible, but I'm not sure If we really want that. @bootstraponline?

[bootstraponline]

I think if we always add hostname="localhost" to match FTL, then the old Flank will be happy because it sees the hostname. The incompatibility is due to us removing hostname which is a required field in the old Flank.

This seems like a small change to make. The hostname value is static and not fetched from FTL.

[jan-goral]

OK, I can fix that tomorrow easly it but I can't promise that there will be no other problems. We will see.