Use API instead of XML for result parsing for android #666

jan-goral · 2020-03-13T17:03:51Z

Fixes #618, Fixes #545

Checklist

Documented
Unit tested
release_notes.md updated

pawelpasterz · 2020-03-13T22:01:17Z

test_runner/src/main/kotlin/ftl/reports/api/CreateTestExecutionData.kt

+            // it is making api calls under the hood
+            testExecution.createTestExecutionData()
+        }
+    }.map { deferred ->


I think we can use .awaitAll() instead, just for simplification

bootstraponline · 2020-03-13T23:45:42Z

test_runner/src/main/kotlin/ftl/reports/api/ProcessFromApi.kt

+    args: IArgs
+): JUnitTestResult = GcTestMatrix.refresh(
+    // Android uses server side sharding, it results only one matrix, so we can simply get first.
+    matrices.map.values.first().matrixId,


iOS doesn't use server side sharding right now. Is it possible to use the API for both Android & iOS?

Edit: I see that processXmlFromApi is only used for Android right now. That makes sense.

bootstraponline · 2020-03-13T23:46:27Z

test_runner/src/test/kotlin/TmpV2.kt

+import java.io.File
+import kotlin.system.exitProcess
+
+object TmpV2 {


we should probably delete TmpV2 and turn this into a proper JUnit test. What do you think?

Also probably delete the original tmp

Of Course I will do that. This is only draft of implementation that I tested manually and looks correct. I now need to update unit tests, cleanup a little, and It will be ready for review.

bootstraponline · 2020-03-14T05:55:20Z

test_runner/src/main/kotlin/ftl/reports/api/Utils.kt

+package ftl.reports.api
+
+import com.google.api.services.toolresults.model.Duration
+import java.util.*


I think we have a lint rule about no wildcard imports.

bootstraponline · 2020-03-14T20:45:46Z

test_runner/src/test/kotlin/TmpV2.kt

+        println(" - OK")
+
+        print("writing api results to file")
+        File(JUNIT_REPORT_FILE).writeText(jUnitTestResult.xmlToString())


To test this, run an Android app with multiple tests using server side sharding that has flaky tests. Use num-flaky-test-attempts set to a non-zero value such as 2. Then compare the JUnit XML file that FTL is creating vs the one we create via the API.

I was testing exactly that case. I know there is a difference between old and new JUnitReport.xml.
Old:

<?xml version='1.0' encoding='UTF-8' ?> <testsuites> <testsuite name="NexusLowRes-28-en-portrait#" tests="3" failures="0" errors="0" skipped="0" time="3.103" timestamp="2020-03-10T17:12:24" hostname="localhost"> <testcase name="test0" classname="com.example.test_app.InstrumentedTest" time="1.161"> <webLink></webLink> </testcase> <testcase name="test1" classname="com.example.test_app.InstrumentedTest" time="1.034"> <webLink></webLink> </testcase> <testcase name="test2" classname="com.example.test_app.InstrumentedTest" time="0.908"> <webLink></webLink> </testcase> </testsuite> </testsuites>

New:

<?xml version='1.0' encoding='UTF-8' ?> <testsuites> <testsuite name="NexusLowRes-28-en-portrait" tests="1" failures="0" errors="0" skipped="0" time="1.161" timestamp="2020-03-10T17:07:54.422Z"> <testcase name="test0" classname="com.example.test_app.InstrumentedTest" time="1.161"/> </testsuite> <testsuite name="NexusLowRes-28-en-portrait" tests="2" failures="0" errors="0" skipped="0" time="1.942" timestamp="2020-03-10T17:07:54.422Z"> <testcase name="test1" classname="com.example.test_app.InstrumentedTest" time="1.034"/> <testcase name="test2" classname="com.example.test_app.InstrumentedTest" time="0.908"/> </testsuite> </testsuites>

In new JUnitReport suites reflects shards. I can fix it easily, but that was first what I achieved. I guess that maybe is better to expose some info about shards in JUnitReport. If you don't like that I will fix it.

The timestamp is different yes. Old timestamp is simply timestamp of first of downloaded reports. New one is not correct I will fix it.

I need to also fix weblinks.

makes sense!

bootstraponline · 2020-03-14T22:31:26Z

When using Android Test Orchestrator, we'll want to verify the API test times match the times returned in the FTL XML file.

#557

codecov-io · 2020-03-23T05:22:58Z

Codecov Report

Merging #666 into master will decrease coverage by 0.99%.
The diff coverage is 65.95%.

@@             Coverage Diff              @@
##             master     #666      +/-   ##
============================================
- Coverage     77.10%   76.10%   -1.00%     
- Complexity      598      615      +17     
============================================
  Files            99      109      +10     
  Lines          2367     2528     +161     
  Branches        337      359      +22     
============================================
+ Hits           1825     1924      +99     
- Misses          335      381      +46     
- Partials        207      223      +16

pawelpasterz

Great job! That is one big PR with lot of work. I am really impressed.

The only thing I can complain is multiple usage of runBlocking. Personally I don't like when this is used as kind of bridge between synchronous and asynchronous code (when the first one is invoked inside a coroutine). But TBH I don't know if we have better solution right now.

Again, great job!

pawelpasterz · 2020-03-23T18:28:33Z

test_runner/src/main/kotlin/ftl/reports/api/CreateJUnitTestSuite.kt

+    return listOf(map["Model"], map["Version"], map["Locale"], map["Orientation"]).joinToString("-")
+}
+
+private fun List<JUnitTestCase>.sumTime() = this


I think this is redundant here but I guess it was left intentionally for sake of readability?

That's right, intentionally for readability. I usually use this for extension functions with expression body. Thats allow clear chain.

pawelpasterz · 2020-03-23T18:35:39Z

test_runner/src/main/kotlin/ftl/reports/api/ProcessFromApi.kt

+    .getTestExecutions()
+    .createJUnitTestResult()
+
+internal fun refreshTestMatrices(


I think we can make it private

pawelpasterz · 2020-03-23T18:35:51Z

test_runner/src/main/kotlin/ftl/reports/api/ProcessFromApi.kt

+    }.awaitAll()
+}
+
+internal fun List<TestMatrix>.getTestExecutions(): List<TestExecution> = this


I think we can make it private

pawelpasterz · 2020-03-23T19:00:02Z

test_runner/src/main/kotlin/ftl/reports/api/CreateJUnitTestCase.kt

+        skipped = if (testCase.status == "skipped") null else "absent"
+    ).apply {
+        if (errors != null || failures != null) {
+            webLink = getWebLink(toolResultsStep, testCase.testCaseId)


pawelpasterz · 2020-03-23T19:12:35Z

test_runner/src/main/kotlin/ftl/reports/api/CreateTestExecutionData.kt

+
+private fun TestExecution.createTestExecutionData(): TestExecutionData {
+    val response: ListTestCasesResponse = GcToolResults.listTestCases(toolResultsStep)
+    val step: Step = GcToolResults.getStepResult(toolResultsStep)


Both GcToolResults.listTestCases and GcToolResults.getStepResult are making api calls and seems that they don't depend on each other. Do you think we could make async call for each?

Right, I will fix that.

pawelpasterz · 2020-03-23T19:33:10Z

test_runner/src/main/kotlin/ftl/reports/api/Utils.kt

+
+fun Long.formatUtcDate() = utcDateFormat.format(this)!!
+
+var TestCase.flaky: Boolean by mutableMapProperty { false }


pawelpasterz · 2020-03-23T19:35:14Z

test_runner/src/main/kotlin/ftl/reports/api/Utils.kt

@@ -0,0 +1,34 @@
+package ftl.reports.api


We already have Utils.kt file, can we rename this one to sth like ApiUtils or somehow different? Just for sake of clarity.

Looks like you don't care about packages :). I guess adding part of package as prefix for file is redundant.

Yeah, true. Sometimes I am kind of bitchy :D ¯_(ツ)_/¯

Easy. You are doing your job. It is always better to express your opinion than to do nothing. There are no ultimate ways, everything we do is based on consensus.

pawelpasterz · 2020-03-23T19:40:53Z

test_runner/src/main/kotlin/ftl/reports/xml/JUnitXml.kt

@@ -15,7 +15,7 @@ private val xmlMapper = XmlMapper(xmlModule)
    .registerModules(KotlinModule())
    .configure(FAIL_ON_UNKNOWN_PROPERTIES, false)

-private val xmlPrettyWriter = xmlMapper.writerWithDefaultPrettyPrinter()
+internal val xmlPrettyWriter = xmlMapper.writerWithDefaultPrettyPrinter()

 private fun xmlBytes(path: Path): ByteArray {
    if (!path.toFile().exists()) RuntimeException("$path doesn't exist!")


I am totally aware it is not part of your PR (so feel free to skip this comment). I think there is throw missing

Good checkup. I can fix it in this PR, no problem.

pawelpasterz · 2020-03-23T19:45:07Z

test_runner/src/main/kotlin/ftl/reports/xml/model/JUnitTestSuite.kt

@@ -29,11 +32,11 @@ data class JUnitTestSuite(
    val timestamp: String?, // String. Android only

    @JacksonXmlProperty(isAttribute = true)
-    val hostname: String, // String.
+    val hostname: String? = null, // String.


Just for my understanding, are we ok with possible null here? Previous implementation did not predicted such value. Thanks!

I have asked @bootstraponline about that, he said #545 (comment). So I decide to change it to nullable here, because I didn't find hostname in API

Got it, thanks for explanation

pawelpasterz · 2020-03-23T19:51:40Z

test_runner/src/main/kotlin/ftl/run/platform/common/AfterRunTests.kt

-        Paths.get(args.localResultDir, FtlConstants.configFileName(args))
-    } else {
+    val configFilePath = if (args.useLocalResultDir())
+        Paths.get(args.localResultDir, FtlConstants.configFileName(args)) else


TBH I don't like it when I need to read whole line to find out there is else at the end. But it is matter of personal preferences :)

TBH I don't like if else and prefer using when but our detect config forces to use if else in cases like this. Anyway I like that kind of formatting cause of clear indentation same for two similar parts.

* Fix coroutines errors after rebase

jan-goral self-assigned this Mar 13, 2020

pawelpasterz reviewed Mar 13, 2020

View reviewed changes

bootstraponline reviewed Mar 13, 2020

View reviewed changes

bootstraponline reviewed Mar 14, 2020

View reviewed changes

jan-goral force-pushed the 545-junit-result-from-api branch 3 times, most recently from 7d905dc to 15d9f39 Compare March 23, 2020 05:18

jan-goral force-pushed the 545-junit-result-from-api branch from 80ecae9 to 25e5867 Compare March 23, 2020 09:42

jan-goral marked this pull request as ready for review March 23, 2020 11:29

bootstraponline previously approved these changes Mar 23, 2020

View reviewed changes

pawelpasterz previously approved these changes Mar 23, 2020

View reviewed changes

jan-goral dismissed stale reviews from pawelpasterz and bootstraponline via 3d5e8bf March 24, 2020 06:33

pawelpasterz self-requested a review March 24, 2020 06:35

pawelpasterz previously approved these changes Mar 24, 2020

View reviewed changes

jan-goral dismissed pawelpasterz’s stale review via e7bd0b0 March 24, 2020 08:51

jan-goral added 10 commits March 24, 2020 09:54

Use API instead of XML for result parsing for android

762de61

CR fixes

260cd77

Fix failing tests

b3985af

Add web link to JUnitTestCase generated from api

b473fa2

* Fix coroutines errors after rebase

Fix timestamp for JUnitTestSuite generated from api

84bfcf2

Fix failed test case filtering for flaky tests

a5610f4

Add additional info about flaky tests

3536005

Add support for multiple matrices

f8c505d

Add legacy-junit-result flag

9d3b4a3

Add tests for createJUnitTestCases & createTestSuitOverviewData

24bbe8e

jan-goral added 7 commits March 24, 2020 09:59

Add test for createJUnitTestSuites

7fb7c1f

Add test for prepareForJUnitResult

33899b6

Cleanup

2c1ce19

Cleanup

5428e82

Cleanup

af78de7

CR fixes

dc48a10

Update release notes

169c7b5

jan-goral force-pushed the 545-junit-result-from-api branch from e7bd0b0 to 169c7b5 Compare March 24, 2020 08:59

pawelpasterz approved these changes Mar 24, 2020

View reviewed changes

jan-goral merged commit f114c83 into master Mar 24, 2020

jan-goral deleted the 545-junit-result-from-api branch March 24, 2020 09:36

jan-goral mentioned this pull request Apr 22, 2020

com.fasterxml.jackson.module.kotlin.MissingKotlinParameterException in v8.1.0 #735

Closed

RainNapper mentioned this pull request Jun 24, 2020

Web link missing from successful test results #867

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use API instead of XML for result parsing for android #666

Use API instead of XML for result parsing for android #666

jan-goral commented Mar 13, 2020 •

edited

Loading

pawelpasterz Mar 13, 2020

bootstraponline Mar 13, 2020 •

edited

Loading

bootstraponline Mar 13, 2020

bootstraponline Mar 13, 2020

jan-goral Mar 16, 2020

bootstraponline Mar 14, 2020

bootstraponline Mar 14, 2020

jan-goral Mar 16, 2020 •

edited

Loading

bootstraponline Mar 17, 2020

bootstraponline commented Mar 14, 2020

codecov-io commented Mar 23, 2020 •

edited

Loading

pawelpasterz left a comment •

edited

Loading

pawelpasterz Mar 23, 2020

jan-goral Mar 24, 2020 •

edited

Loading

pawelpasterz Mar 23, 2020

pawelpasterz Mar 23, 2020

pawelpasterz Mar 23, 2020

pawelpasterz Mar 23, 2020

jan-goral Mar 24, 2020

pawelpasterz Mar 23, 2020

pawelpasterz Mar 23, 2020

jan-goral Mar 24, 2020 •

edited

Loading

pawelpasterz Mar 24, 2020

jan-goral Mar 24, 2020 •

edited

Loading

pawelpasterz Mar 23, 2020

jan-goral Mar 24, 2020

pawelpasterz Mar 23, 2020

jan-goral Mar 24, 2020

pawelpasterz Mar 24, 2020

pawelpasterz Mar 23, 2020

jan-goral Mar 24, 2020 •

edited

Loading


		fun Long.formatUtcDate() = utcDateFormat.format(this)!!

		var TestCase.flaky: Boolean by mutableMapProperty { false }

Use API instead of XML for result parsing for android #666

Use API instead of XML for result parsing for android #666

Conversation

jan-goral commented Mar 13, 2020 • edited Loading

Checklist

Choose a reason for hiding this comment

bootstraponline Mar 13, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jan-goral Mar 16, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bootstraponline commented Mar 14, 2020

codecov-io commented Mar 23, 2020 • edited Loading

Codecov Report

pawelpasterz left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jan-goral Mar 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jan-goral Mar 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jan-goral Mar 24, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jan-goral Mar 24, 2020 • edited Loading

Choose a reason for hiding this comment

jan-goral commented Mar 13, 2020 •

edited

Loading

bootstraponline Mar 13, 2020 •

edited

Loading

jan-goral Mar 16, 2020 •

edited

Loading

codecov-io commented Mar 23, 2020 •

edited

Loading

pawelpasterz left a comment •

edited

Loading

jan-goral Mar 24, 2020 •

edited

Loading

jan-goral Mar 24, 2020 •

edited

Loading

jan-goral Mar 24, 2020 •

edited

Loading

jan-goral Mar 24, 2020 •

edited

Loading