Add weblink to all test results, not just failures (#868)

Co-authored-by: Mansfield Mark <[email protected]>

release_notes.md

@@ -1,16 +1,14 @@
 ## next (unreleased)
 - [#857](https://github.com/Flank/flank/pull/857) Added multimodule setup for test app. ([piotradamczyk5](https://github.com/piotradamczyk5))
 - [#837](https://github.com/Flank/flank/pull/837) Added obfuscate option to dump shards. ([piotradamczyk5](https://github.com/piotradamczyk5))
+- [#868](https://github.com/Flank/flank/pull/868) Restored weblinks to all test results, not just failures. ([rainnapper](https://github.com/rainnapper))
 - [#828](https://github.com/Flank/flank/pull/828) Store test results in gcloud bucket. ([adamfilipow92](https://github.com/adamfilipow92))
 -
 -
--
 
 ## v20.06.2
 - [#853](https://github.com/Flank/flank/pull/853) Store @Ignore tests in the JUnit XML without sending ignored tests to FTL. ([piotradamczyk5](https://github.com/piotradamczyk5), [adamfilipow92](https://github.com/adamfilipow92))
 - [#853](https://github.com/Flank/flank/pull/858) Handle duplicated apk names. ([jan-gogo](https://github.com/jan-gogo))
--
--
 
 ## v20.06.1
 - [#840](https://github.com/Flank/flank/pull/840) Fix parametrized tests. ([jan-gogo](https://github.com/jan-gogo), [adamfilipow92](https://github.com/adamfilipow92), [pawelpasterz](https://github.com/pawelpasterz), [piotradamczyk5](https://github.com/piotradamczyk5))

test_runner/docs/ascii/flank.jar_-android-run.adoc

@@ -114,12 +114,12 @@ Configuration is read from flank.yml
 *--num-flaky-test-attempts*=_<flakyTestAttempts>_::
   The number of times a TestExecution should be re-attempted if one or more of its test cases fail for any reason. The maximum number of reruns allowed is 10. Default is 0, which implies no reruns.
 
-*--shard-time*=_<shardTime>_::
-  The max amount of seconds each shard should run.
-
 *--max-test-shards*=_<maxTestShards>_::
   The amount of matrices to split the tests across.
 
+*--shard-time*=_<shardTime>_::
+  The max amount of seconds each shard should run.
+
 *--num-test-runs*=_<repeatTests>_::
   The amount of times to run the test executions.
 
@@ -230,12 +230,12 @@ You can guide the Robo test to perform specific actions by recording a Robo Scri
 +
 Learn more at https://firebase.google.com/docs/test-lab/robo-ux-test#scripting. 
 
-*--legacy-junit-result*::
-  Fallback for legacy xml junit results parsing.
-
 *--additional-app-test-apks*=_<String=String>_[,_<String=String>_...]::
   A list of app & test apks to include in the run. Useful for running multiple module tests within a single Flank run.
 
+*--legacy-junit-result*::
+  Fallback for legacy xml junit results parsing.
+
 *--dump-shards*::
   Measures test shards from given test apks and writes them into android_shards.json file instead of executing.
 

test_runner/docs/ascii/flank.jar_-firebase-test-android-run.adoc

@@ -127,12 +127,12 @@ Configuration is read from flank.yml
 *--num-flaky-test-attempts*=_<flakyTestAttempts>_::
   The number of times a TestExecution should be re-attempted if one or more of its test cases fail for any reason. The maximum number of reruns allowed is 10. Default is 0, which implies no reruns.
 
-*--shard-time*=_<shardTime>_::
-  The max amount of seconds each shard should run.
-
 *--max-test-shards*=_<maxTestShards>_::
   The amount of matrices to split the tests across.
 
+*--shard-time*=_<shardTime>_::
+  The max amount of seconds each shard should run.
+
 *--num-test-runs*=_<repeatTests>_::
   The amount of times to run the test executions.
 
@@ -240,12 +240,12 @@ You can guide the Robo test to perform specific actions by recording a Robo Scri
 +
 Learn more at https://firebase.google.com/docs/test-lab/robo-ux-test#scripting. 
 
-*--legacy-junit-result*::
-  Fallback for legacy xml junit results parsing.
-
 *--additional-app-test-apks*=_<String=String>_[,_<String=String>_...]::
   A list of app & test apks to include in the run. Useful for running multiple module tests within a single Flank run.
 
+*--legacy-junit-result*::
+  Fallback for legacy xml junit results parsing.
+
 *--dump-shards*::
   Measures test shards from given test apks and writes them into android_shards.json file instead of executing.
 

test_runner/docs/ascii/flank.jar_-firebase-test-ios-run.adoc

@@ -106,12 +106,12 @@ Configuration is read from flank.yml
 *--num-flaky-test-attempts*=_<flakyTestAttempts>_::
   The number of times a TestExecution should be re-attempted if one or more of its test cases fail for any reason. The maximum number of reruns allowed is 10. Default is 0, which implies no reruns.
 
-*--shard-time*=_<shardTime>_::
-  The max amount of seconds each shard should run.
-
 *--max-test-shards*=_<maxTestShards>_::
   The amount of matrices to split the tests across.
 
+*--shard-time*=_<shardTime>_::
+  The max amount of seconds each shard should run.
+
 *--num-test-runs*=_<repeatTests>_::
   The amount of times to run the test executions.
 
@@ -151,6 +151,9 @@ Configuration is read from flank.yml
 *--output-style*=_<outputStyle>_::
   Output style of execution status. May be one of [verbose, multi, single]. For runs with only one test execution the default value is 'verbose', in other cases 'multi' is used as the default. The output style 'multi' is not displayed correctly on consoles which don't support ansi codes, to avoid corrupted output use `single` or `verbose`.
 
+*--test*=_<test>_::
+  The path to the test package (a zip file containing the iOS app and XCTest files). The given path may be in the local filesystem or in Google Cloud Storage using a URL beginning with gs://. Note: any .xctestrun file in this zip file will be ignored if --xctestrun-file is specified.
+
 *--xctestrun-file*=_<xctestrunFile>_::
   The path to an .xctestrun file that will override any .xctestrun file contained in the --test package. Because the .xctestrun file contains environment variables along with test methods to run and/or ignore, this can be useful for customizing or sharding test suites. The given path may be in the local filesystem or in Google Cloud Storage using a URL beginning with gs://.
 

test_runner/docs/ascii/flank.jar_-ios-run.adoc

@@ -97,12 +97,12 @@ Configuration is read from flank.yml
 *--num-flaky-test-attempts*=_<flakyTestAttempts>_::
   The number of times a TestExecution should be re-attempted if one or more of its test cases fail for any reason. The maximum number of reruns allowed is 10. Default is 0, which implies no reruns.
 
-*--shard-time*=_<shardTime>_::
-  The max amount of seconds each shard should run.
-
 *--max-test-shards*=_<maxTestShards>_::
   The amount of matrices to split the tests across.
 
+*--shard-time*=_<shardTime>_::
+  The max amount of seconds each shard should run.
+
 *--num-test-runs*=_<repeatTests>_::
   The amount of times to run the test executions.
 
@@ -142,6 +142,9 @@ Configuration is read from flank.yml
 *--output-style*=_<outputStyle>_::
   Output style of execution status. May be one of [verbose, multi, single]. For runs with only one test execution the default value is 'verbose', in other cases 'multi' is used as the default. The output style 'multi' is not displayed correctly on consoles which don't support ansi codes, to avoid corrupted output use `single` or `verbose`.
 
+*--test*=_<test>_::
+  The path to the test package (a zip file containing the iOS app and XCTest files). The given path may be in the local filesystem or in Google Cloud Storage using a URL beginning with gs://. Note: any .xctestrun file in this zip file will be ignored if --xctestrun-file is specified.
+
 *--xctestrun-file*=_<xctestrunFile>_::
   The path to an .xctestrun file that will override any .xctestrun file contained in the --test package. Because the .xctestrun file contains environment variables along with test methods to run and/or ignore, this can be useful for customizing or sharding test suites. The given path may be in the local filesystem or in Google Cloud Storage using a URL beginning with gs://.
 

test_runner/src/main/kotlin/ftl/reports/api/CreateJUnitTestCase.kt

@@ -34,9 +34,7 @@ private fun createJUnitTestCase(
         // skipped = true is represented by null. skipped = false is "absent"
         skipped = if (testCase.status == "skipped") null else "absent"
     ).apply {
-        if (errors != null || failures != null) {
-            webLink = getWebLink(toolResultsStep, testCase.testCaseId)
-        }
+        webLink = getWebLink(toolResultsStep, testCase.testCaseId)
         if (testCase.flaky) {
             flaky = true
         }

test_runner/src/test/kotlin/ftl/reports/api/CreateJUnitTestCaseKtTest.kt

@@ -49,36 +49,45 @@ class CreateJUnitTestCaseKtTest {
                 name = "test1",
                 classname = "TestClassName",
                 time = "2.200"
-            ),
+            ).apply {
+                webLink =
+                    "https://console.firebase.google.com/project/projectId/testlab/histories/historyId/matrices/executionId/executions/stepId/testcases/test1"
+            },
             JUnitTestCase(
                 name = "test2",
                 classname = "TestClassName",
                 time = "2.200",
                 skipped = null
-            ),
+            ).apply {
+                webLink =
+                    "https://console.firebase.google.com/project/projectId/testlab/histories/historyId/matrices/executionId/executions/stepId/testcases/test2"
+            },
             JUnitTestCase(
                 name = "test3",
                 classname = "TestClassName",
                 time = "2.200",
                 errors = listOf("exception")
             ).apply {
-                webLink = "https://console.firebase.google.com/project/projectId/testlab/histories/historyId/matrices/executionId/executions/stepId/testcases/test3"
+                webLink =
+                    "https://console.firebase.google.com/project/projectId/testlab/histories/historyId/matrices/executionId/executions/stepId/testcases/test3"
             },
             JUnitTestCase(
                 name = "test4",
                 classname = "TestClassName",
                 time = "2.200",
                 failures = listOf("exception")
             ).apply {
-                webLink = "https://console.firebase.google.com/project/projectId/testlab/histories/historyId/matrices/executionId/executions/stepId/testcases/test4"
+                webLink =
+                    "https://console.firebase.google.com/project/projectId/testlab/histories/historyId/matrices/executionId/executions/stepId/testcases/test4"
             },
             JUnitTestCase(
                 name = "test5",
                 classname = "TestClassName",
                 time = "2.200",
                 failures = listOf("exception")
             ).apply {
-                webLink = "https://console.firebase.google.com/project/projectId/testlab/histories/historyId/matrices/executionId/executions/stepId/testcases/test5"
+                webLink =
+                    "https://console.firebase.google.com/project/projectId/testlab/histories/historyId/matrices/executionId/executions/stepId/testcases/test5"
                 flaky = true
             }
         )