[Automatic PR] Generate release notes (#1115)

Co-authored-by: piotradamczyk5 <[email protected]>

release_notes.md

@@ -1,3 +1,15 @@
+## v20.09.2
+### Features
+- [#1111](https://github.com/Flank/flank/pull/1111) Check if gcs path exist before run ([piotradamczyk5](https://github.com/piotradamczyk5))
+- [#1110](https://github.com/Flank/flank/pull/1110) Finialize the slack sending integration with actions ([Sloox](https://github.com/Sloox))
+- [#1108](https://github.com/Flank/flank/pull/1108) Support for robo tests without robo script ([pawelpasterz](https://github.com/pawelpasterz))
+- [#1104](https://github.com/Flank/flank/pull/1104) Send slack message for releases ([Sloox](https://github.com/Sloox))
+- [#1097](https://github.com/Flank/flank/pull/1097) Print the matrices web link at the end of a run ([adamfilipow92](https://github.com/adamfilipow92))
+### Bug Fixes
+- [#1107](https://github.com/Flank/flank/pull/1107) Running iOS test on Linux ([piotradamczyk5](https://github.com/piotradamczyk5))
+- [#1059](https://github.com/Flank/flank/pull/1059) Shards and Tests count do not match ([pawelpasterz](https://github.com/pawelpasterz), [adamfilipow92](https://github.com/adamfilipow92))
+- [#1067](https://github.com/Flank/flank/pull/1067) Matrix path not found in json ([adamfilipow92](https://github.com/adamfilipow92), [piotradamczyk5](https://github.com/piotradamczyk5))
+
 ## v20.09.1
 ### Bug Fixes
 - [#1100](https://github.com/Flank/flank/pull/1100) Fix gcs files validation ([pawelpasterz](https://github.com/pawelpasterz))

test_runner/docs/ascii/flank.jar_-android-run.adoc

@@ -87,6 +87,9 @@ Configuration is read from flank.yml
 *--device*=_<String=String>_[,_<String=String>_...]::
   A list of DIMENSION=VALUE pairs which specify a target device to test against. This flag may be repeated to specify multiple devices. The four device dimensions are: model, version, locale, and orientation. If any dimensions are omitted, they will use a default value. Omitting all of the preceding dimension-related flags will run tests against a single device using defaults for all four device dimensions.
 
+*--async*::
+  Invoke a test asynchronously without waiting for test results.
+
 *--results-bucket*=_<resultsBucket>_::
   The name of a Google Cloud Storage bucket where raw test results will be stored (default: "test-lab-<random-UUID>"). Note that the bucket must be owned by a billing-enabled project, and that using a non-default bucket will result in billing charges for the storage used.
 
@@ -102,9 +105,6 @@ Configuration is read from flank.yml
 *--timeout*=_<timeout>_::
   The max time this test execution can run before it is cancelled (default: 15m). It does not include any time necessary to prepare and clean up the target device. The maximum possible testing time is 30m on physical devices and 60m on virtual devices. The TIMEOUT units can be h, m, or s. If no unit is given, seconds are assumed. 
 
-*--async*::
-  Invoke a test asynchronously without waiting for test results.
-
 *--client-details*=_<String=String>_[,_<String=String>_...]::
   Comma-separated, KEY=VALUE map of additional details to attach to the test matrix.Arbitrary KEY=VALUE pairs may be attached to a test matrix to provide additional context about the tests being run.When consuming the test results, such as in Cloud Functions or a CI system,these details can add additional context such as a link to the corresponding pull request.
 
@@ -117,12 +117,12 @@ Configuration is read from flank.yml
 *--num-flaky-test-attempts*=_<flakyTestAttempts>_::
   The number of times a TestExecution should be re-attempted if one or more of its test cases fail for any reason. The maximum number of reruns allowed is 10. Default is 0, which implies no reruns.
 
-*--shard-time*=_<shardTime>_::
-  The max amount of seconds each shard should run.
-
 *--max-test-shards*=_<maxTestShards>_::
   The amount of matrices to split the tests across.
 
+*--shard-time*=_<shardTime>_::
+  The max amount of seconds each shard should run.
+
 *--num-test-runs*=_<repeatTests>_::
   The amount of times to run the test executions.
 
@@ -171,6 +171,9 @@ Configuration is read from flank.yml
 *--use-average-test-time-for-new-tests*::
   Enable using average time from previous tests duration when using SmartShard and tests did not run before.
 
+*--app*=_<app>_::
+  The path to the application binary file. The path may be in the local filesystem or in Google Cloud Storage using gs:// notation.
+
 *--test*=_<test>_::
   The path to the binary file containing instrumentation tests. The given path may be in the local filesystem or in Google Cloud Storage using a URL beginning with gs://.
 
@@ -183,9 +186,6 @@ Configuration is read from flank.yml
 *--no-auto-google-login*::
   Google account not logged in (default behavior). Use --auto-google-login to enable
 
-*--app*=_<app>_::
-  The path to the application binary file. The path may be in the local filesystem or in Google Cloud Storage using gs:// notation.
-
 *--use-orchestrator*::
   Whether each test runs in its own Instrumentation instance with the Android Test Orchestrator (default: Orchestrator is used. To disable, use --no-use-orchestrator). Orchestrator is only compatible with AndroidJUnitRunner v1.0 or higher. See https://developer.android.com/training/testing/junit-runner.html#using-android-test-orchestrator for more information about Android Test Orchestrator.
 
@@ -236,12 +236,12 @@ You can guide the Robo test to perform specific actions by recording a Robo Scri
 +
 Learn more at https://firebase.google.com/docs/test-lab/robo-ux-test#scripting. 
 
-*--legacy-junit-result*::
-  Fallback for legacy xml junit results parsing.
-
 *--additional-app-test-apks*=_<String=String>_[,_<String=String>_...]::
   A list of app & test apks to include in the run. Useful for running multiple module tests within a single Flank run.
 
+*--legacy-junit-result*::
+  Fallback for legacy xml junit results parsing.
+
 *--dump-shards*::
   Measures test shards from given test apks and writes them into android_shards.json file instead of executing.
 

test_runner/docs/ascii/flank.jar_-firebase-test-android-run.adoc

@@ -99,6 +99,9 @@ Configuration is read from flank.yml
 *--device*=_<String=String>_[,_<String=String>_...]::
   A list of DIMENSION=VALUE pairs which specify a target device to test against. This flag may be repeated to specify multiple devices. The four device dimensions are: model, version, locale, and orientation. If any dimensions are omitted, they will use a default value. Omitting all of the preceding dimension-related flags will run tests against a single device using defaults for all four device dimensions.
 
+*--async*::
+  Invoke a test asynchronously without waiting for test results.
+
 *--results-bucket*=_<resultsBucket>_::
   The name of a Google Cloud Storage bucket where raw test results will be stored (default: "test-lab-<random-UUID>"). Note that the bucket must be owned by a billing-enabled project, and that using a non-default bucket will result in billing charges for the storage used.
 
@@ -114,9 +117,6 @@ Configuration is read from flank.yml
 *--timeout*=_<timeout>_::
   The max time this test execution can run before it is cancelled (default: 15m). It does not include any time necessary to prepare and clean up the target device. The maximum possible testing time is 30m on physical devices and 60m on virtual devices. The TIMEOUT units can be h, m, or s. If no unit is given, seconds are assumed. 
 
-*--async*::
-  Invoke a test asynchronously without waiting for test results.
-
 *--client-details*=_<String=String>_[,_<String=String>_...]::
   Comma-separated, KEY=VALUE map of additional details to attach to the test matrix.Arbitrary KEY=VALUE pairs may be attached to a test matrix to provide additional context about the tests being run.When consuming the test results, such as in Cloud Functions or a CI system,these details can add additional context such as a link to the corresponding pull request.
 
@@ -129,12 +129,12 @@ Configuration is read from flank.yml
 *--num-flaky-test-attempts*=_<flakyTestAttempts>_::
   The number of times a TestExecution should be re-attempted if one or more of its test cases fail for any reason. The maximum number of reruns allowed is 10. Default is 0, which implies no reruns.
 
-*--shard-time*=_<shardTime>_::
-  The max amount of seconds each shard should run.
-
 *--max-test-shards*=_<maxTestShards>_::
   The amount of matrices to split the tests across.
 
+*--shard-time*=_<shardTime>_::
+  The max amount of seconds each shard should run.
+
 *--num-test-runs*=_<repeatTests>_::
   The amount of times to run the test executions.
 
@@ -183,6 +183,9 @@ Configuration is read from flank.yml
 *--use-average-test-time-for-new-tests*::
   Enable using average time from previous tests duration when using SmartShard and tests did not run before.
 
+*--app*=_<app>_::
+  The path to the application binary file. The path may be in the local filesystem or in Google Cloud Storage using gs:// notation.
+
 *--test*=_<test>_::
   The path to the binary file containing instrumentation tests. The given path may be in the local filesystem or in Google Cloud Storage using a URL beginning with gs://.
 
@@ -195,9 +198,6 @@ Configuration is read from flank.yml
 *--no-auto-google-login*::
   Google account not logged in (default behavior). Use --auto-google-login to enable
 
-*--app*=_<app>_::
-  The path to the application binary file. The path may be in the local filesystem or in Google Cloud Storage using gs:// notation.
-
 *--use-orchestrator*::
   Whether each test runs in its own Instrumentation instance with the Android Test Orchestrator (default: Orchestrator is used. To disable, use --no-use-orchestrator). Orchestrator is only compatible with AndroidJUnitRunner v1.0 or higher. See https://developer.android.com/training/testing/junit-runner.html#using-android-test-orchestrator for more information about Android Test Orchestrator.
 
@@ -248,12 +248,12 @@ You can guide the Robo test to perform specific actions by recording a Robo Scri
 +
 Learn more at https://firebase.google.com/docs/test-lab/robo-ux-test#scripting. 
 
-*--legacy-junit-result*::
-  Fallback for legacy xml junit results parsing.
-
 *--additional-app-test-apks*=_<String=String>_[,_<String=String>_...]::
   A list of app & test apks to include in the run. Useful for running multiple module tests within a single Flank run.
 
+*--legacy-junit-result*::
+  Fallback for legacy xml junit results parsing.
+
 *--dump-shards*::
   Measures test shards from given test apks and writes them into android_shards.json file instead of executing.
 

test_runner/docs/ascii/flank.jar_-firebase-test-ios-run.adoc

@@ -78,6 +78,9 @@ Configuration is read from flank.yml
 *--device*=_<String=String>_[,_<String=String>_...]::
   A list of DIMENSION=VALUE pairs which specify a target device to test against. This flag may be repeated to specify multiple devices. The four device dimensions are: model, version, locale, and orientation. If any dimensions are omitted, they will use a default value. Omitting all of the preceding dimension-related flags will run tests against a single device using defaults for all four device dimensions.
 
+*--async*::
+  Invoke a test asynchronously without waiting for test results.
+
 *--results-bucket*=_<resultsBucket>_::
   The name of a Google Cloud Storage bucket where raw test results will be stored (default: "test-lab-<random-UUID>"). Note that the bucket must be owned by a billing-enabled project, and that using a non-default bucket will result in billing charges for the storage used.
 
@@ -93,9 +96,6 @@ Configuration is read from flank.yml
 *--timeout*=_<timeout>_::
   The max time this test execution can run before it is cancelled (default: 15m). It does not include any time necessary to prepare and clean up the target device. The maximum possible testing time is 30m on physical devices and 60m on virtual devices. The TIMEOUT units can be h, m, or s. If no unit is given, seconds are assumed. 
 
-*--async*::
-  Invoke a test asynchronously without waiting for test results.
-
 *--client-details*=_<String=String>_[,_<String=String>_...]::
   Comma-separated, KEY=VALUE map of additional details to attach to the test matrix.Arbitrary KEY=VALUE pairs may be attached to a test matrix to provide additional context about the tests being run.When consuming the test results, such as in Cloud Functions or a CI system,these details can add additional context such as a link to the corresponding pull request.
 
@@ -108,12 +108,12 @@ Configuration is read from flank.yml
 *--num-flaky-test-attempts*=_<flakyTestAttempts>_::
   The number of times a TestExecution should be re-attempted if one or more of its test cases fail for any reason. The maximum number of reruns allowed is 10. Default is 0, which implies no reruns.
 
-*--shard-time*=_<shardTime>_::
-  The max amount of seconds each shard should run.
-
 *--max-test-shards*=_<maxTestShards>_::
   The amount of matrices to split the tests across.
 
+*--shard-time*=_<shardTime>_::
+  The max amount of seconds each shard should run.
+
 *--num-test-runs*=_<repeatTests>_::
   The amount of times to run the test executions.
 
@@ -162,15 +162,15 @@ Configuration is read from flank.yml
 *--use-average-test-time-for-new-tests*::
   Enable using average time from previous tests duration when using SmartShard and tests did not run before.
 
+*--test*=_<test>_::
+  The path to the test package (a zip file containing the iOS app and XCTest files). The given path may be in the local filesystem or in Google Cloud Storage using a URL beginning with gs://. Note: any .xctestrun file in this zip file will be ignored if --xctestrun-file is specified.
+
 *--xctestrun-file*=_<xctestrunFile>_::
   The path to an .xctestrun file that will override any .xctestrun file contained in the --test package. Because the .xctestrun file contains environment variables along with test methods to run and/or ignore, this can be useful for customizing or sharding test suites. The given path may be in the local filesystem or in Google Cloud Storage using a URL beginning with gs://.
 
 *--xcode-version*=_<xcodeVersion>_::
   The version of Xcode that should be used to run an XCTest. Defaults to the latest Xcode version supported in Firebase Test Lab. This Xcode version must be supported by all iOS versions selected in the test matrix.
 
-*--test*=_<test>_::
-  The path to the test package (a zip file containing the iOS app and XCTest files). The given path may be in the local filesystem or in Google Cloud Storage using a URL beginning with gs://. Note: any .xctestrun file in this zip file will be ignored if --xctestrun-file is specified.
-
 *--test-targets*=_<testTargets>_[,_<testTargets>_...]::
   A list of one or more test method names to run (default: run all test targets).
 

test_runner/docs/ascii/flank.jar_-ios-run.adoc

@@ -70,6 +70,9 @@ Configuration is read from flank.yml
 *--device*=_<String=String>_[,_<String=String>_...]::
   A list of DIMENSION=VALUE pairs which specify a target device to test against. This flag may be repeated to specify multiple devices. The four device dimensions are: model, version, locale, and orientation. If any dimensions are omitted, they will use a default value. Omitting all of the preceding dimension-related flags will run tests against a single device using defaults for all four device dimensions.
 
+*--async*::
+  Invoke a test asynchronously without waiting for test results.
+
 *--results-bucket*=_<resultsBucket>_::
   The name of a Google Cloud Storage bucket where raw test results will be stored (default: "test-lab-<random-UUID>"). Note that the bucket must be owned by a billing-enabled project, and that using a non-default bucket will result in billing charges for the storage used.
 
@@ -85,9 +88,6 @@ Configuration is read from flank.yml
 *--timeout*=_<timeout>_::
   The max time this test execution can run before it is cancelled (default: 15m). It does not include any time necessary to prepare and clean up the target device. The maximum possible testing time is 30m on physical devices and 60m on virtual devices. The TIMEOUT units can be h, m, or s. If no unit is given, seconds are assumed. 
 
-*--async*::
-  Invoke a test asynchronously without waiting for test results.
-
 *--client-details*=_<String=String>_[,_<String=String>_...]::
   Comma-separated, KEY=VALUE map of additional details to attach to the test matrix.Arbitrary KEY=VALUE pairs may be attached to a test matrix to provide additional context about the tests being run.When consuming the test results, such as in Cloud Functions or a CI system,these details can add additional context such as a link to the corresponding pull request.
 
@@ -100,12 +100,12 @@ Configuration is read from flank.yml
 *--num-flaky-test-attempts*=_<flakyTestAttempts>_::
   The number of times a TestExecution should be re-attempted if one or more of its test cases fail for any reason. The maximum number of reruns allowed is 10. Default is 0, which implies no reruns.
 
-*--shard-time*=_<shardTime>_::
-  The max amount of seconds each shard should run.
-
 *--max-test-shards*=_<maxTestShards>_::
   The amount of matrices to split the tests across.
 
+*--shard-time*=_<shardTime>_::
+  The max amount of seconds each shard should run.
+
 *--num-test-runs*=_<repeatTests>_::
   The amount of times to run the test executions.
 
@@ -154,15 +154,15 @@ Configuration is read from flank.yml
 *--use-average-test-time-for-new-tests*::
   Enable using average time from previous tests duration when using SmartShard and tests did not run before.
 
+*--test*=_<test>_::
+  The path to the test package (a zip file containing the iOS app and XCTest files). The given path may be in the local filesystem or in Google Cloud Storage using a URL beginning with gs://. Note: any .xctestrun file in this zip file will be ignored if --xctestrun-file is specified.
+
 *--xctestrun-file*=_<xctestrunFile>_::
   The path to an .xctestrun file that will override any .xctestrun file contained in the --test package. Because the .xctestrun file contains environment variables along with test methods to run and/or ignore, this can be useful for customizing or sharding test suites. The given path may be in the local filesystem or in Google Cloud Storage using a URL beginning with gs://.
 
 *--xcode-version*=_<xcodeVersion>_::
   The version of Xcode that should be used to run an XCTest. Defaults to the latest Xcode version supported in Firebase Test Lab. This Xcode version must be supported by all iOS versions selected in the test matrix.
 
-*--test*=_<test>_::
-  The path to the test package (a zip file containing the iOS app and XCTest files). The given path may be in the local filesystem or in Google Cloud Storage using a URL beginning with gs://. Note: any .xctestrun file in this zip file will be ignored if --xctestrun-file is specified.
-
 *--test-targets*=_<testTargets>_[,_<testTargets>_...]::
   A list of one or more test method names to run (default: run all test targets).