chore: release notes for v21.01.0 (#1442)

docs/ascii/flank.jar_-android-run.adoc

@@ -14,7 +14,7 @@ flank.jar
 *flank.jar
  android run* [*-h*] [*--async*] [*--auto-google-login*]
                        [*--disable-results-upload*] [*--disable-sharding*] [*--dry*]
-                       [*--dump-shards*] [*--full-junit-result*]
+                       [*--dump-shards*] [*--fail-fast*] [*--full-junit-result*]
                        [*--ignore-failed-tests*] [*--keep-file-path*]
                        [*--legacy-junit-result*] [*--no-auto-google-login*]
                        [*--no-performance-metrics*] [*--no-record-video*]
@@ -135,6 +135,9 @@ Configuration is read from flank.yml
 *--scenario-numbers*=_<scenarioNumbers>_[,_<scenarioNumbers>_...]::
   A list of game-loop scenario numbers which will be run as part of the test (default: all scenarios). A maximum of 1024 scenarios may be specified in one test matrix, but the maximum number may also be limited by the overall test --timeout setting.
 
+*--fail-fast*::
+  If true, only a single attempt at most will be made to run each execution/shard in the matrix. Flaky test attempts are not affected. Normally, 2 or more attempts are made if a potential infrastructure issue is detected. This feature is for latency sensitive workloads.
+
 *--type*=_<type>_::
   The type of test to run. TYPE must be one of: instrumentation, robo, xctest, game-loop.
 

docs/ascii/flank.jar_-firebase-test-android-run.adoc

@@ -15,7 +15,8 @@ flank.jar
  firebase test android run* [*-h*] [*--async*] [*--auto-google-login*]
                                      [*--disable-results-upload*]
                                      [*--disable-sharding*] [*--dry*]
-                                     [*--dump-shards*] [*--full-junit-result*]
+                                     [*--dump-shards*] [*--fail-fast*]
+                                     [*--full-junit-result*]
                                      [*--ignore-failed-tests*] [*--keep-file-path*]
                                      [*--legacy-junit-result*]
                                      [*--no-auto-google-login*]
@@ -148,6 +149,9 @@ Configuration is read from flank.yml
 *--scenario-numbers*=_<scenarioNumbers>_[,_<scenarioNumbers>_...]::
   A list of game-loop scenario numbers which will be run as part of the test (default: all scenarios). A maximum of 1024 scenarios may be specified in one test matrix, but the maximum number may also be limited by the overall test --timeout setting.
 
+*--fail-fast*::
+  If true, only a single attempt at most will be made to run each execution/shard in the matrix. Flaky test attempts are not affected. Normally, 2 or more attempts are made if a potential infrastructure issue is detected. This feature is for latency sensitive workloads.
+
 *--type*=_<type>_::
   The type of test to run. TYPE must be one of: instrumentation, robo, xctest, game-loop.
 

docs/ascii/flank.jar_-firebase-test-ios-run.adoc

@@ -14,9 +14,10 @@ flank.jar
 *flank.jar
  firebase test ios run* [*-h*] [*--async*] [*--disable-results-upload*]
                                  [*--disable-sharding*] [*--dry*] [*--dump-shards*]
-                                 [*--full-junit-result*] [*--ignore-failed-tests*]
-                                 [*--keep-file-path*] [*--no-record-video*]
-                                 [*--obfuscate*] [*--record-video*]
+                                 [*--fail-fast*] [*--full-junit-result*]
+                                 [*--ignore-failed-tests*] [*--keep-file-path*]
+                                 [*--no-record-video*] [*--obfuscate*]
+                                 [*--record-video*]
                                  [*--smart-flank-disable-upload*]
                                  [*--test-special-entitlements*]
                                  [*--use-average-test-time-for-new-tests*]
@@ -127,6 +128,9 @@ Configuration is read from flank.yml
 *--scenario-numbers*=_<scenarioNumbers>_[,_<scenarioNumbers>_...]::
   A list of game-loop scenario numbers which will be run as part of the test (default: all scenarios). A maximum of 1024 scenarios may be specified in one test matrix, but the maximum number may also be limited by the overall test --timeout setting.
 
+*--fail-fast*::
+  If true, only a single attempt at most will be made to run each execution/shard in the matrix. Flaky test attempts are not affected. Normally, 2 or more attempts are made if a potential infrastructure issue is detected. This feature is for latency sensitive workloads.
+
 *--type*=_<type>_::
   The type of test to run. TYPE must be one of: instrumentation, robo, xctest, game-loop.
 
@@ -184,6 +188,12 @@ Configuration is read from flank.yml
 *--use-average-test-time-for-new-tests*::
   Enable using average time from previous tests duration when using SmartShard and tests did not run before.
 
+*--app*=_<app>_::
+  The path to the application archive (.ipa file) for game-loop testing. The path may be in the local filesystem or in Google Cloud Storage using gs:// notation. This flag is only valid when --type=game-loop is also set
+
+*--test*=_<test>_::
+  The path to the test package (a zip file containing the iOS app and XCTest files). The given path may be in the local filesystem or in Google Cloud Storage using a URL beginning with gs://. Note: any .xctestrun file in this zip file will be ignored if --xctestrun-file is specified.
+
 *--xctestrun-file*=_<xctestrunFile>_::
   The path to an .xctestrun file that will override any .xctestrun file contained in the --test package. Because the .xctestrun file contains environment variables along with test methods to run and/or ignore, this can be useful for customizing or sharding test suites. The given path may be in the local filesystem or in Google Cloud Storage using a URL beginning with gs://.
 
@@ -198,12 +208,6 @@ Configuration is read from flank.yml
 +
 Note: Because this changes the app's identifier, make sure none of the resources in your zip file contain direct references to the test app's bundle id.
 
-*--app*=_<app>_::
-  The path to the application archive (.ipa file) for game-loop testing. The path may be in the local filesystem or in Google Cloud Storage using gs:// notation. This flag is only valid when --type=game-loop is also set
-
-*--test*=_<test>_::
-  The path to the test package (a zip file containing the iOS app and XCTest files). The given path may be in the local filesystem or in Google Cloud Storage using a URL beginning with gs://. Note: any .xctestrun file in this zip file will be ignored if --xctestrun-file is specified.
-
 *--test-targets*=_<testTargets>_[,_<testTargets>_...]::
   A list of one or more test method names to run (default: run all test targets).
 

docs/ascii/flank.jar_-ios-run.adoc

@@ -13,7 +13,7 @@ flank.jar
 
 *flank.jar
  ios run* [*-h*] [*--async*] [*--disable-results-upload*]
-                   [*--disable-sharding*] [*--dry*] [*--dump-shards*]
+                   [*--disable-sharding*] [*--dry*] [*--dump-shards*] [*--fail-fast*]
                    [*--full-junit-result*] [*--ignore-failed-tests*]
                    [*--keep-file-path*] [*--no-record-video*] [*--obfuscate*]
                    [*--record-video*] [*--smart-flank-disable-upload*]
@@ -117,6 +117,9 @@ Configuration is read from flank.yml
 *--scenario-numbers*=_<scenarioNumbers>_[,_<scenarioNumbers>_...]::
   A list of game-loop scenario numbers which will be run as part of the test (default: all scenarios). A maximum of 1024 scenarios may be specified in one test matrix, but the maximum number may also be limited by the overall test --timeout setting.
 
+*--fail-fast*::
+  If true, only a single attempt at most will be made to run each execution/shard in the matrix. Flaky test attempts are not affected. Normally, 2 or more attempts are made if a potential infrastructure issue is detected. This feature is for latency sensitive workloads.
+
 *--type*=_<type>_::
   The type of test to run. TYPE must be one of: instrumentation, robo, xctest, game-loop.
 
@@ -174,6 +177,12 @@ Configuration is read from flank.yml
 *--use-average-test-time-for-new-tests*::
   Enable using average time from previous tests duration when using SmartShard and tests did not run before.
 
+*--app*=_<app>_::
+  The path to the application archive (.ipa file) for game-loop testing. The path may be in the local filesystem or in Google Cloud Storage using gs:// notation. This flag is only valid when --type=game-loop is also set
+
+*--test*=_<test>_::
+  The path to the test package (a zip file containing the iOS app and XCTest files). The given path may be in the local filesystem or in Google Cloud Storage using a URL beginning with gs://. Note: any .xctestrun file in this zip file will be ignored if --xctestrun-file is specified.
+
 *--xctestrun-file*=_<xctestrunFile>_::
   The path to an .xctestrun file that will override any .xctestrun file contained in the --test package. Because the .xctestrun file contains environment variables along with test methods to run and/or ignore, this can be useful for customizing or sharding test suites. The given path may be in the local filesystem or in Google Cloud Storage using a URL beginning with gs://.
 
@@ -188,12 +197,6 @@ Configuration is read from flank.yml
 +
 Note: Because this changes the app's identifier, make sure none of the resources in your zip file contain direct references to the test app's bundle id.
 
-*--app*=_<app>_::
-  The path to the application archive (.ipa file) for game-loop testing. The path may be in the local filesystem or in Google Cloud Storage using gs:// notation. This flag is only valid when --type=game-loop is also set
-
-*--test*=_<test>_::
-  The path to the test package (a zip file containing the iOS app and XCTest files). The given path may be in the local filesystem or in Google Cloud Storage using a URL beginning with gs://. Note: any .xctestrun file in this zip file will be ignored if --xctestrun-file is specified.
-
 *--test-targets*=_<testTargets>_[,_<testTargets>_...]::
   A list of one or more test method names to run (default: run all test targets).
 

flank-scripts/build.gradle.kts

@@ -28,7 +28,7 @@ shadowJar.apply {
     }
 }
 // <breaking change>.<feature added>.<fix/minor change>
-version = "1.3.0"
+version = "1.3.1"
 group = "com.github.flank"
 
 application {

flank-scripts/src/main/kotlin/flank/scripts/ci/nexttag/NextReleaseTagGenerator.kt

@@ -1,7 +1,6 @@
 package flank.scripts.ci.nexttag
 
 import java.time.LocalDate
-import java.time.Year
 import java.time.format.DateTimeFormatter
 
 fun generateNextReleaseTag(previousReleaseTag: String): String {
@@ -14,7 +13,7 @@ fun generateNextReleaseTag(previousReleaseTag: String): String {
 }
 
 private fun isNextReleaseInCurrentMonth(year: String, month: String) =
-    DateTimeFormatter.ofPattern("yy").format(Year.now()) == year &&
+    DateTimeFormatter.ofPattern("yy").format(LocalDate.now()) == year &&
         DateTimeFormatter.ofPattern("MM").format(LocalDate.now()) == month
 
 private fun currentMonthFirstTag() = "v${DateTimeFormatter.ofPattern("yy.MM").format(LocalDate.now())}.0"

release_notes.md

@@ -1,3 +1,44 @@
+## v21.01.0
+### CI Changes
+- [#1439](https://github.com/Flank/flank/pull/1439) Remove sanity checks and change full suite IT OS to macOS ([piotradamczyk5](https://github.com/piotradamczyk5))
+- [#1433](https://github.com/Flank/flank/pull/1433) Change IT workflow comment ([pawelpasterz](https://github.com/pawelpasterz))
+- [#1424](https://github.com/Flank/flank/pull/1424) Auto update firebase api client ([piotradamczyk5](https://github.com/piotradamczyk5))
+- [#1412](https://github.com/Flank/flank/pull/1412) 1411 implement version ([pawelpasterz](https://github.com/pawelpasterz))
+- [#1409](https://github.com/Flank/flank/pull/1409) Trigger integration tests from pr comment ([adamfilipow92](https://github.com/adamfilipow92), [piotradamczyk5](https://github.com/piotradamczyk5))
+- [#1385](https://github.com/Flank/flank/pull/1385) Get rid of binaries submodule ([jan-gogo](https://github.com/jan-gogo), [piotradamczyk5](https://github.com/piotradamczyk5))
+- [#1400](https://github.com/Flank/flank/pull/1400) Change gradle command ([pawelpasterz](https://github.com/pawelpasterz))
+- [#1398](https://github.com/Flank/flank/pull/1398) Add create google account config step ([pawelpasterz](https://github.com/pawelpasterz))
+- [#1353](https://github.com/Flank/flank/pull/1353) Implement workflow for full suite IT (cron + manual) ([pawelpasterz](https://github.com/pawelpasterz), [piotradamczyk5](https://github.com/piotradamczyk5))
+- [#1378](https://github.com/Flank/flank/pull/1378) Update GH API ([pawelpasterz](https://github.com/pawelpasterz))
+### Features
+- [#1436](https://github.com/Flank/flank/pull/1436) 1422 autodetect new gcloud features ([pawelpasterz](https://github.com/pawelpasterz))
+- [#1429](https://github.com/Flank/flank/pull/1429) Change uploading file output ([adamfilipow92](https://github.com/adamfilipow92), [piotradamczyk5](https://github.com/piotradamczyk5))
+- [#1426](https://github.com/Flank/flank/pull/1426) Support for iOS gameloop tests ([Sloox](https://github.com/Sloox), [jan-gogo](https://github.com/jan-gogo), [adamfilipow92](https://github.com/adamfilipow92))
+- [#1421](https://github.com/Flank/flank/pull/1421) Default project Id - use GOOGLE_APPLICATION_CREDENTIALS first, then GOOGLE_CLOUD_PROJECT ([pawelpasterz](https://github.com/pawelpasterz))
+- [#1404](https://github.com/Flank/flank/pull/1404) Add basic gameloop IT for Android ([Sloox](https://github.com/Sloox), [adamfilipow92](https://github.com/adamfilipow92))
+- [#1393](https://github.com/Flank/flank/pull/1393) Upload matrix ids ([Sloox](https://github.com/Sloox), [jan-gogo](https://github.com/jan-gogo))
+- [#1362](https://github.com/Flank/flank/pull/1362) Add logic to verify xml results ([adamfilipow92](https://github.com/adamfilipow92), [piotradamczyk5](https://github.com/piotradamczyk5))
+- [#1321](https://github.com/Flank/flank/pull/1321) IOS support for testplans ([axelzuziak-gogo](https://github.com/axelzuziak-gogo), [jan-gogo](https://github.com/jan-gogo))
+- [#1375](https://github.com/Flank/flank/pull/1375) Add new output style to print smaller output ([adamfilipow92](https://github.com/adamfilipow92), [piotradamczyk5](https://github.com/piotradamczyk5))
+- [#1370](https://github.com/Flank/flank/pull/1370) Add support for "fail fast" feature on FTL ([RainNapper](https://github.com/RainNapper))
+### Refactor
+- [#1432](https://github.com/Flank/flank/pull/1432) Small changes in flank-scripts module ([pawelpasterz](https://github.com/pawelpasterz))
+- [#1418](https://github.com/Flank/flank/pull/1418) Create module with shared utilities ([piotradamczyk5](https://github.com/piotradamczyk5))
+- [#1417](https://github.com/Flank/flank/pull/1417) Update firebase_apis module ([pawelpasterz](https://github.com/pawelpasterz))
+### Bug Fixes
+- [#1414](https://github.com/Flank/flank/pull/1414) Add attempts to performance test ([jan-gogo](https://github.com/jan-gogo))
+- [#1410](https://github.com/Flank/flank/pull/1410) Fix integration tests on CI. ([adamfilipow92](https://github.com/adamfilipow92), [piotradamczyk5](https://github.com/piotradamczyk5))
+- [#1413](https://github.com/Flank/flank/pull/1413) Fix calculate shards test ([pawelpasterz](https://github.com/pawelpasterz), [jan-gogo](https://github.com/jan-gogo))
+- [#1405](https://github.com/Flank/flank/pull/1405) CancelCommandRuns should works independent ([Sloox](https://github.com/Sloox), [adamfilipow92](https://github.com/adamfilipow92))
+- [#1395](https://github.com/Flank/flank/pull/1395) Print args when iOS tests are not found ([jan-gogo](https://github.com/jan-gogo), [adamfilipow92](https://github.com/adamfilipow92))
+- [#1387](https://github.com/Flank/flank/pull/1387) Fix NPE ([pawelpasterz](https://github.com/pawelpasterz), [jan-gogo](https://github.com/jan-gogo))
+- [#1372](https://github.com/Flank/flank/pull/1372) Fix Flank Scripts version verification tasks ([piotradamczyk5](https://github.com/piotradamczyk5))
+### Documentation
+- [#1401](https://github.com/Flank/flank/pull/1401) Update orchestrator version in docs ([adamfilipow92](https://github.com/adamfilipow92))
+- [#1380](https://github.com/Flank/flank/pull/1380) Added documentation about handling #1374 issue ([piotradamczyk5](https://github.com/piotradamczyk5))
+### Tests update
+- [#1316](https://github.com/Flank/flank/pull/1316) [IT] Add test cases ([pawelpasterz](https://github.com/pawelpasterz), [adamfilipow92](https://github.com/adamfilipow92))
+
 ## v20.12.0
 ### Bug Fixes
 - [#1366](https://github.com/Flank/flank/pull/1366) Generation of release notes ([piotradamczyk5](https://github.com/piotradamczyk5))