Flank · piotradamczyk5 · Jul 9, 2020 · Jul 10, 2020 · Jul 10, 2020 · Jul 13, 2020
diff --git a/release_notes.md b/release_notes.md
@@ -8,6 +8,8 @@
 - [#862](https://github.com/Flank/flank/pull/862) Added printing outcome details. ([piotradamczyk5](https://github.com/piotradamczyk5), [jan-gogo](https://github.com/jan-gogo))
 - [#876](https://github.com/Flank/flank/pull/876) Added --directories-to-pull validation and avoid making request with empty toolStepResult. ([piotradamczyk5](https://github.com/piotradamczyk5))
 - [#875](https://github.com/Flank/flank/pull/875) Enhance permission denied exception logs. ([adamfilipow92](https://github.com/adamfilipow92), [pawelpasterz](https://github.com/pawelpasterz))
+- [#876](https://github.com/Flank/flank/pull/876) Added option to print available devices and OS version to test against. ([piotradamczyk5](https://github.com/piotradamczyk5))
+-
 -
 
 ## v20.06.2

diff --git a/test_runner/docs/ascii/flank.jar_-android-run.adoc b/test_runner/docs/ascii/flank.jar_-android-run.adoc
@@ -85,6 +85,9 @@ Configuration is read from flank.yml
 *--device*=_<String=String>_[,_<String=String>_...]::
   A list of DIMENSION=VALUE pairs which specify a target device to test against. This flag may be repeated to specify multiple devices. The four device dimensions are: model, version, locale, and orientation. If any dimensions are omitted, they will use a default value. Omitting all of the preceding dimension-related flags will run tests against a single device using defaults for all four device dimensions.
 
+*--async*::
+  Invoke a test asynchronously without waiting for test results.
+
 *--results-bucket*=_<resultsBucket>_::
   The name of a Google Cloud Storage bucket where raw test results will be stored (default: "test-lab-<random-UUID>"). Note that the bucket must be owned by a billing-enabled project, and that using a non-default bucket will result in billing charges for the storage used.
 
@@ -100,9 +103,6 @@ Configuration is read from flank.yml
 *--timeout*=_<timeout>_::
   The max time this test execution can run before it is cancelled (default: 15m). It does not include any time necessary to prepare and clean up the target device. The maximum possible testing time is 30m on physical devices and 60m on virtual devices. The TIMEOUT units can be h, m, or s. If no unit is given, seconds are assumed. 
 
-*--async*::
-  Invoke a test asynchronously without waiting for test results.
-
 *--client-details*=_<String=String>_[,_<String=String>_...]::
   Comma-separated, KEY=VALUE map of additional details to attach to the test matrix.Arbitrary KEY=VALUE pairs may be attached to a test matrix to provide additional context about the tests being run.When consuming the test results, such as in Cloud Functions or a CI system,these details can add additional context such as a link to the corresponding pull request.
 
@@ -115,12 +115,12 @@ Configuration is read from flank.yml
 *--num-flaky-test-attempts*=_<flakyTestAttempts>_::
   The number of times a TestExecution should be re-attempted if one or more of its test cases fail for any reason. The maximum number of reruns allowed is 10. Default is 0, which implies no reruns.
 
-*--max-test-shards*=_<maxTestShards>_::
-  The amount of matrices to split the tests across.
-
 *--shard-time*=_<shardTime>_::
   The max amount of seconds each shard should run.
 
+*--max-test-shards*=_<maxTestShards>_::
+  The amount of matrices to split the tests across.
+
 *--num-test-runs*=_<repeatTests>_::
   The amount of times to run the test executions.
 

diff --git a/test_runner/docs/ascii/flank.jar_-android.adoc b/test_runner/docs/ascii/flank.jar_-android.adoc
@@ -32,6 +32,15 @@ flank.jar
 *doctor*::
   Verifies flank firebase is setup correctly
 
+*models*::
+  Information about available models
+
+*versions*::
+  Information about available software versions
+
+*test-environment*::
+  Print available devices and OS versions list to test against
+
 // end::picocli-generated-man-section-commands[]
 
 // end::picocli-generated-full-manpage[]
diff --git a/test_runner/docs/ascii/flank.jar_-firebase-test-android-run.adoc b/test_runner/docs/ascii/flank.jar_-firebase-test-android-run.adoc
@@ -97,6 +97,9 @@ Configuration is read from flank.yml
 *--device*=_<String=String>_[,_<String=String>_...]::
   A list of DIMENSION=VALUE pairs which specify a target device to test against. This flag may be repeated to specify multiple devices. The four device dimensions are: model, version, locale, and orientation. If any dimensions are omitted, they will use a default value. Omitting all of the preceding dimension-related flags will run tests against a single device using defaults for all four device dimensions.
 
+*--async*::
+  Invoke a test asynchronously without waiting for test results.
+
 *--results-bucket*=_<resultsBucket>_::
   The name of a Google Cloud Storage bucket where raw test results will be stored (default: "test-lab-<random-UUID>"). Note that the bucket must be owned by a billing-enabled project, and that using a non-default bucket will result in billing charges for the storage used.
 
@@ -112,9 +115,6 @@ Configuration is read from flank.yml
 *--timeout*=_<timeout>_::
   The max time this test execution can run before it is cancelled (default: 15m). It does not include any time necessary to prepare and clean up the target device. The maximum possible testing time is 30m on physical devices and 60m on virtual devices. The TIMEOUT units can be h, m, or s. If no unit is given, seconds are assumed. 
 
-*--async*::
-  Invoke a test asynchronously without waiting for test results.
-
 *--client-details*=_<String=String>_[,_<String=String>_...]::
   Comma-separated, KEY=VALUE map of additional details to attach to the test matrix.Arbitrary KEY=VALUE pairs may be attached to a test matrix to provide additional context about the tests being run.When consuming the test results, such as in Cloud Functions or a CI system,these details can add additional context such as a link to the corresponding pull request.
 
@@ -127,12 +127,12 @@ Configuration is read from flank.yml
 *--num-flaky-test-attempts*=_<flakyTestAttempts>_::
   The number of times a TestExecution should be re-attempted if one or more of its test cases fail for any reason. The maximum number of reruns allowed is 10. Default is 0, which implies no reruns.
 
-*--max-test-shards*=_<maxTestShards>_::
-  The amount of matrices to split the tests across.
-
 *--shard-time*=_<shardTime>_::
   The max amount of seconds each shard should run.
 
+*--max-test-shards*=_<maxTestShards>_::
+  The amount of matrices to split the tests across.
+
 *--num-test-runs*=_<repeatTests>_::
   The amount of times to run the test executions.
 

diff --git a/test_runner/docs/ascii/flank.jar_-firebase-test-android.adoc b/test_runner/docs/ascii/flank.jar_-firebase-test-android.adoc
@@ -32,6 +32,15 @@ flank.jar
 *doctor*::
   Verifies flank firebase is setup correctly
 
+*models*::
+  Information about available models
+
+*versions*::
+  Information about available software versions
+
+*test-environment*::
+  Print available devices and OS versions list to test against
+
 // end::picocli-generated-man-section-commands[]
 
 // end::picocli-generated-full-manpage[]
diff --git a/test_runner/docs/ascii/flank.jar_-firebase-test-ios-run.adoc b/test_runner/docs/ascii/flank.jar_-firebase-test-ios-run.adoc
@@ -76,6 +76,9 @@ Configuration is read from flank.yml
 *--device*=_<String=String>_[,_<String=String>_...]::
   A list of DIMENSION=VALUE pairs which specify a target device to test against. This flag may be repeated to specify multiple devices. The four device dimensions are: model, version, locale, and orientation. If any dimensions are omitted, they will use a default value. Omitting all of the preceding dimension-related flags will run tests against a single device using defaults for all four device dimensions.
 
+*--async*::
+  Invoke a test asynchronously without waiting for test results.
+
 *--results-bucket*=_<resultsBucket>_::
   The name of a Google Cloud Storage bucket where raw test results will be stored (default: "test-lab-<random-UUID>"). Note that the bucket must be owned by a billing-enabled project, and that using a non-default bucket will result in billing charges for the storage used.
 
@@ -91,9 +94,6 @@ Configuration is read from flank.yml
 *--timeout*=_<timeout>_::
   The max time this test execution can run before it is cancelled (default: 15m). It does not include any time necessary to prepare and clean up the target device. The maximum possible testing time is 30m on physical devices and 60m on virtual devices. The TIMEOUT units can be h, m, or s. If no unit is given, seconds are assumed. 
 
-*--async*::
-  Invoke a test asynchronously without waiting for test results.
-
 *--client-details*=_<String=String>_[,_<String=String>_...]::
   Comma-separated, KEY=VALUE map of additional details to attach to the test matrix.Arbitrary KEY=VALUE pairs may be attached to a test matrix to provide additional context about the tests being run.When consuming the test results, such as in Cloud Functions or a CI system,these details can add additional context such as a link to the corresponding pull request.
 
@@ -106,12 +106,12 @@ Configuration is read from flank.yml
 *--num-flaky-test-attempts*=_<flakyTestAttempts>_::
   The number of times a TestExecution should be re-attempted if one or more of its test cases fail for any reason. The maximum number of reruns allowed is 10. Default is 0, which implies no reruns.
 
-*--max-test-shards*=_<maxTestShards>_::
-  The amount of matrices to split the tests across.
-
 *--shard-time*=_<shardTime>_::
   The max amount of seconds each shard should run.
 
+*--max-test-shards*=_<maxTestShards>_::
+  The amount of matrices to split the tests across.
+
 *--num-test-runs*=_<repeatTests>_::
   The amount of times to run the test executions.
 

diff --git a/test_runner/docs/ascii/flank.jar_-firebase-test-ios.adoc b/test_runner/docs/ascii/flank.jar_-firebase-test-ios.adoc
@@ -32,6 +32,15 @@ flank.jar
 *doctor*::
   Verifies flank firebase is setup correctly
 
+*models*::
+  Information about available models
+
+*versions*::
+  Information about available software versions
+
+*test-environment*::
+  Print available devices and OS versions list to test against
+
 // end::picocli-generated-man-section-commands[]
 
 // end::picocli-generated-full-manpage[]
diff --git a/test_runner/docs/ascii/flank.jar_-ios-run.adoc b/test_runner/docs/ascii/flank.jar_-ios-run.adoc
@@ -68,6 +68,9 @@ Configuration is read from flank.yml
 *--device*=_<String=String>_[,_<String=String>_...]::
   A list of DIMENSION=VALUE pairs which specify a target device to test against. This flag may be repeated to specify multiple devices. The four device dimensions are: model, version, locale, and orientation. If any dimensions are omitted, they will use a default value. Omitting all of the preceding dimension-related flags will run tests against a single device using defaults for all four device dimensions.
 
+*--async*::
+  Invoke a test asynchronously without waiting for test results.
+
 *--results-bucket*=_<resultsBucket>_::
   The name of a Google Cloud Storage bucket where raw test results will be stored (default: "test-lab-<random-UUID>"). Note that the bucket must be owned by a billing-enabled project, and that using a non-default bucket will result in billing charges for the storage used.
 
@@ -83,9 +86,6 @@ Configuration is read from flank.yml
 *--timeout*=_<timeout>_::
   The max time this test execution can run before it is cancelled (default: 15m). It does not include any time necessary to prepare and clean up the target device. The maximum possible testing time is 30m on physical devices and 60m on virtual devices. The TIMEOUT units can be h, m, or s. If no unit is given, seconds are assumed. 
 
-*--async*::
-  Invoke a test asynchronously without waiting for test results.
-
 *--client-details*=_<String=String>_[,_<String=String>_...]::
   Comma-separated, KEY=VALUE map of additional details to attach to the test matrix.Arbitrary KEY=VALUE pairs may be attached to a test matrix to provide additional context about the tests being run.When consuming the test results, such as in Cloud Functions or a CI system,these details can add additional context such as a link to the corresponding pull request.
 
@@ -98,12 +98,12 @@ Configuration is read from flank.yml
 *--num-flaky-test-attempts*=_<flakyTestAttempts>_::
   The number of times a TestExecution should be re-attempted if one or more of its test cases fail for any reason. The maximum number of reruns allowed is 10. Default is 0, which implies no reruns.
 
-*--max-test-shards*=_<maxTestShards>_::
-  The amount of matrices to split the tests across.
-
 *--shard-time*=_<shardTime>_::
   The max amount of seconds each shard should run.
 
+*--max-test-shards*=_<maxTestShards>_::
+  The amount of matrices to split the tests across.
+
 *--num-test-runs*=_<repeatTests>_::
   The amount of times to run the test executions.
 

diff --git a/test_runner/docs/ascii/flank.jar_-ios.adoc b/test_runner/docs/ascii/flank.jar_-ios.adoc
@@ -32,6 +32,15 @@ flank.jar
 *doctor*::
   Verifies flank firebase is setup correctly
 
+*models*::
+  Information about available models
+
+*versions*::
+  Information about available software versions
+
+*test-environment*::
+  Print available devices and OS versions list to test against
+
 // end::picocli-generated-man-section-commands[]
 
 // end::picocli-generated-full-manpage[]
diff --git a/test_runner/src/main/kotlin/ftl/android/AndroidCatalog.kt b/test_runner/src/main/kotlin/ftl/android/AndroidCatalog.kt
@@ -2,6 +2,7 @@ package ftl.android
 
 import com.google.api.services.testing.model.AndroidDevice
 import com.google.api.services.testing.model.AndroidDeviceCatalog
+import ftl.environment.asPrintableTable
 import ftl.gc.GcTesting
 import ftl.http.executeWithRetry
 
@@ -14,42 +15,30 @@ object AndroidCatalog {
     private val modelMap: MutableMap<String, List<String>> = mutableMapOf()
     private val versionMap: MutableMap<String, List<String>> = mutableMapOf()
 
-    private fun androidDeviceCatalog(projectId: String): AndroidDeviceCatalog {
-        val cached = catalogMap[projectId]
-        if (cached != null) return cached
-
-        val newCatalog = GcTesting.get.testEnvironmentCatalog()
+    private fun deviceCatalog(projectId: String) = catalogMap.getOrPut(projectId) {
+        GcTesting.get.testEnvironmentCatalog()
             .get("android")
             .setProjectId(projectId)
-            .executeWithRetry().androidDeviceCatalog
-        catalogMap[projectId] = newCatalog
-        return newCatalog
+            .executeWithRetry()
+            .androidDeviceCatalog
     }
 
-    fun androidModelIds(projectId: String): List<String> {
-        val cached = modelMap[projectId]
-        if (cached != null) return cached
+    fun devicesCatalogAsTable(projectId: String) = deviceCatalog(projectId).models.asPrintableTable()
 
-        val newModels = androidDeviceCatalog(projectId).models.map { it.id }
-        modelMap[projectId] = newModels
-        return newModels
-    }
+    fun supportedVersionsAsTable(projectId: String) = deviceCatalog(projectId).versions.asPrintableTable()
 
-    fun androidVersionIds(projectId: String): List<String> {
-        val cached = versionMap[projectId]
-        if (cached != null) return cached
+    fun androidModelIds(projectId: String) =
+        modelMap.getOrPut(projectId) { deviceCatalog(projectId).models.map { it.id } }
 
-        val newVersions = androidDeviceCatalog(projectId).versions.map { it.id }
-        versionMap[projectId] = newVersions
-        return newVersions
-    }
+    fun androidVersionIds(projectId: String) =
+        versionMap.getOrPut(projectId) { deviceCatalog(projectId).versions.map { it.id } }
 
     fun supportedDeviceConfig(modelId: String, versionId: String, projectId: String): DeviceConfigCheck {
-        val foundModel = androidDeviceCatalog(projectId).models.find { it.id == modelId } ?: return UnsupportedModelId
+
+        val foundModel = deviceCatalog(projectId).models.find { it.id == modelId } ?: return UnsupportedModelId
         if (!androidVersionIds(projectId).contains(versionId)) return UnsupportedVersionId
 
-        val supportedVersionIds = foundModel.supportedVersionIds
-        supportedVersionIds?.let {
+        foundModel.supportedVersionIds?.let {
             if (!it.contains(versionId)) return IncompatibleModelVersion
         } ?: return UnsupportedModelId
 
@@ -58,7 +47,7 @@ object AndroidCatalog {
 
     fun isVirtualDevice(device: AndroidDevice?, projectId: String): Boolean {
         val modelId = device?.androidModelId ?: return false
-        val form = androidDeviceCatalog(projectId).models.find { it.id == modelId }?.form ?: "PHYSICAL"
+        val form = deviceCatalog(projectId).models.find { it.id == modelId }?.form ?: "PHYSICAL"
         return form == "VIRTUAL"
     }
 }

diff --git a/test_runner/src/main/kotlin/ftl/cli/firebase/test/AndroidCommand.kt b/test_runner/src/main/kotlin/ftl/cli/firebase/test/AndroidCommand.kt
@@ -2,6 +2,9 @@ package ftl.cli.firebase.test
 
 import ftl.cli.firebase.test.android.AndroidDoctorCommand
 import ftl.cli.firebase.test.android.AndroidRunCommand
+import ftl.cli.firebase.test.android.AndroidTestEnvironmentCommand
+import ftl.cli.firebase.test.android.models.AndroidModelsCommand
+import ftl.cli.firebase.test.android.versions.AndroidVersionsCommand
 import picocli.CommandLine
 import picocli.CommandLine.Command
 
@@ -10,7 +13,10 @@ import picocli.CommandLine.Command
     synopsisHeading = "",
     subcommands = [
         AndroidRunCommand::class,
-        AndroidDoctorCommand::class
+        AndroidDoctorCommand::class,
+        AndroidModelsCommand::class,
+        AndroidVersionsCommand::class,
+        AndroidTestEnvironmentCommand::class
     ],
     usageHelpAutoWidth = true
 )

diff --git a/test_runner/src/main/kotlin/ftl/cli/firebase/test/CommonRunCommand.kt b/test_runner/src/main/kotlin/ftl/cli/firebase/test/CommonRunCommand.kt
@@ -49,6 +49,7 @@ abstract class CommonRunCommand {
     @CommandLine.Option(
         names = ["--obfuscate"],
         description = ["Replacing internal test names with unique identifiers when using --dump-shards option " +
-                "to avoid exposing internal details"])
+                "to avoid exposing internal details"]
+    )
     var obfuscate: Boolean = false
 }
diff --git a/test_runner/src/main/kotlin/ftl/cli/firebase/test/IosCommand.kt b/test_runner/src/main/kotlin/ftl/cli/firebase/test/IosCommand.kt
@@ -2,6 +2,9 @@ package ftl.cli.firebase.test
 
 import ftl.cli.firebase.test.ios.IosDoctorCommand
 import ftl.cli.firebase.test.ios.IosRunCommand
+import ftl.cli.firebase.test.ios.IosTestEnvironmentCommand
+import ftl.cli.firebase.test.ios.models.IosModelsCommand
+import ftl.cli.firebase.test.ios.versions.IosVersionsCommand
 import picocli.CommandLine
 import picocli.CommandLine.Command
 
@@ -10,7 +13,10 @@ import picocli.CommandLine.Command
     synopsisHeading = "",
     subcommands = [
         IosRunCommand::class,
-        IosDoctorCommand::class
+        IosDoctorCommand::class,
+        IosModelsCommand::class,
+        IosVersionsCommand::class,
+        IosTestEnvironmentCommand::class
     ],
     usageHelpAutoWidth = true
 )

diff --git a/test_runner/src/main/kotlin/ftl/cli/firebase/test/android/AndroidDoctorCommand.kt b/test_runner/src/main/kotlin/ftl/cli/firebase/test/android/AndroidDoctorCommand.kt
@@ -2,6 +2,7 @@ package ftl.cli.firebase.test.android
 
 import ftl.args.AndroidArgs
 import ftl.cli.firebase.test.processValidation
+import ftl.config.FtlConstants
 import ftl.doctor.Doctor.validateYaml
 import java.nio.file.Paths
 import picocli.CommandLine.Command
@@ -28,7 +29,7 @@ class AndroidDoctorCommand : Runnable {
     }
 
     @Option(names = ["-c", "--config"], description = ["YAML config file path"])
-    var configPath: String = "./flank.yml"
+    var configPath: String = FtlConstants.defaultAndroidConfig
 
     @Option(names = ["-h", "--help"], usageHelp = true, description = ["Prints this help message"])
     var usageHelpRequested: Boolean = false

diff --git a/test_runner/src/main/kotlin/ftl/cli/firebase/test/android/AndroidTestEnvironmentCommand.kt b/test_runner/src/main/kotlin/ftl/cli/firebase/test/android/AndroidTestEnvironmentCommand.kt
@@ -0,0 +1,32 @@
+package ftl.cli.firebase.test.android
+
+import ftl.android.AndroidCatalog.devicesCatalogAsTable
+import ftl.android.AndroidCatalog.supportedVersionsAsTable
+import ftl.args.AndroidArgs
+import ftl.config.FtlConstants
+import picocli.CommandLine
+import java.nio.file.Paths
+
+@CommandLine.Command(
+    name = "test-environment",
+    headerHeading = "",
+    synopsisHeading = "%n",
+    descriptionHeading = "%n@|bold,underline Description:|@%n%n",
+    parameterListHeading = "%n@|bold,underline Parameters:|@%n",
+    optionListHeading = "%n@|bold,underline Options:|@%n",
+    header = ["Print available devices and OS versions list to test against"],
+    description = ["Print available Android devices and Android OS versions list to test against"],
+    usageHelpAutoWidth = true
+)
+class AndroidTestEnvironmentCommand : Runnable {
+    override fun run() {
+        println(devicesCatalogAsTable(AndroidArgs.load(Paths.get(configPath)).project))
+        println(supportedVersionsAsTable(AndroidArgs.load(Paths.get(configPath)).project))
+    }
+
+    @CommandLine.Option(names = ["-c", "--config"], description = ["YAML config file path"])
+    var configPath: String = FtlConstants.defaultAndroidConfig
+
+    @CommandLine.Option(names = ["-h", "--help"], usageHelp = true, description = ["Prints this help message"])
+    var usageHelpRequested: Boolean = false
+}