Add support for robo-directives & robo-script options #709

jan-goral · 2020-04-08T10:28:09Z

Fixes #706

Checklist

Documented
Unit tested
release_notes.md updated

codecov-io · 2020-04-09T23:56:07Z

Codecov Report

Merging #709 into master will decrease coverage by 1.26%.
The diff coverage is 66.66%.

@@             Coverage Diff              @@
##             master     #709      +/-   ##
============================================
- Coverage     79.48%   78.22%   -1.27%     
- Complexity      666      676      +10     
============================================
  Files           113      125      +12     
  Lines          2696     2787      +91     
  Branches        385      402      +17     
============================================
+ Hits           2143     2180      +37     
- Misses          305      348      +43     
- Partials        248      259      +11

jan-goral · 2020-04-10T07:25:57Z

There is a little bit more work to do here. Android instrumentation tests and robo tests are mutually exclusive, they cannot be specified both. It requires additional changes how args are validated and test matrix build.

bootstraponline · 2020-04-12T18:54:06Z

test_runner/flank.yml

+  ## The path may be in the local filesystem or in Google Cloud Storage using gs:// notation.
+  ## You can guide the Robo test to perform specific actions by recording a Robo Script in Android Studio and then specifying this argument.
+  ## Learn more at https://firebase.google.com/docs/test-lab/robo-ux-test#scripting.
+  # robo-script: path_to_robo_script


Nice work! Remember to update readme.md as well.

I've updated bunch of config options in README.md and flank.yml because both of them had lack of some updates.

bootstraponline · 2020-04-12T18:56:28Z

For new features from gcloud, it's good to test the yaml syntax with gcloud CLI.

gcloud firebase test android run flank.yml:gcloud

I'd test and confirm that robo-directives and robo-script works well both from Flank and gcloud.

* Prevent crash by null testSuiteOverviews when using robo test * Prepare test_app for robo test * Add robo script to fixtures

jan-goral · 2020-04-21T11:33:41Z

For new features from gcloud, it's good to test the yaml syntax with gcloud CLI.
* `gcloud firebase test android run flank.yml:gcloud`
I'd test and confirm that robo-directives and robo-script works well both from Flank and gcloud.

Thanks for advise, that was helpful to keep yml config consistent with gcloud. I tested robo-directives and robo-script with Flank and gcloud firebase test android run flank.yml:gcloud. Both seems to work well.

pawelpasterz · 2020-04-21T17:30:42Z

test_app/app/src/androidTest/java/com/example/test_app/BaseInstrumentedTest.kt

@@ -29,7 +29,7 @@ abstract class BaseInstrumentedTest {
        val result: Boolean = when (BuildConfig.FLAVOR_type) {
            "success" -> true
            "error" -> false
-            "flaky" -> SystemClock.uptimeMillis() % 2 == 0L
+            "flaky" -> Random.nextBoolean()


pawelpasterz · 2020-04-22T15:08:21Z

test_runner/src/main/kotlin/ftl/args/AndroidArgs.kt

@@ -104,9 +107,21 @@ class AndroidArgs(
            "Option num-uniform-shards cannot be specified along with max-test-shards. Use only one of them"
        )

+        if (!(isRoboTest xor isInstrumentationTest)) throw FlankFatalError(
+            "Option test xor robo-directives xor robo-script must be specified"


How about xor -> or?
Maybe we could use sth like One of bla bla bla. Just thinking

Flank is for programmers not end users, I guess every programmer knows what xor does mean. Anyway if you propose something better than One of bla bla bla ;), I can replace.

we have a community of QA engineers using Firebase Test Lab, they may not be familiar with what xor means 🙂

pawelpasterz · 2020-04-22T15:14:06Z

test_runner/src/main/kotlin/ftl/config/FlankRoboDirective.kt

+        get() = if (input.isNotBlank()) input else "\"\""
+}
+
+fun List<String>.parseRoboDirectives() = map {


List<String>#parseRoboDirectives and Map<...>#parseRoboDirectives are only used inside AndroidArgs class. How about moving those there and make private?

I put those mappers here because AndroidArgs has over 200 lines and do to much.

pawelpasterz · 2020-04-22T15:32:11Z

test_runner/src/main/kotlin/ftl/reports/MatrixResultsReport.kt

@@ -23,7 +25,7 @@ Example:
 object MatrixResultsReport : IReport {
    override val extension = ".txt"

-    private val percentFormat by lazy { DecimalFormat("#0.00") }
+    private val percentFormat by lazy { DecimalFormat("#0.00", DecimalFormatSymbols(Locale.US)) }


Why this change? Previous implementation takes default locale

public DecimalFormat(String pattern) { // Always applyPattern after the symbols are set this.symbols = DecimalFormatSymbols.getInstance(Locale.getDefault(Locale.Category.FORMAT)); applyPattern(pattern, false); }

I am just curious

Because defaults on some locales change . into , which breaks 3 unit tests. That fix is required because in commit 18b72b5 maven( url = "https://maven-central.storage-download.googleapis.com/repos/central/data/" ) was changed to jcenter() I guess.

I moved to jcenter since the old URL was causing build failures locally and is non-standard.

I think this code is related to printing the cost in US dollar so having that locale makes sense.

pawelpasterz · 2020-04-22T15:57:48Z

test_runner/src/main/kotlin/ftl/run/platform/android/CreateAndroidTestConfig.kt

+): AndroidTestConfig = when {
+    isInstrumentationTest -> createInstrumentationConfig(
+        uploadedApks = uploadedApks,
+        testShards = testShards!!


IMO we should avoid !! operators. I think we should check and throw an error with detailed info. I know that now we are 100% sure (are we?) that null value will never happen here. But this could be useful during refactor or other changes.

pawelpasterz · 2020-04-22T15:59:46Z

test_runner/src/main/kotlin/ftl/run/platform/android/CreateAndroidTestConfig.kt

+    )
+    isRoboTest -> createRoboConfig(
+        uploadedApks = uploadedApks,
+        runGcsPath = runGcsPath!!


As previous comment

jan-goral self-assigned this Apr 8, 2020

jan-goral added the Feature label Apr 8, 2020

jan-goral added this to the May 2020 milestone Apr 8, 2020

jan-goral changed the title ~~Add robo for robo-directives & robo-script options~~ Add support for robo-directives & robo-script options Apr 9, 2020

jan-goral force-pushed the 706-robo-script-beta-option branch from 63265ed to 00f6817 Compare April 9, 2020 23:51

jan-goral marked this pull request as ready for review April 10, 2020 00:06

jan-goral requested review from bootstraponline and pawelpasterz April 10, 2020 00:07

jan-goral marked this pull request as draft April 10, 2020 07:09

bootstraponline reviewed Apr 12, 2020

View reviewed changes

jan-goral added 2 commits April 14, 2020 05:35

Add robo for robo-directives & robo-script options

30f3aa1

Update release_notes.md

401b93b

jan-goral force-pushed the 706-robo-script-beta-option branch from 00f6817 to 401b93b Compare April 14, 2020 03:38

jan-goral added 7 commits April 14, 2020 05:46

Prevent from replacing '.' with ',' on some locales

2946bcc

Make robo and instrumented tests mutually exclusive

55d682f

Fix detekt issue

50fdd82

Update missing config options in README.md which exist in flank.yml

ccd535d

Update flank.yml options that was previously changed only in README.md

4733194

Cleanup

5ff256a

Fix robo-directives yml format

dc59bd6

* Prevent crash by null testSuiteOverviews when using robo test * Prepare test_app for robo test * Add robo script to fixtures

jan-goral marked this pull request as ready for review April 21, 2020 11:21

jan-goral requested a review from bootstraponline April 21, 2020 11:33

Fix example flank.yml and README.md

142c0fb

pawelpasterz reviewed Apr 21, 2020

View reviewed changes

jan-goral requested a review from pawelpasterz April 22, 2020 10:17

pawelpasterz reviewed Apr 22, 2020

View reviewed changes

CR fixes

1b42db5

jan-goral requested a review from pawelpasterz April 22, 2020 21:13

pawelpasterz approved these changes Apr 23, 2020

View reviewed changes

jan-goral merged commit b9596bc into master Apr 23, 2020

jan-goral deleted the 706-robo-script-beta-option branch April 23, 2020 12:45

jan-goral added the New Option Used to track PR with new configuration option (useful when updating fladle) label Oct 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for robo-directives & robo-script options #709

Add support for robo-directives & robo-script options #709

jan-goral commented Apr 8, 2020 •

edited

Loading

codecov-io commented Apr 9, 2020 •

edited

Loading

jan-goral commented Apr 10, 2020

bootstraponline Apr 12, 2020

jan-goral Apr 15, 2020

bootstraponline commented Apr 12, 2020

jan-goral commented Apr 21, 2020

pawelpasterz Apr 21, 2020

pawelpasterz Apr 22, 2020

jan-goral Apr 22, 2020

bootstraponline Apr 22, 2020

pawelpasterz Apr 22, 2020

jan-goral Apr 22, 2020

pawelpasterz Apr 22, 2020

jan-goral Apr 22, 2020

bootstraponline Apr 22, 2020

pawelpasterz Apr 22, 2020

pawelpasterz Apr 22, 2020

Add support for robo-directives & robo-script options #709

Add support for robo-directives & robo-script options #709

Conversation

jan-goral commented Apr 8, 2020 • edited Loading

Checklist

codecov-io commented Apr 9, 2020 • edited Loading

Codecov Report

jan-goral commented Apr 10, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bootstraponline commented Apr 12, 2020

jan-goral commented Apr 21, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jan-goral commented Apr 8, 2020 •

edited

Loading

codecov-io commented Apr 9, 2020 •

edited

Loading