Allow fine turning configurations per APK pair whose tests are ran

[MatthewTPage]

Author the user story for this feature

As a consumer of the Flank project, I want to be able to override configures per test APK in additional-app-test-apks so I can run tests for different modules with different configurations.

My personal use case is:

1. I want to specify which tests to run so Flank doesn't need to download and parse the APKs from the gcloud bucket.
2. We'd like to runs different modules on different devices.

Describe the solution

I'll attach a SDD describing a solution shortly.

Describe alternatives considered

Again see the SDD.

[MatthewTPage]

SDD

Project name

Allow fine turning configurations per APK pair whose tests are ran

References [optional for issue commens]

Slack convo: https://firebase-community.slack.com/archives/C72V6UW8M/p1615398975025500

Motivation

For my personal use case, my team has tests distributed among many modules (~25 or so) and we use the additional-app-test-apks property to run all of our tests at once. But we have a need to fine tune configurations per test pair being ran, specifically:

1. I want to specify which tests to run so Flank doesn't need to download and parse the APKs from the gcloud bucket.
2. We'd like to runs different modules on different devices.

In general other teams using the additional-app-test-apks property could probably also benefit from being able to customize test runs with different parameters / configurations.

Goals

Implementation of the above feature, not sure what else to specify.

Non-Goals [optional]

N/A

Design

See the API below, each app/test APK pair will also accept optional parameters. To avoid needing to duplicate every parameter in the AppTestPair.kt data class, it'll initialize objects for AndroidGcloudConfig, AndroidFlankConfig, CommonGcloudConfig, and CommonFlankConfig all initialized by the yaml map, and with all unspecified parameters as null.

In CreateAndroidArgs.kt we will map ApkTestPair into a new data type that condenses the four data objects.

The parameter verification step will assert no parameters that should be reserved for the macro configuration are specified (ie smart flank configurations, or specifying additional test apk groups).

InstrumentationTestContext will take an additional parameter for the arguments to use, and this is the args object that should be referenced instead of the global args object for the sake of test execution (from what I can tell, the only file needing to change which args object it references is the CreateAndroidTestContext.kt kotlin file).

API

  ### Additional App/Test APKS 
  ## Include additional app/test apk pairs in the run. Apks are unique by just filename and not by path!
  ## If app is omitted, then the top level app is used for that pair.
  # additional-app-test-apks:
  #  - app: ../test_projects/android/apks/app-debug.apk
  #    test: ../test_projects/android/apks/app1-debug-androidTest.apk 
  #    test-targets:
  #     - class com.example.app.ExampleUiTest#testPasses
  #    device:
  #      - model: NexusLowRes
  #         version: 28
  #         locale: en
  #         orientation: portrait
  #      - model: NexusLowRes
  #         version: 27  
  #    
  #  - test: ../test_projects/android/apks/app2-debug-androidTest.apk

etc etc. I won't list out every option but you get the idea.

Results

Implementation of the above feature, not sure what else to specify.

Dependencies

Nothing Flank depends on should need updating. Projects depending on flank wanting to incorporate this feature (ie Fladle) will obviously need updating.

Testing

Automated test coverage of parsing Yaml configs and determining the correct parameters to run a test APK under. I will also try to plug the snapshot into my team's build and test it there, for a full end-to-end test.

Alternatives Considered [optional]

N/A

Timeline [optional for regular tigers]

Not sure, let's first discuss if anyone objects to this.

DONE:

• SDD Drafting

NEXT:

• SDD review
• Implementation
• Testing
• Rollout

[bootstraponline]

The SDD looks good to me!

[asadsalman]

I picked this feature up since there hasn't been an update from OP.
Working on parsing fields in AppTestPair, added the following fields to the class:

@JsonProperty("gcloud")
val gcloudConfig: CommonGcloudConfig? = null,
@JsonProperty("flank")
val flankConfig: CommonFlankConfig? = null,

The problem is, as expected, I'm only getting the properties that exist inside the CommonGcloudConfig and CommonFlankConfig classes, not in the AndroidGcloudConfig or AndroidFlankConfig classes. The opposite happens if I use Android*Config classes while parsing.

Is there anything I'm missing? Are there any classes I should use instead of Common or Android configs that will catch all gcloud/flank configs?

@pawelpasterz thoughts?

[pawelpasterz]

Hm...I think there is no out of the box class that can be used here 🤔
@jan-gogo might have input on this matter

[Sloox]

@pawelpasterz @jan-gogo As discussed, @asadsalman will get his changes ready within this branch: #1898
We will then pull this ticket in ASAP into the next sprint. Will bump up the priority.

[adamfilipow92]

#1898 is closed, should we close this ticket or there is an additional work? @Sloox

[pawelpasterz]

@MatthewTPage
Coming back to you with the outcome of this task:

I want to specify which tests to run so Flank doesn't need to download and parse the APKs from the gcloud bucket.

You can achieve it with CUSTOM SHARDING FEATURE.

We'd like to runs different modules on different devices.

Implemented in #2004 , not all options are implemented (currently device, max-test-shards, test-targets, client-details), if you will need anything more, let us know