Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Environment Variables set using additional-app-test-apks are not overriding #2111

Closed
jamiehuson opened this issue Jul 30, 2021 · 0 comments · Fixed by #2122
Closed

Environment Variables set using additional-app-test-apks are not overriding #2111

jamiehuson opened this issue Jul 30, 2021 · 0 comments · Fixed by #2122
Assignees
Labels

Comments

@jamiehuson
Copy link

Describe the bug
When following the docs for setting different environment variables in different matrices (https://github.com/Flank/flank/blob/master/docs/using_different_environment_variables_in_different_matrices.md) I am not seeing my environment variables changing between runs on Firebase.

To Reproduce
A slightly simplified version with relevant blocks is provided. I have a custom test runner which looks for an argument called theme which then sets the theme to either light mode or dark mode during the test run. I would like to run both themes in a single flank run across the same tests so I am setting the same test apk on both.

# See the docs for full gcloud details https://cloud.google.com/sdk/gcloud/reference/firebase/test/android/run
gcloud:
  # -- GcloudYml --

  ### Number of Flaky Test Attempts
  ## The number of times a TestExecution should be re-attempted if one or more\nof its test cases fail for any reason.
  ## The maximum number of reruns allowed is 10. Default is 0, which implies no reruns.
  num-flaky-test-attempts: 1

  # -- AndroidGcloudYml --

  ## Android Application Path
  ## The path to the application binary file.
  ## The path may be in the local filesystem or in Google Cloud Storage using gs:// notation.
  ## Android App Bundles are specified as .aab, all other files are assumed to be APKs.
  app: ./StyleGuide/build/outputs/apk/debug/StyleGuide-debug.apk

  ### Android Binary File Path
  ## The path to the binary file containing instrumentation tests.
  ## The given path may be in the local filesystem or in Google Cloud Storage using a URL beginning with gs://.
  test: ./StyleGuide/build/outputs/apk/androidTest/debug/StyleGuide-debug-androidTest.apk

  ### Use Orchestrator Flag
  ## Whether each test runs in its own Instrumentation instance with the Android Test Orchestrator
  ## (default: Orchestrator is used). Disable with --no-use-orchestrator.
  ## See https://developer.android.com/training/testing/junit-runner.html#using-android-test-orchestrator
  use-orchestrator: true

  ### Environment Variables
  ## A comma-separated, key=value map of environment variables and their desired values. This flag is repeatable.
  ## The environment variables are mirrored as extra options to the am instrument -e KEY1 VALUE1 … command and
  ## passed to your test runner (typically AndroidJUnitRunner)
  environment-variables:
    clearPackageData: true
    theme: darkMode

  ### Directories to Pull
  ## A list of paths that will be copied from the device's storage to the designated results bucket after the test
  ## is complete. These must be absolute paths under /sdcard or /data/local/tmp
  directories-to-pull:
    - /sdcard/Pictures/SnapshotsTestResults/

  ### Test Type
  ## The type of test to run. TYPE must be one of: instrumentation, robo, game-loop.
  type: instrumentation

flank:
# -- FlankYml --

### Max Test Shards
## test shards - the amount of groups to split the test suite into
## set to -1 to use one shard per test. default: 1
max-test-shards: 50

### Local Results Directory
## Local folder to store the test result. Folder is DELETED before each run to ensure only artifacts from the new run are saved.
local-result-dir: flank

### Additional App/Test APKS
## Include additional app/test apk pairs in the run. Apks are unique by just filename and not by path!
## If app is omitted, then the top level app is used for that pair.
## You can overwrite global config per each test pair.
## Currently supported options are: max-test-shards, test-targets, client-details, environment-variables, device
additional-app-test-apks:
  - test: ./StyleGuide/build/outputs/apk/androidTest/debug/StyleGuide-debug-androidTest.apk
    environment-variables:
      clearPackageData: true
      theme: lightMode

### Enable output report with set type
## Saves output results as parsable file and optionally upload it to Gcloud. Possible values are [none, json].
## Default: none
output-report: json

Expected behavior
Flank would execute two runs on the test matrix, once with the theme passed in being lightMode and once with the theme passed in being darkMode

Details (please complete the following information):
Using the Bitrise Flank step, which uses latest flank version

Additional context

Add any other context about the problem here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants