Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

iOS: Support for filtering Test Configurations #1379

Closed
zuziaka opened this issue Dec 7, 2020 · 0 comments · Fixed by #1448
Closed

iOS: Support for filtering Test Configurations #1379

zuziaka opened this issue Dec 7, 2020 · 0 comments · Fixed by #1448

Comments

@zuziaka
Copy link
Contributor

zuziaka commented Dec 7, 2020

Currently Flank supports .xctestrun file from Test Plan #685, but user is not able to specify which test configuration Flank should run.
For example if .xctestrun file contains 2 configurations Flank will run both of them.

FTL doesn't support specifying test configuration yet, so Flank needs to handle this on his side.

Example: We have .xctestrun file containing two test configurations: pl and en:

{
  "en": [ // test configuration: en
    [
      "UITestsClass/test1_ENLocale"
    ]
  ],
  "pl": [ // test configuration: pl
    [
      "UITestsClass/test1_ENLocale"
    ]
  ]
}

Actual result:

Flank will generate two .xctestrun files for each configuration and run two separate test runs on FTL.

Expected result:

User should be able to specify which configuration Flank should run by passing argument: (proposal - to be discussed)

 -only-test-configuration: en

or

-skip-test-configuration: pl

Proposed argument names are from actual xcodebuild test-without-building command. Reference:
man xcodebuild

 -skip-test-configuration  test-configuration-name,
    -only-test-configuration test-configuration-name
       Constrain      test     configurations     in     test     actions.
       -only-test-configuration constrains a test action to  only  test  a
       specified  test  configuration  within a test plan, and exclude all
       other test configurations.  -skip-test-configuration  constrains  a
       test action to skip a specified test configuration, but include all
       other test configurations. Each test configuration name must  match
       the  name  of a configuration specified in a test plan and is case-
       sensitive. An xcodebuild command can  combine  multiple  constraint
       options,   but   -only-test-configuration   has   precedence   over
       -skip-test-configuration.
@zuziaka zuziaka self-assigned this Dec 10, 2020
@jan-goral jan-goral self-assigned this Dec 16, 2020
@zuziaka zuziaka linked a pull request Jan 4, 2021 that will close this issue
2 tasks
@mergify mergify bot closed this as completed in #1448 Jan 21, 2021
mergify bot pushed a commit that referenced this issue Jan 21, 2021
Fixes #1379 

This PR adds to new options for flank: 
- `skip-test-configuration`
- `only-test-configuration`

Flank users can filter test configurations (inside of a test plan).

## Test Plan
> How do we know the code works?

From project root:
```
. .env
cd ./test_runner/
flankScripts shell buildFlank 
flank ios run -c ./src/test/kotlin/ftl/fixtures/ios_test_plan.yml
```
additionally, edit `os_test_plan.yml` for testing different configurations.

## Checklist

- [x] Documented - updated main docs 
- [x] Unit tested
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants