Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parameterized tests support #2052

Closed
jan-goral opened this issue Jun 29, 2021 · 1 comment · Fixed by #2062 or #2056
Closed

Parameterized tests support #2052

jan-goral opened this issue Jun 29, 2021 · 1 comment · Fixed by #2062 or #2056

Comments

@jan-goral
Copy link
Contributor

Motivation

Flank-Corellium doesn't support various types of parameterized tests.
So, the following features may not works or works partially:

  • Standard parameterized tests annotated with @RunWith(Parameterized::class) can be shared and executed but test cases duration from previous runs for smart sharding cannot be resolved properly due to modified parameterized test cases names in test results.
  • Test cases from classes annotated with @RunWith(JUnitParamsRunner::class) are required to be run as a whole class without splitting on to single test cases which currently is not supported.

Goal

Have implemented full support for currently known types of parameterized tests

Design

  • Tests from classes annotated @RunWith(Parameterized::class) :
    • Are running the same as normal test cases.
    • Calculating test cases duration requires a merge of parameterized runs because one parameterized test case typically is producing many executions for different parameters. The parameterized test case duration for sharding is a summary of this test case runs with different parameters.
  • Tests from classes annotated @RunWith(JUnitParamsRunner::class):
    • Cannot be split and must be run as a whole class.
    • Needs to be recognized by class instead of method.
    • Methods of this type need to be filtered out before sharding, only classes must be taken into account.

Dependencies

Affected modules:

  • :tool:apks - Expand Apk.ParseTestCases implementation for parsing classes annotated by @RunWith(JUnitParamsRunner::class) as test cases.
  • :tool:junit - Add additional function for merging parameterized test cases with durations.
  • :corellium:domain - Expand RunTestCorelliumAndroid.Context.loadPreviousDurations task for merging parameterized tests duration before running List<JUnit.TestResult>.calculateTestCaseDurations
@jan-goral
Copy link
Contributor Author

The part related to not splitting parameterized classes into test cases will be may by fixed by #2056.

@jan-goral jan-goral linked a pull request Jul 1, 2021 that will close this issue
3 tasks
@mergify mergify bot closed this as completed in #2056 Jul 6, 2021
@jan-goral jan-goral reopened this Jul 9, 2021
@mergify mergify bot closed this as completed in #2062 Jul 9, 2021
mergify bot pushed a commit that referenced this issue Jul 9, 2021
Fixes #2052 

## Changes

* Adds to module `:tool:junit` function for merging test methods into test-class with accumulated duration.
* Merges test methods for given class names while loading previous durations.

## Test Plan
> How do we know the code works?

Build flank:
```shell
. .env
flankScripts assemble flank -d
```
Run the following command twice:
```shell
flank corellium test android run -c="./test_configs/flank-corellium.yml"
```
The second run should calculate durations for parameterized classes.
Check generated `android-shards.json` each parameterized class should have a duration different than the default (120).

## Checklist

- [x] Documented
- [x] Unit tested
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant