You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Flank-Corellium doesn't support various types of parameterized tests.
So, the following features may not works or works partially:
Standard parameterized tests annotated with @RunWith(Parameterized::class) can be shared and executed but test cases duration from previous runs for smart sharding cannot be resolved properly due to modified parameterized test cases names in test results.
Test cases from classes annotated with @RunWith(JUnitParamsRunner::class) are required to be run as a whole class without splitting on to single test cases which currently is not supported.
Goal
Have implemented full support for currently known types of parameterized tests
Design
Tests from classes annotated @RunWith(Parameterized::class) :
Are running the same as normal test cases.
Calculating test cases duration requires a merge of parameterized runs because one parameterized test case typically is producing many executions for different parameters. The parameterized test case duration for sharding is a summary of this test case runs with different parameters.
Tests from classes annotated @RunWith(JUnitParamsRunner::class):
Cannot be split and must be run as a whole class.
Needs to be recognized by class instead of method.
Methods of this type need to be filtered out before sharding, only classes must be taken into account.
Dependencies
Affected modules:
:tool:apks - Expand Apk.ParseTestCases implementation for parsing classes annotated by @RunWith(JUnitParamsRunner::class) as test cases.
:tool:junit - Add additional function for merging parameterized test cases with durations.
:corellium:domain - Expand RunTestCorelliumAndroid.Context.loadPreviousDurations task for merging parameterized tests duration before running List<JUnit.TestResult>.calculateTestCaseDurations
The text was updated successfully, but these errors were encountered:
Fixes#2052
## Changes
* Adds to module `:tool:junit` function for merging test methods into test-class with accumulated duration.
* Merges test methods for given class names while loading previous durations.
## Test Plan
> How do we know the code works?
Build flank:
```shell
. .env
flankScripts assemble flank -d
```
Run the following command twice:
```shell
flank corellium test android run -c="./test_configs/flank-corellium.yml"
```
The second run should calculate durations for parameterized classes.
Check generated `android-shards.json` each parameterized class should have a duration different than the default (120).
## Checklist
- [x] Documented
- [x] Unit tested
Motivation
Flank-Corellium doesn't support various types of parameterized tests.
So, the following features may not works or works partially:
@RunWith(Parameterized::class)
can be shared and executed but test cases duration from previous runs for smart sharding cannot be resolved properly due to modified parameterized test cases names in test results.@RunWith(JUnitParamsRunner::class)
are required to be run as a whole class without splitting on to single test cases which currently is not supported.Goal
Have implemented full support for currently known types of parameterized tests
Design
@RunWith(Parameterized::class)
:@RunWith(JUnitParamsRunner::class)
:Dependencies
Affected modules:
:tool:apks
- ExpandApk.ParseTestCases
implementation for parsing classes annotated by@RunWith(JUnitParamsRunner::class)
as test cases.:tool:junit
- Add additional function for merging parameterized test cases with durations.:corellium:domain
- ExpandRunTestCorelliumAndroid.Context.loadPreviousDurations
task for merging parameterized tests duration before runningList<JUnit.TestResult>.calculateTestCaseDurations
The text was updated successfully, but these errors were encountered: