From 63c1c459f1890200b612c54557426fc4098c7c80 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Janek=20G=C3=B3ral?= Date: Fri, 28 May 2021 13:48:37 +0200 Subject: [PATCH 1/5] Add user guide for Corellium MVP --- .../cli/RunTestCorelliumAndroidCommand.kt | 4 +- docs/flank_corellium.md | 131 ++++++++++++++++++ mkdocs.yml | 1 + 3 files changed, 134 insertions(+), 2 deletions(-) create mode 100644 docs/flank_corellium.md diff --git a/corellium/cli/src/main/kotlin/flank/corellium/cli/RunTestCorelliumAndroidCommand.kt b/corellium/cli/src/main/kotlin/flank/corellium/cli/RunTestCorelliumAndroidCommand.kt index d1fe9fa9ac..17ea51c5eb 100644 --- a/corellium/cli/src/main/kotlin/flank/corellium/cli/RunTestCorelliumAndroidCommand.kt +++ b/corellium/cli/src/main/kotlin/flank/corellium/cli/RunTestCorelliumAndroidCommand.kt @@ -114,9 +114,9 @@ class RunTestCorelliumAndroidCommand : private fun defaultConfig() = Config().apply { project = "Default Project" - auth = "auth.yml" + auth = "corellium_auth.yml" apks = emptyList() - maxTestShards = 3 + maxTestShards = 1 localResultsDir = null obfuscate = false } diff --git a/docs/flank_corellium.md b/docs/flank_corellium.md new file mode 100644 index 0000000000..abd79dd671 --- /dev/null +++ b/docs/flank_corellium.md @@ -0,0 +1,131 @@ +# Flank - Corellium + +Run your tests in parallel on virtual devices driven by Corellium backend. + +## Status + +The Flank - Corellium integration is at the MVP stage, so only core, most important features are available. + +[Read more](https://github.com/sparkrnyc/sparkr_general_docs/wiki/What-are-Prototype,-POC,-Alpha,-Beta-and-MVP#minimum-viable-product-mvp) about Minimum Viable Product. + +## Why Corellium + +Flank is just a client-side application that can prepare the time-efficient parallel test plan to run on several devices. It requires a third-party provider that can serve a huge amount of devices to run tests on them. +Corellium is solving this problem as follows: + +* Provides technology to virtualize mobile operating systems on servers powered by ARM cores. - Which is giving incredible ability for scaling amount of devices. +* Gives access to the bare operating system (Android or iOS). - Which is allowing to run optimized sharding algorithm for improving test execution time and reducing costs. + +## How to get + +Flank - Corellium integration is built in the `flank.jar` executable, so the latest Flank build is giving you access to the features driven on the Corellium backend. + +## How to run + +To call the root command for Corellium related features run: + +```bash +$ flank.jar corellium +``` + +## Authorization + +To allow Flank working with Corellium backend is necessary to provide authentication data. By default, Flank is recognizing `corellium_auth.yml` as an authentication file, which should look as follows: + +```yaml +host: your.corellium.backend.host +username: your_username +password: your_password +``` + +## Android test execution + +To execute android instrumented tests, run following command: + +```bash +$ flank.jar corellium test android run -c="./flank_corellium.yml" +``` + +### Config + +The example configuration file looks following: + +```yaml + +### Test apks +## The list of app and test apks. +## Each app apk must have one or more corresponding test apks. +apks: + - path: "app1.apk" + tests: + - path: "app1-test1.apk" + - path: "app2.apk" + tests: + - path: "app2-test1.apk" + - path: "app2-test2.apk" + + + ### Authorization file + ## Path to YAML file with host address and credentials. + ## default: corellium_auth.yml + # auth: auth_file.yml + + ### Project name + ## The name of Corellium project. + ## default: "Default project" + # project: "project name" + +### Max Test Shards +## test shards - the amount of groups to split the test suite into. +## default: 1 +# max-test-shards: 10000 + +### Results Directory +## The name of a local directory where test results will be stored. +## default: results/corellium/android/yyyy-MM-dd_HH-mm-ss-SSS +# local-result-dir: test-results + +### Obfuscate dump +## Replace internal test names with unique identifiers, before dumping them to "android-shards.json". +## This option is hiding sensitive details about tests. +## default: false +# obfuscate: true +``` + +### Command-line arguments + +Instead of or along with configuration file you can specify same options, in a command-line, which will override config file options. + +### Execution + +The test execution is composing small steps to fulfill operations like: + +* Calculating time-efficient sharding. +* Preparing (creating or reusing) virtual devices for test execution. +* Executing tests on prepared virtual devices. +* Generating report. + +To learn more about the execution process check the: + +* [Architecture design](https://github.com/Flank/flank/blob/master/docs/architecture.md#domain-) - which is explaining some core abstract concepts. +* [Corellium integration diagrams](https://github.com/Flank/flank/tree/master/corellium#flank---corellium) - to visualise relations and dependencies between layers. +* [Domain module design & implementation](https://github.com/Flank/flank/tree/master/corellium/domain#flank---corellium---domain) - for detailed information about business logic and low-level implementation design. + +### Results + +The command execution process is generating different types of output. + +#### Console logs + +The process during its runtime is printing detailed information about execution steps to console log. + +#### Output files + +The successful run should generate the following files: + +* JUnitReport.xml +* android_shards.json + +## iOS test execution + +Not available yet diff --git a/mkdocs.yml b/mkdocs.yml index 1e4ba86598..6720bcbd57 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -23,6 +23,7 @@ nav: - Building & updating Flank: building_updating_flank.md - Windows WSL: windows_wsl_guide.md - Cucumber: cucumber_support.md + - Corellium: flank_corellium.md - Contribution: - Pull request: pr_titles.md - Development: From e77a02d98c9f4cfe32a0e156bc838ccc5531ecad Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Janek=20G=C3=B3ral?= Date: Mon, 31 May 2021 11:32:46 +0200 Subject: [PATCH 2/5] Update --- docs/flank_corellium.md | 75 ++++++++++++++++++++++++++++++++++------- 1 file changed, 62 insertions(+), 13 deletions(-) diff --git a/docs/flank_corellium.md b/docs/flank_corellium.md index abd79dd671..4857262cc2 100644 --- a/docs/flank_corellium.md +++ b/docs/flank_corellium.md @@ -18,14 +18,46 @@ Corellium is solving this problem as follows: ## How to get +### The Latest build + Flank - Corellium integration is built in the `flank.jar` executable, so the latest Flank build is giving you access to the features driven on the Corellium backend. +### Manual compilation + +Clone the repository and go to dir: + +```shell +git clone git@github.com:Flank/flank.git +cd flank +``` + +Build flank using flank-scripts (this method will give you access to `flank.jar` through `flank` shell command): + +```shell +. .env +flankScripts assemble flank +``` + +Or build directly using gradle command: + +```shell +./gradlew :test_runner:build :test_runner:shadowJar +``` + ## How to run -To call the root command for Corellium related features run: +To call the root command for Corellium related features: + +locate your flank.jar in terminal and run: + +```bash +flank.jar corellium +``` + +or if flank was build using `. .env && flankScripts assemble flank`, just type: ```bash -$ flank.jar corellium +flank corellium ``` ## Authorization @@ -64,16 +96,15 @@ apks: - path: "app2-test1.apk" - path: "app2-test2.apk" +### Authorization file +## Path to YAML file with host address and credentials. +## default: corellium_auth.yml +# auth: auth_file.yml - ### Authorization file - ## Path to YAML file with host address and credentials. - ## default: corellium_auth.yml - # auth: auth_file.yml - - ### Project name - ## The name of Corellium project. - ## default: "Default project" - # project: "project name" +### Project name +## The name of Corellium project. +## default: "Default project" +# project: "project name" ### Max Test Shards ## test shards - the amount of groups to split the test suite into. @@ -126,6 +157,24 @@ The successful run should generate the following files: * JUnitReport.xml * android_shards.json -## iOS test execution +# Features + +* Calculating multi-module shards +* Creating or reusing instances (devices) +* Installing APKs on remote devices +* Running android tests +* Dumping shards to file +* Parsing `adb am instrument` logs +* Generating JUnit report + +# Roadmap + +* Cleaning devices after test execution. +* Reusing test cases duration for sharding. +* Flaky test detection. +* Structural logging. +* iOS support + +# Known bugs -Not available yet +* Missing `` tag for skipped test cases in JUnit report From ec5f1bc205c60b274720d0e380a3b8b17336ae46 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Jan=20G=C3=B3ral?= <60390247+jan-gogo@users.noreply.github.com> Date: Mon, 31 May 2021 11:57:15 +0200 Subject: [PATCH 3/5] Apply suggestions from code review Co-authored-by: Michael Wright --- docs/flank_corellium.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/flank_corellium.md b/docs/flank_corellium.md index 4857262cc2..60d03879a6 100644 --- a/docs/flank_corellium.md +++ b/docs/flank_corellium.md @@ -1,26 +1,26 @@ # Flank - Corellium -Run your tests in parallel on virtual devices driven by Corellium backend. +Run mobile tests in parallel on virtual devices driven by Corellium backend. ## Status -The Flank - Corellium integration is at the MVP stage, so only core, most important features are available. +The Flank - Corellium integration is at the MVP stage, so only the core and most important features are available. [Read more](https://github.com/sparkrnyc/sparkr_general_docs/wiki/What-are-Prototype,-POC,-Alpha,-Beta-and-MVP#minimum-viable-product-mvp) about Minimum Viable Product. ## Why Corellium -Flank is just a client-side application that can prepare the time-efficient parallel test plan to run on several devices. It requires a third-party provider that can serve a huge amount of devices to run tests on them. +Flank is just a client-side application that can prepare a time-efficient parallel test plan to run on several devices. It requires a third-party provider that can serve a huge amount of devices to run tests on them. Corellium is solving this problem as follows: -* Provides technology to virtualize mobile operating systems on servers powered by ARM cores. - Which is giving incredible ability for scaling amount of devices. -* Gives access to the bare operating system (Android or iOS). - Which is allowing to run optimized sharding algorithm for improving test execution time and reducing costs. +* Provides technology to virtualize mobile operating systems on servers powered by ARM cores. - This gives an incredible ability for scaling with regards to the amount of devices. +* Gives access to the bare operating system (Android or iOS). - Which allows a run optimized sharding algorithm that improves test execution time and reduces costs. ## How to get ### The Latest build -Flank - Corellium integration is built in the `flank.jar` executable, so the latest Flank build is giving you access to the features driven on the Corellium backend. +Flank - Corellium integration is built in the `flank.jar` executable, so the latest Flank build gives you access to the features driven on the Corellium backend. ### Manual compilation @@ -125,7 +125,7 @@ apks: ### Command-line arguments -Instead of or along with configuration file you can specify same options, in a command-line, which will override config file options. +These can be included alongside the configuration file. They will override or supplement the configuration file depending on the usage. ### Execution From fa879b99a7cdd93a5999f1f335d937b9d8afa67a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Janek=20G=C3=B3ral?= Date: Mon, 31 May 2021 11:32:46 +0200 Subject: [PATCH 4/5] Update --- docs/flank_corellium.md | 39 +++++++++++++++++++++++++++++++-------- 1 file changed, 31 insertions(+), 8 deletions(-) diff --git a/docs/flank_corellium.md b/docs/flank_corellium.md index 60d03879a6..8d19ee1b0b 100644 --- a/docs/flank_corellium.md +++ b/docs/flank_corellium.md @@ -44,6 +44,28 @@ Or build directly using gradle command: ./gradlew :test_runner:build :test_runner:shadowJar ``` +### Manual compilation + +Clone the repository and go to dir: + +```shell +git clone git@github.com:Flank/flank.git +cd flank +``` + +Build flank using flank-scripts (this method will give you access to `flank.jar` through `flank` shell command): + +```shell +. .env +flankScripts assemble flank +``` + +Or build directly using gradle command: + +```shell +./gradlew :test_runner:build :test_runner:shadowJar +``` + ## How to run To call the root command for Corellium related features: @@ -159,13 +181,13 @@ The successful run should generate the following files: # Features -* Calculating multi-module shards -* Creating or reusing instances (devices) -* Installing APKs on remote devices -* Running android tests -* Dumping shards to file -* Parsing `adb am instrument` logs -* Generating JUnit report +* Calculating multi-module shards. +* Creating or reusing instances (devices). +* Installing APKs on remote devices. +* Running android tests. +* Dumping shards to file. +* Parsing `adb am instrument` logs. +* Generating JUnit report. # Roadmap @@ -173,7 +195,8 @@ The successful run should generate the following files: * Reusing test cases duration for sharding. * Flaky test detection. * Structural logging. -* iOS support +* iOS support. +* and much more... # Known bugs From d35fe708998c1994962b117b1000420807dd5ad7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Janek=20G=C3=B3ral?= Date: Mon, 31 May 2021 12:14:14 +0200 Subject: [PATCH 5/5] Remove duplicated section --- docs/flank_corellium.md | 22 ---------------------- 1 file changed, 22 deletions(-) diff --git a/docs/flank_corellium.md b/docs/flank_corellium.md index 8d19ee1b0b..1386ccaa02 100644 --- a/docs/flank_corellium.md +++ b/docs/flank_corellium.md @@ -44,28 +44,6 @@ Or build directly using gradle command: ./gradlew :test_runner:build :test_runner:shadowJar ``` -### Manual compilation - -Clone the repository and go to dir: - -```shell -git clone git@github.com:Flank/flank.git -cd flank -``` - -Build flank using flank-scripts (this method will give you access to `flank.jar` through `flank` shell command): - -```shell -. .env -flankScripts assemble flank -``` - -Or build directly using gradle command: - -```shell -./gradlew :test_runner:build :test_runner:shadowJar -``` - ## How to run To call the root command for Corellium related features: