Skip to content

Commit

Permalink
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: Add Flank output research docs
Browse files Browse the repository at this point in the history
Piotr Adamczyk authored and mergify-bot committed Mar 5, 2021

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.
1 parent fe72d53 commit 569c5a7
Showing 3 changed files with 575 additions and 0 deletions.
310 changes: 310 additions & 0 deletions docs/flank-output-investigation/flank_current_output.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,310 @@
# Flank output

## Android

### Version section

```bash

version: local_snapshot
revision: d5e7c76ab206373c6cdf0c1a25e2575f323e5929
session id: 7a7ceda1-658e-461f-bd37-04f2b2a90a37

```

### Args section

```bash

AndroidArgs
gcloud:
results-bucket: test-lab-v9cn46bb990nx-kz69ymd4nm9aq
results-dir: 2021-03-01_13-01-56.722225_lYrQ
record-video: false
timeout: 15m
async: false
client-details:
network-profile: null
results-history-name: null
# Android gcloud
app: /Users/adamfilipowicz/Repos/flank/test_runner/src/test/kotlin/ftl/fixtures/tmp/apk/app-debug.apk
test: /Users/adamfilipowicz/Repos/flank/test_runner/src/test/kotlin/ftl/fixtures/tmp/apk/app-single-success-debug-androidTest.apk
additional-apks:
auto-google-login: false
use-orchestrator: true
directories-to-pull:
grant-permissions: all
type: null
other-files:
scenario-numbers:
scenario-labels:
obb-files:
obb-names:
performance-metrics: false
num-uniform-shards: null
test-runner-class: null
test-targets:
robo-directives:
robo-script: null
device:
- model: NexusLowRes
version: 28
locale: en
orientation: portrait
num-flaky-test-attempts: 0
test-targets-for-shard:
fail-fast: false

flank:
max-test-shards: 1
shard-time: -1
num-test-runs: 1
smart-flank-gcs-path:
smart-flank-disable-upload: false
default-test-time: 120.0
use-average-test-time-for-new-tests: false
files-to-download:
test-targets-always-run:
disable-sharding: true
project: flank-open-source
local-result-dir: results
full-junit-result: false
# Android Flank Yml
keep-file-path: false
additional-app-test-apks:
run-timeout: -1
legacy-junit-result: false
ignore-failed-tests: false
output-style: single
disable-results-upload: false
default-class-test-time: 240.0
disable-usage-statistics: false
output-report: none

```

### Run tests section

```bash

RunTests
Saved 1 shards to /Users/adamfilipowicz/Repos/flank/results/2021-03-01_13-01-56.722225_lYrQ/android_shards.json
Uploading [android_shards.json] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2021-03-01_13-01-56.722225_lYrQ/...
Uploading [app-debug.apk] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2021-03-01_13-01-56.722225_lYrQ/...
Uploading [app-single-success-debug-androidTest.apk] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2021-03-01_13-01-56.722225_lYrQ/...

1 test / 1 shard

Uploading [session_id.txt] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2021-03-01_13-01-56.722225_lYrQ/...
1 matrix ids created in 0m 5s
Raw results will be stored in your GCS bucket at [https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2021-03-01_13-01-56.722225_lYrQ]


```

### Matrices webLink section

```bash

matrix-1kozhsv2imkru https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.da0c237aaa33732/matrices/8233077922466140188/executions/bs.d3f60304f671ce86

```

### Test status

```bash

3m 11s Test executions status: FINISHED:1
3m 11s matrix-1kozhsv2imkru FINISHED

```

### Cost report section

```bash

CostReport
Virtual devices
$0.02 for 1m

Uploading [CostReport.txt] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2021-03-01_13-01-56.722225_lYrQ/...

```

### Results section

```bash

MatrixResultsReport
1 / 1 (100.00%)
┌─────────┬──────────────────────┬────────────────────────────┬────────────────────────────────┐
│ OUTCOME │ MATRIX ID │ TEST AXIS VALUE │ TEST DETAILS │
├─────────┼──────────────────────┼────────────────────────────┼────────────────────────────────┤
│ success │ matrix-1kozhsv2imkru │ NexusLowRes-28-en-portrait │ 1 test cases passed, 1 skipped │
└─────────┴──────────────────────┴────────────────────────────┴────────────────────────────────┘
Uploading [MatrixResultsReport.txt] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2021-03-01_13-01-56.722225_lYrQ/...
Uploading [JUnitReport.xml] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2021-03-01_13-01-56.722225_lYrQ/...
Uploading [matrix_ids.json] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2021-03-01_13-01-56.722225_lYrQ/...

FetchArtifacts
Updating matrix file

Matrices webLink
matrix-1kozhsv2imkru https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.da0c237aaa33732/matrices/8233077922466140188/executions/bs.d3f60304f671ce86

```

## Ios

### Version section

```bash

version: local_snapshot
revision: d5e7c76ab206373c6cdf0c1a25e2575f323e5929
session id: 7a7ceda1-658e-461f-bd37-04f2b2a90a37

```

### Args section

```bash

IosArgs
gcloud:
results-bucket: test-lab-v9cn46bb990nx-kz69ymd4nm9aq
results-dir: test_dir
record-video: false
timeout: 15m
async: false
client-details:
network-profile: null
results-history-name: null
# iOS gcloud
test: /Users/adamfilipowicz/Repos/flank/test_runner/src/test/kotlin/ftl/fixtures/tmp/ios/EarlGreyExample/EarlGreyExample.zip
xctestrun-file: /Users/adamfilipowicz/Repos/flank/test_runner/src/test/kotlin/ftl/fixtures/tmp/ios/EarlGreyExample/EarlGreyExampleSwiftTests.xctestrun
xcode-version: null
device:
- model: iphone8
version: 13.6
locale: en
orientation: portrait
num-flaky-test-attempts: 0
directories-to-pull:
other-files:
additional-ipas:
scenario-numbers:
type: xctest
app:
test-special-entitlements: false
fail-fast: false

flank:
max-test-shards: 1
shard-time: -1
num-test-runs: 1
smart-flank-gcs-path:
smart-flank-disable-upload: false
default-test-time: 120.0
use-average-test-time-for-new-tests: false
test-targets-always-run:
files-to-download:
keep-file-path: false
full-junit-result: false
# iOS flank
test-targets:
disable-sharding: false
project: flank-open-source
local-result-dir: results
run-timeout: -1
ignore-failed-tests: false
output-style: single
disable-results-upload: false
default-class-test-time: 240.0
disable-usage-statistics: false
only-test-configuration:
skip-test-configuration:
output-report: none
WARNING: Google cloud storage result directory should be unique, otherwise results from multiple test matrices will be overwritten or intermingled

```

### Run tests section

```bash

RunTests
Found xctest: /Users/adamfilipowicz/Repos/flank/test_runner/src/test/kotlin/ftl/fixtures/tmp/ios/EarlGreyExample/Debug-iphoneos/EarlGreyExampleSwift.app/PlugIns/EarlGreyExampleSwiftTests.xctest
isMacOS = true (mac os x)
nm -U "/Users/adamfilipowicz/Repos/flank/test_runner/src/test/kotlin/ftl/fixtures/tmp/ios/EarlGreyExample/Debug-iphoneos/EarlGreyExampleSwift.app/PlugIns/EarlGreyExampleSwiftTests.xctest/EarlGreyExampleSwiftTests"
nm -gU "/Users/adamfilipowicz/Repos/flank/test_runner/src/test/kotlin/ftl/fixtures/tmp/ios/EarlGreyExample/Debug-iphoneos/EarlGreyExampleSwift.app/PlugIns/EarlGreyExampleSwiftTests.xctest/EarlGreyExampleSwiftTests" | xargs -s 262144 xcrun swift-demangle

Smart Flank cache hit: 0% (0 / 17)
Shard times: 2040s

Saved 1 shards to /Users/adamfilipowicz/Repos/flank/results/test_dir/ios_shards.json
Uploading [ios_shards.json] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/test_dir/...

17 tests / 1 shard

Uploading [EarlGreyExample.zip] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/test_dir/.....
Uploading [EarlGreyExampleSwiftTests_shard_0.xctestrun] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/test_dir/shard_0/...
Uploading [session_id.txt] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/test_dir/...
1 matrix ids created in 0m 29s
Raw results will be stored in your GCS bucket at [https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/test_dir]

```

### Matrices webLink section

```bash

Matrices webLink
matrix-3d331uzs97mlt https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.a3b607c9bb6d0088/matrices/5523514684002242128/executions/bs.70a259d113387c0c

```

### Test status

```bash

3m 11s Test executions status: FINISHED:1
3m 11s matrix-1kozhsv2imkru FINISHED

```

### Cost report section

```bash

CostReport
Physical devices
$0.08 for 1m

Uploading [CostReport.txt] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/test_dir/...


```

### Results section

```bash

MatrixResultsReport
1 / 1 (100.00%)
┌─────────┬──────────────────────┬──────────────────────────┬──────────────────────┐
│ OUTCOME │ MATRIX ID │ TEST AXIS VALUE │ TEST DETAILS │
├─────────┼──────────────────────┼──────────────────────────┼──────────────────────┤
│ success │ matrix-3d331uzs97mlt │ iphone8-13.6-en-portrait │ 17 test cases passed │
└─────────┴──────────────────────┴──────────────────────────┴──────────────────────┘
Uploading [MatrixResultsReport.txt] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/test_dir/...
Uploading [JUnitReport.xml] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/test_dir/...
Uploading [matrix_ids.json] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/test_dir/...

FetchArtifacts
Updating matrix file

Matrices webLink
matrix-3d331uzs97mlt https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.a3b607c9bb6d0088/matrices/5523514684002242128/executions/bs.70a259d113387c0c

```
101 changes: 101 additions & 0 deletions docs/flank-output-investigation/flank_output_propositon.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,101 @@
# Flank output proposition

## Android

### Version section

```bash

version: local_snapshot
revision: d5e7c76ab206373c6cdf0c1a25e2575f323e5929
session id: 7a7ceda1-658e-461f-bd37-04f2b2a90a37

```

### Args section

```bash
<Diff of arguments>

```

### Dump shards

```bash
[Dump shards]
Saved 1 shards to /Users/adamfilipowicz/Repos/flank/results/2021-03-01_13-01-56.722225_lYrQ/android_shards.json
Uploading [android_shards.json] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2021-03-01_13-01-56.722225_lYrQ/...
```

### Uploading files

```
[Uploading files]
Uploading [app-debug.apk] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2021-03-01_13-01-56.722225_lYrQ/...
Uploading [app-single-success-debug-androidTest.apk] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2021-03-01_13-01-56.722225_lYrQ/...
Uploading [session_id.txt] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2021-03-01_13-01-56.722225_lYrQ/...
```

### Matrix info

```
[Matrix info]
Found 1 test / 1 shard
1 matrix ids created in 0m 5s
matrix-1kozhsv2imkru https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.da0c237aaa33732/matrices/8233077922466140188/executions/bs.d3f60304f671ce86
```



### Storage information

```bash

Raw results will be stored in your GCS bucket at [https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2021-03-01_13-01-56.722225_lYrQ]

```

### Test status

```bash
[Execution statuses]
3m 11s Test executions status: FINISHED:1
3m 11s matrix-1kozhsv2imkru FINISHED

```

### Cost report section

```bash
[CostReport]
Virtual devices
$0.02 for 1m

Uploading [CostReport.txt] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2021-03-01_13-01-56.722225_lYrQ/...

```

### Results section

```bash

[MatrixResultsReport]
1 / 1 (100.00%)

┌─────────┬──────────────────────┬────────────────────────────┬────────────────────────────────┐
│ OUTCOME │ MATRIX ID │ TEST AXIS VALUE │ TEST DETAILS │
├─────────┼──────────────────────┼────────────────────────────┼────────────────────────────────┤
│ success │ matrix-1kozhsv2imkru │ NexusLowRes-28-en-portrait │ 1 test cases passed, 1 skipped │
└─────────┴──────────────────────┴────────────────────────────┴────────────────────────────────┘

Uploading [MatrixResultsReport.txt] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2021-03-01_13-01-56.722225_lYrQ/...
Uploading [JUnitReport.xml] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2021-03-01_13-01-56.722225_lYrQ/...
Uploading [matrix_ids.json] to https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2021-03-01_13-01-56.722225_lYrQ/...

matrix-1kozhsv2imkru https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.da0c237aaa33732/matrices/8233077922466140188/executions/bs.d3f60304f671ce86

```

164 changes: 164 additions & 0 deletions docs/flank-output-investigation/gcloud_current_output.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,164 @@
# Gcloud output

## Android

### Welcome section

```bash
Have questions, feedback, or issues? Get support by visiting:
https://firebase.google.com/support/
```


### Upload info

```bash
Uploading [../test_projects/android/apks/app-debug.apk] to Firebase Test Lab...
Uploading [../test_projects/android/apks/app-debug-androidTest.apk] to Firebase Test Lab...
```

### Storage information

```bash
Raw results will be stored in your GCS bucket at [https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2021-02-28_19:08:02.351035_LadZ/]
```

### Info before running test

```
Test [matrix-1uohkjt5rsr28] has been created in the Google Cloud.
Firebase Test Lab will execute your instrumentation test on 1 device(s).
Creating individual test executions...done.
```

### Test status

```bash
Test results will be streamed to [https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.bf178d418be9d33e/matrices/4823249401837879607].
19:08:18 Test is Pending
19:08:46 Starting attempt 1.
19:08:46 Started logcat recording.
19:08:46 Test is Running
19:08:53 Started crash monitoring.
19:08:53 Preparing device.
19:09:00 Logging in to Google account on device.
19:09:06 Installing apps.
19:09:06 Retrieving Pre-Test Package Stats information from the device.
19:09:06 Retrieving Performance Environment information from the device.
19:09:06 Started crash detection.
19:09:06 Started Out of memory detection
19:09:06 Started performance monitoring.
19:09:06 Started video recording.
19:09:06 Starting instrumentation test.
19:09:06 Completed instrumentation test.
19:09:13 Stopped performance monitoring.
19:09:13 Retrieving Post-test Package Stats information from the device.
19:09:13 Logging out of Google account on device.
19:09:20 Stopped crash monitoring.
19:09:20 Stopped logcat recording.
19:09:40 Done. Test time = 1 (secs)
19:09:40 Starting results processing. Attempt: 1
19:09:47 Completed results processing. Time taken = 4 (secs)
19:09:47 Test is Finished
```

### Completion info

```
Instrumentation testing complete.
```



### Results

```bash
More details are available at [https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.bf178d418be9d33e/matrices/4823249401837879607].
┌─────────┬────────────────────────┬─────────────────────┐
│ OUTCOME │ TEST_AXIS_VALUE │ TEST_DETAILS │
├─────────┼────────────────────────┼─────────────────────┤
│ Passed │ walleye-27-en-portrait │ 1 test cases passed │
└─────────┴────────────────────────┴─────────────────────┘
```



### Survey at the end

```
To take a quick anonymous survey, run:
$ gcloud survey
```



## iOS

### Welcome section

```bash
Have questions, feedback, or issues? Get support by emailing:
ftl-ios-feedback@google.com
```


### Upload info

```bash
Uploading [./src/test/kotlin/ftl/fixtures/tmp/ios/EarlGreyExample/EarlGreyExample.zip] to Firebase Test Lab...
Uploading [./src/test/kotlin/ftl/fixtures/tmp/ios/EarlGreyExample/EarlGreyExampleSwiftTests.xctestrun] to Firebase Test Lab...
Raw results will be stored in your GCS bucket at [https://console.developers.google.com/storage/browser/test-lab-v9cn46bb990nx-kz69ymd4nm9aq/2021-02-28_19:15:20.392870_nuvf/]
```

### Storage information

```bash
Test [matrix-2k6ldtp789lvy] has been created in the Google Cloud.
Firebase Test Lab will execute your xctest test on 1 device(s).
Creating individual test executions...done.
```

### Info before running test

```
Test [matrix-2k6ldtp789lvy] has been created in the Google Cloud.
Firebase Test Lab will execute your xctest test on 1 device(s).
Creating individual test executions...done.
```

### Test status

```bash
Test results will be streamed to [https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.a3b607c9bb6d0088/matrices/7622498233210161623].
19:15:49 Test is Pending
19:16:44 Starting attempt 1.
19:16:44 Checking Internet connection...
19:16:44 Test is Running
19:18:22 Internet connection stable!
19:18:41 Started device logs task
19:20:20 Stopped device logs task
19:20:39 Done. Test time = 62 (secs)
19:20:39 Starting results processing. Attempt: 1
19:20:45 Completed results processing. Time taken = 5 (secs)
19:20:45 Test is Finished
```

### Completion info

```
Xctest testing complete.
```



### Results

```bash
More details are available at [https://console.firebase.google.com/project/flank-open-source/testlab/histories/bh.a3b607c9bb6d0088/matrices/7622498233210161623].
┌─────────┬──────────────────────────┬──────────────────────┐
│ OUTCOME │ TEST_AXIS_VALUE │ TEST_DETAILS │
├─────────┼──────────────────────────┼──────────────────────┤
│ Passed │ iphone8-11.2-en-portrait │ 17 test cases passed │
└─────────┴──────────────────────────┴──────────────────────┘
```

0 comments on commit 569c5a7

Please sign in to comment.