fix: Uploading performance metrics for multiple matrices

[piotradamczyk5]

Fixes #1322

Test Plan

How do we know the code works?

1. Run flank on Android physical device. For example with configuration entry:

gcloud:
...
  device:
    - model: OnePlus6T
      version: 28
...

2. Run Flank with multiple matrices. Example with using option

...
flank:
  num-test-runs: 2
...

3. Check that performanceMetrics.json is uploaded for each matrix

Checklist

• Unit tested

[codecov-io]

Codecov Report

Merging #1329 (8072aaa) into master (074cf0e) will decrease coverage by 0.04%.
The diff coverage is 78.78%.

@@             Coverage Diff              @@
##             master    #1329      +/-   ##
============================================
- Coverage     79.13%   79.09%   -0.05%     
+ Complexity      738      702      -36     
============================================
  Files           237      244       +7     
  Lines          4640     4683      +43     
  Branches        808      824      +16     
============================================
+ Hits           3672     3704      +32     
- Misses          550      551       +1     
- Partials        418      428      +10     

[pawelpasterz]

https://github.com/Flank/flank/pull/1329/checks?check_run_id=1414353358

[Sloox]

Fix the tests and we all good!

[github-actions]

Timestamp: 2020-11-19 13:56:12
Buildscan url for ubuntu-workflow run 372369667
https://gradle.com/s/mr4dgjlgofh42

[adamfilipow92]

https://github.com/Flank/flank/pull/1329/checks?check_run_id=1414353358

Tests fixed

[pawelpasterz]

I followed the test plan but can't find any performanceMetrics.json
Here is my config

gcloud:
  app: ~/app-debug.apk
  test: ~/app-single-success-debug-androidTest.apk
flank:
    num-test-runs: 3
    output-style: single

Let me know if I am doing something wrong

[piotradamczyk5]

I followed the test plan but can't find any performanceMetrics.json
Here is my config

gcloud:
  app: ~/app-debug.apk
  test: ~/app-single-success-debug-androidTest.apk
flank:
    num-test-runs: 3
    output-style: single

Let me know if I am doing something wrong

@pawelpasterz you must run on physical device. I have updated test plan

[pawelpasterz]

I followed the test plan but can't find any performanceMetrics.json
Here is my config

gcloud:
  app: ~/app-debug.apk
  test: ~/app-single-success-debug-androidTest.apk
flank:
    num-test-runs: 3
    output-style: single

Let me know if I am doing something wrong

@pawelpasterz you must run on physical device. I have updated test plan

Thanks!

[pawelpasterz]

We have the option disable-results-upload. If set true we should not upload any results to the bucket. But I can see performanceMetrics.json are uploaded anyway

With disable-results-upload set to false output is rather ugly

I see some items we need to decide/discuss/fix

1. Are performanceMetrics.json considered to be 'the same' results as the rest of flank's results?
2. Let's align/beautify the output
3. Let's decide what we want to print actually. Currently, we just print the message Uploading [file name] do we want to stick with that? Or we can print a gcs link but on a matrix level? So every matrix will have it's own. IMO we should decide on one convention and not mix them

Of course, I can be wrong. I'd like to get your opinion.

cc
@bootstraponline @jan-gogo

[piotradamczyk5]

We have the option disable-results-upload. If set true we should not upload any results to the bucket. But I can see performanceMetrics.json are uploaded anyway

With disable-results-upload set to false output is rather ugly

I see some items we need to decide/discuss/fix

1. Are performanceMetrics.json considered to be 'the same' results as the rest of flank's results?
2. Let's align/beautify the output
3. Let's decide what we want to print actually. Currently, we just print the message Uploading [file name] do we want to stick with that? Or we can print a gcs link but on a matrix level? So every matrix will have it's own. IMO we should decide on one convention and not mix them

Of course, I can be wrong. I'd like to get your opinion.

cc
@bootstraponline @jan-gogo

I will disable uploading if disable-results-upload present
In the case of output, I agreed that I should not add a confirmation message (with the link), because it brokes our convention, I will delete it.
I will create the task to research and think about good output printing for uploading files

I fixed all reported issues

[adamfilipow92]

• Tested without disable-results-upload works and output looks good
• Tested with disable-results-upload works as expected

[adamfilipow92]

Tested and works! 👍