Sharding #621

bootstraponline · 2019-12-01T21:20:03Z

No description provided.

codecov-io · 2019-12-01T21:24:25Z

Codecov Report

Merging #621 into master will decrease coverage by 0.08%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##             master     #621      +/-   ##
============================================
- Coverage     76.31%   76.22%   -0.09%     
  Complexity      610      610              
============================================
  Files            81       81              
  Lines          2297     2297              
  Branches        327      327              
============================================
- Hits           1753     1751       -2     
  Misses          324      324              
- Partials        220      222       +2

runningcode · 2019-12-20T11:10:05Z

test_runner/src/main/kotlin/ftl/shard/Shard.kt

@@ -136,7 +136,8 @@ object Shard {
        testcases.sortByDescending { it.time }

        testcases.forEach { testMethod ->
-            val shard = shards.first()
+            // num_shards must be > 1, and <= 50


just for clarification. num shards must be less than 50 or number of tests per shard must be less than 50?

--num-uniform-shards=int

The number of shards should be less than the total number of test cases. The number of shards specified must be >= 1 and <= 50.

https://cloud.google.com/sdk/gcloud/reference/alpha/firebase/test/android/run

I think (hopefully) there's no limit on the number of tests per shard.

I think this is >= 1, and <= 50

jan-goral · 2020-02-19T16:08:37Z

test_runner/src/main/kotlin/ftl/shard/Shard.kt

@@ -136,7 +136,8 @@ object Shard {
        testcases.sortByDescending { it.time }

        testcases.forEach { testMethod ->
-            val shard = shards.first()
+            // num_shards must be > 1, and <= 50
+            val shard = shards.first { it.testMethods.size + args.testTargetsAlwaysRun.size < 50 }


There is a problem here. According to doc the number of shards should be less than 50, but this condition prevents from putting more than 50 test methods per shard, so probably it's not what we want.

jan-goral · 2020-02-21T13:22:53Z

Hi, I have added some additional apks for specific test cases. Currently we are checking if new sharding is working properly on android, also with num-flaky-test-attempts flag on. And if everything will be ok I guess we can merge that.
Additionally the output directory structure for shards will change slightly, because now gc API are handling shards and can returns correct directories in output.

github-actions · 2020-02-24T04:42:56Z

CLA Assistant Lite All Contributors have signed the CLA.

bootstraponline · 2020-02-24T05:06:08Z

I have read the CLA Document and I hereby sign the CLA

jan-goral · 2020-02-24T17:20:47Z

I have read the CLA Document and I hereby sign the CLA

pawelpasterz · 2020-02-24T18:04:50Z

I have read the CLA Document and I hereby sign the CLA

pawelpasterz · 2020-02-24T18:54:11Z

I have read the CLA Document and I hereby sign the CLA

bootstraponline · 2020-02-25T01:40:53Z

@pawelpasterz hey, the code looks great. Let's move all the apks into https://github.com/Flank/test_artifacts

We don't want large binary files in the repo increasing the repo size. To solve for this, I publish the artifact bundle saparately which is downloaded as part of the test execution process.

For now, delete the apks and I can merge the PR. As a follow up, we can add the APKs to the test_artifacts repo and I'll publish a new test bundle.

jan-goral · 2020-02-25T14:51:27Z

@bootstraponline @pawelpasterz I have removed test apks but there is still one bug during test results fetch related to output directory, so we cannot merge until it will be resolved. Also i have 404 error when i tried to upload test apks in to https://github.com/Flank/test_artifacts. I need some time to investigate it.

bootstraponline · 2020-02-25T15:59:43Z

@jan-gogo I added you to Flank/test_artifacts with write access. Let me know when this PR is ready to merge.

pawelpasterz

LGTM
@bootstraponline
I've tested solution with various options, as much as I've been able to to verify -- works very well

bootstraponline · 2020-02-26T19:09:17Z

@pawelpasterz Thank you so much! Awesome work. I'll merge this.

andreschab90 previously approved these changes Dec 6, 2019

View reviewed changes

bootstraponline dismissed andreschab90’s stale review via f347d41 December 19, 2019 22:57

runningcode reviewed Dec 20, 2019

View reviewed changes

pawelpasterz force-pushed the sharding branch 3 times, most recently from 5e98257 to 4827fe0 Compare February 18, 2020 06:03

jan-goral reviewed Feb 19, 2020

View reviewed changes

bootstraponline force-pushed the sharding branch from 577e2bc to c4fc2d8 Compare February 24, 2020 04:42

bootstraponline mentioned this pull request Feb 24, 2020

Committer check failing contributor-assistant/github-action#7

Closed

bootstraponline closed this Feb 24, 2020

bootstraponline reopened this Feb 24, 2020

github-actions bot locked and limited conversation to collaborators Feb 24, 2020

jan-goral force-pushed the sharding branch from c4fc2d8 to 316d248 Compare February 24, 2020 17:17

pawelpasterz force-pushed the sharding branch from 316d248 to a74fc56 Compare February 24, 2020 17:57

pawelpasterz force-pushed the sharding branch from a74fc56 to 3be5841 Compare February 24, 2020 18:52

bootstraponline and others added 5 commits February 26, 2020 06:09

Update testing_v1.json with ManualSharding support

c969850

Regenerate test api

744d1ed

WIP manual sharding

425a989

Fix shard guard logic

22e1e1e

Fix wildcards import

eea10a8

Pawel Pasterz and others added 10 commits February 26, 2020 06:13

Update client

faa1dbd

Restrict shard count instead of test methods count per shard.

9da54bc

Fix num_shard comment

f35b27a

Generate different test apks automatically

c240237

Fix GCS output directory path for android manual sharding

93929ac

Fix wrong path's in test yml files

1b96aa9

Change depricated flag 'disableSharding' => 'disable-sharding'

cb5a9bf

Fix bug where flag 'disable-sharding' causes bad request to TL

e0e85ea

Remove test artifacts

d23d910

Fix ObjPath parsing for new Android manual sharding

eefb5f2

jan-goral force-pushed the sharding branch from dd28dc1 to eefb5f2 Compare February 26, 2020 05:35

jan-goral requested a review from pawelpasterz February 26, 2020 05:38

Fix shard name Regex

c2c7363

pawelpasterz approved these changes Feb 26, 2020

View reviewed changes

bootstraponline merged commit 9871d9e into master Feb 26, 2020

bootstraponline deleted the sharding branch February 26, 2020 19:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sharding #621

Sharding #621

bootstraponline commented Dec 1, 2019

codecov-io commented Dec 1, 2019

runningcode Dec 20, 2019

bootstraponline Dec 20, 2019 •

edited

Loading

bootstraponline Feb 11, 2020

jan-goral Feb 19, 2020 •

edited

Loading

jan-goral commented Feb 21, 2020 •

edited

Loading

github-actions bot commented Feb 24, 2020 •

edited

Loading

bootstraponline commented Feb 24, 2020

jan-goral commented Feb 24, 2020

pawelpasterz commented Feb 24, 2020

pawelpasterz commented Feb 24, 2020

bootstraponline commented Feb 25, 2020

jan-goral commented Feb 25, 2020

bootstraponline commented Feb 25, 2020

pawelpasterz left a comment

bootstraponline commented Feb 26, 2020

Sharding #621

Sharding #621

Conversation

bootstraponline commented Dec 1, 2019

codecov-io commented Dec 1, 2019

Codecov Report

runningcode Dec 20, 2019

Choose a reason for hiding this comment

bootstraponline Dec 20, 2019 • edited Loading

Choose a reason for hiding this comment

bootstraponline Feb 11, 2020

Choose a reason for hiding this comment

jan-goral Feb 19, 2020 • edited Loading

Choose a reason for hiding this comment

jan-goral commented Feb 21, 2020 • edited Loading

github-actions bot commented Feb 24, 2020 • edited Loading

bootstraponline commented Feb 24, 2020

jan-goral commented Feb 24, 2020

pawelpasterz commented Feb 24, 2020

pawelpasterz commented Feb 24, 2020

bootstraponline commented Feb 25, 2020

jan-goral commented Feb 25, 2020

bootstraponline commented Feb 25, 2020

pawelpasterz left a comment

Choose a reason for hiding this comment

bootstraponline commented Feb 26, 2020

bootstraponline Dec 20, 2019 •

edited

Loading

jan-goral Feb 19, 2020 •

edited

Loading

jan-goral commented Feb 21, 2020 •

edited

Loading

github-actions bot commented Feb 24, 2020 •

edited

Loading