-
Notifications
You must be signed in to change notification settings - Fork 118
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
docs: Explore test-targets-for-shards SDD
- Loading branch information
Piotr Adamczyk
committed
Feb 23, 2021
1 parent
78ec3fb
commit 46e813e
Showing
1 changed file
with
56 additions
and
0 deletions.
There are no files selected for viewing
56 changes: 56 additions & 0 deletions
56
docs/feature/1532-explore-test-target-for-shards-with-extra-features.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,56 @@ | ||
# Auto discover rest of tests when using test-target-for-shards option | ||
|
||
Currently, when using `test-target-for-shards` you need to specify all tests manually to split to shards. If you do not | ||
do that then only listed tests are executed. | ||
It will be good to provide an option to auto-discover other than specified tests to make `test-target-for-shards` | ||
better and easier to use. | ||
|
||
# References | ||
|
||
- [Conversation on Slack](https://firebase-community.slack.com/archives/C72V6UW8M/p1611322014000600) | ||
- [1532](https://github.com/Flank/flank/issues/1532) | ||
|
||
# Motivation | ||
|
||
`test-target-for-shards` is not so easy to use, this change will improve it when using Flank and give users more | ||
flexibility and control over creating shards. | ||
|
||
# Goals | ||
|
||
1. New option is available on the flank configuration level | ||
1. User could specify which packages/class/tests run on specific shards by using a new option | ||
1. Other not specified tests are automatically added to the left shard/shards | ||
|
||
# Design | ||
|
||
- Add new option under Flank configuration, possible ideas are: `plan-sharding`, `tests-for-shards`. | ||
- New option should behave the same as [`test-target-for-shards`](https://cloud.google.com/sdk/gcloud/reference/beta/firebase/test/android/run#--test-targets-for-shard). | ||
- After that Flank should split the test into shards specified by the user and make another shard/shards with the left one. | ||
- Flank should fast fail if `disable-sharding` is set to `true` or `max-test-shards` is lower than specified test shards | ||
by user + 1. | ||
|
||
# API | ||
|
||
It is hard to plan API for that change because the codebase could be different when doing this task and also usage of this | ||
the option will be applied to many places in code. | ||
Please follow up `Design` section for the implementation plan. | ||
|
||
# Results | ||
|
||
A new option which split sharding based on user input or alternative flag (described in `Alternative Considered`) which | ||
will change behavior of `test-target-for-shards`. | ||
|
||
# Dependencies | ||
|
||
There are not any dependencies that could have a real impact on this task. However, it will be best to do this after | ||
[Flank's refactor](https://github.com/Flank/flank/issues/1317) to make it easier. | ||
|
||
# Testing | ||
|
||
1. Add some tests(not all) as a value to a new option | ||
1. Verify that they are correctly split into shards and there is one more shard with rest of tests(not specified) | ||
|
||
# Alternatives Considered | ||
|
||
Add a new flag to Flank configuration, which overrides `test-target-for-shards` and automatically add the rest of the tests to | ||
separate shard/shards(if max tests shards are greater than left count). |