Skip to content

Commit

Permalink
docs: Explore test-targets-for-shards SDD
Browse files Browse the repository at this point in the history
  • Loading branch information
Piotr Adamczyk committed Feb 23, 2021
1 parent 78ec3fb commit 46e813e
Showing 1 changed file with 56 additions and 0 deletions.
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Auto discover rest of tests when using test-target-for-shards option

Currently, when using `test-target-for-shards` you need to specify all tests manually to split to shards. If you do not
do that then only listed tests are executed.
It will be good to provide an option to auto-discover other than specified tests to make `test-target-for-shards`
better and easier to use.

# References

- [Conversation on Slack](https://firebase-community.slack.com/archives/C72V6UW8M/p1611322014000600)
- [1532](https://github.com/Flank/flank/issues/1532)

# Motivation

`test-target-for-shards` is not so easy to use, this change will improve it when using Flank and give users more
flexibility and control over creating shards.

# Goals

1. New option is available on the flank configuration level
1. User could specify which packages/class/tests run on specific shards by using a new option
1. Other not specified tests are automatically added to the left shard/shards

# Design

- Add new option under Flank configuration, possible ideas are: `plan-sharding`, `tests-for-shards`.
- New option should behave the same as [`test-target-for-shards`](https://cloud.google.com/sdk/gcloud/reference/beta/firebase/test/android/run#--test-targets-for-shard).
- After that Flank should split the test into shards specified by the user and make another shard/shards with the left one.
- Flank should fast fail if `disable-sharding` is set to `true` or `max-test-shards` is lower than specified test shards
by user + 1.

# API

It is hard to plan API for that change because the codebase could be different when doing this task and also usage of this
the option will be applied to many places in code.
Please follow up `Design` section for the implementation plan.

# Results

A new option which split sharding based on user input or alternative flag (described in `Alternative Considered`) which
will change behavior of `test-target-for-shards`.

# Dependencies

There are not any dependencies that could have a real impact on this task. However, it will be best to do this after
[Flank's refactor](https://github.com/Flank/flank/issues/1317) to make it easier.

# Testing

1. Add some tests(not all) as a value to a new option
1. Verify that they are correctly split into shards and there is one more shard with rest of tests(not specified)

# Alternatives Considered

Add a new flag to Flank configuration, which overrides `test-target-for-shards` and automatically add the rest of the tests to
separate shard/shards(if max tests shards are greater than left count).

0 comments on commit 46e813e

Please sign in to comment.