test: Optimize quota usage in IT #1825
Conversation
pawelpasterz
requested review from
Sloox,
zuziaka,
jan-goral,
adamfilipow92 and
piotradamczyk5
|
CLA Assistant Lite bot All contributors have signed the CLA ✍️ ✅ |
|
Title does not follow the guidelines of Conventional Commits.
|
|
Timestamp: 2021-04-22 12:41:18 |
pawelpasterz
changed the title
integration_tests/build.gradle.kts
Outdated
| useJUnit { | ||
| includeCategories = setOf("integration.config.AndroidTest") | ||
| excludeCategories = setOf("integration.config.IosTest") | ||
| } | ||
| testLogging { | ||
| events("skipped", "failed") | ||
| exceptionFormat = TestExceptionFormat.FULL | ||
| } | ||
| maxParallelForks = Runtime.getRuntime().availableProcessors() / 2 |
There was a problem hiding this comment.
Is this necessary? Lets remove this and see if it speeds it up.
There was a problem hiding this comment.
i.e.:
| maxParallelForks = Runtime.getRuntime().availableProcessors() / 2 | |
| maxParallelForks = Runtime.getRuntime().availableProcessors() |
There was a problem hiding this comment.
Why not, let's give it a try 👍🏻
|
Title does not follow the guidelines of Conventional Commits.
|
|
As a follow-up task we can think of implementing additional comment triggers for each category (i.e.: |
|
Integration tests were triggered at 2021-04-21 08:26:16, you can track progress here |
Damn 😅 |
PR introduces a category approach in IT. Tests are divided into
AndroidTestandIosTestcategory. It can be achieved with annotating tests with either@Category(AndroidTest::class)or@Category(IosTest::class)Changes:
There are slightly different rules applied to each category.
AndroidTestcategory:IosTestcategory:AndroidTestcategoryFor each category 2 new gradle tasks were created
instrumentationTestsAnroidandinstrumentationTestsIos-- each runs corresponding category.instrumentationTeststask was refactored and now it depends on android and ios tasks, forcing android category tests to run before iOSGreat, but what does it give us actually?
Flank project keeps facing exceeded quota problem, mainly due to iOS tests that use physical devices.
With proposed changes, flank first runs android tests. If 'common' logic (meaning, shared by both android and ios) tests will fail, only virtual devices will be used. Since flank has like 2-3x more android tests we keep parallel settings to make it faster.
Once android category is done, ios starts. A combination of sequential run and
fail-fast: truewill terminate the test run with a minimal number of devices used to detect a problem.To sum up:
➕ test coverage is the same
➕ we burn less quota if a problem is found
➖ full IT run takes more time, locally, on my machine its
16m 40sold vs19m 12snew -- so it's about +15% longer time run