Reduce VCR recording runtime after package split #9722
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I've observed some significant recording runtime increases, especially for large size of affected tests, after VCR split.
Example PR: #9666
Before VCR split: ~20 mins for recording #9666 (comment)
After VCR split: ~2 hours for recording #9666 (comment)
What I suspect is, even though VCR split only changed how we run tests in the REPLAYING mode, it skips package compiling for all non-affected services and therefore it needs to compile them (probably multiple times) in the RECORDING mode (previously using compilation cache from the REPLAYING)
Given how we enable the parallelism in RECORDING and the lower size of it due to the limitation of Terraform log setup, it needs much more total compilation time and therefore leads to the long RECORDING runtime.
To resolve this issue, since we've known what packages are affected and needed to run tests against, we can only loop through these packages and run failed tests against them.
Testing: #9717 (comment): recording mode took ~2.5h for 12
TestAccComputeFirewall*
and after apply the change in this PR, it takes only ~10 minutes #9721 (comment).Release Note Template for Downstream PRs (will be copied)