Cache all uploads and downloads to GCS #639
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Currently, when using local apk artifacts, Flank always uploads every app/test apk pair even if the app apk has already been uploaded. This is also true for pulling the old JUnitReport from GCS for smart flank where every test apk re-downloads the results. To fix this we simply cache uploads and downloads in a ConcurrentHashMap where the key is the GCS path for downloads and the filename for uploads. This change also fixes a bug where local test apks would be uploaded and then re-downloaded parse their tests. The sharding loop now loops over the local AppTestPair where it uploads them if necessary and scans for tests on the local test apk. All in all this shaves almost 4 minutes off of shard generation for a project with 20+ modules. There is also likely more work that can be done to parallelize matrix creation (uploading apks, parsing tests, creating the shard), but it is out of scope of this change.
|
This PR is awesome! Thanks for contributing. |
Make sure that when an exception is ignored, we still close our file system resources (ReadChannel/FileOutputStream). Also rename some variables and methods to better describe what they are and how they are used.
pawelpasterz
approved these changes
Feb 25, 2020
|
Merged! Thanks for your contribution. |
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, when using local apk artifacts, Flank always uploads every
app/test apk pair even if the app apk has already been uploaded. This is
also true for pulling the old JUnitReport from GCS for smart flank where
every test apk re-downloads the results. To fix this we simply cache
uploads and downloads in a ConcurrentHashMap where the key is the GCS
path for downloads and the filename for uploads.
This change also fixes a bug where local test apks would be uploaded and
then re-downloaded to parse their tests. The sharding loop now loops over
the local AppTestPair where it uploads them if necessary and scans for
tests on the local test apk.
All in all this shaves almost 4 minutes off of shard generation for a
project with 20+ modules. There is also likely more work that can be
done to parallelize matrix creation (uploading apks, parsing tests,
creating the shard), but it is out of scope of this change.
Matrix/shard generation output (truncated and sanitized)
Before (60s smart flank shard time):
After (45s smart flank shard time hence more shards):