Batched Avro export [VS-630] #8020
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mcovarr
requested review from
gbggrant,
rsasch,
koncheto-broad and
RoriCremer
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## ah_var_store #8020 +/- ##
================================================
Coverage ? 86.244%
Complexity ? 35197
================================================
Files ? 2173
Lines ? 165004
Branches ? 17792
================================================
Hits ? 142306
Misses ? 16372
Partials ? 6326 |
rsasch
approved these changes
Sep 16, 2022
There was a problem hiding this comment.
Looks great! Test run with AoU 194k at https://job-manager.dsde-prod.broadinstitute.org/jobs/da1e40cb-e10b-476b-9c5e-3673582795b7.
RoriCremer
reviewed
Sep 19, 2022
RoriCremer
reviewed
Sep 19, 2022
| bq query --nouse_legacy_sql --project_id=~{project_id} " | ||
| EXPORT DATA OPTIONS( | ||
| uri='${avro_prefix}/vets/vet_${str_table_index}/vet_${str_table_index}_*.avro', format='AVRO', compression='SNAPPY') AS | ||
| SELECT location, sample_id, ref, REPLACE(alt,',<NON_REF>','') alt, call_GT as GT, call_AD as AD, call_GQ as GQ, cast(SPLIT(call_pl,',')[OFFSET(0)] as int64) as RGQ |
There was a problem hiding this comment.
I dont think we need to extract PLs? it doesn't look like Tim uses them. Let's ask on Wednesday. (Fine to include for now, I just happened to notice it in this pr)
RoriCremer
approved these changes
Sep 19, 2022
RoriCremer
reviewed
Sep 19, 2022
| # appropriate partition, the outer '+ 1' is to iterate over the correct number of partitions. | ||
| scatter (i in range(((CountSamples.num_samples - 1) / 4000) + 1)) { | ||
| Int num_samples = CountSamples.num_samples | ||
| Int num_superpartitions = if (num_samples % 4000 == 0) then num_samples / 4000 else (num_samples / 4000 + 1) |
There was a problem hiding this comment.
a comment on this math would be helpful
mcovarr
force-pushed
the
vs_630_batched_avro_export
branch
from
eed5a98 to
5e8e597
Compare
This was referenced Mar 17, 2023
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
To address scalability failings with unbatched Avro exports.