Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Vs 299 spike speed up integration tests #8706

Draft
wants to merge 901 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
901 commits
Select commit Hold shift + click to select a range
4a1c203
Document VCF generation [VS-795] (#8202)
rsasch Feb 16, 2023
1ce13b3
Variants GATK Docker image building docs + script [VS-827] (#8207)
mcovarr Feb 22, 2023
fdbaa14
Update GATK jar used in GvsJointVariantCalling WDL (#8216)
rsasch Feb 22, 2023
0014005
Hello Azure SQL Database from Cromwell on Azure [VS-812] (#8220)
mcovarr Feb 27, 2023
6f747d0
Remove what appear to be accidentally added files [VS-834] (#8225)
mcovarr Feb 28, 2023
6d41adf
VS-815: Add Support for YNG to VQSR Lite (#8206)
gbggrant Mar 1, 2023
3a7f6e2
Disentangle non-GVS code from GVS code [VS-834] (#8229)
mcovarr Mar 3, 2023
5645e88
VS-695. Updates to run Precision and Sensitivity on VQSR Lite (#8230)
gbggrant Mar 3, 2023
781bb14
Track avro export costs [VS-769] (#8236)
rsasch Mar 8, 2023
b09d909
Add note that we deleted a VDS! (#8214)
RoriCremer Mar 13, 2023
3e5a756
Vs 822 Add documentation for the work that we did on the latest itera…
RoriCremer Mar 14, 2023
291bfd0
Rc vs 822 gq0 documentation (#8240)
RoriCremer Mar 14, 2023
bb6806b
Add a test exclusion for gvs scripts (#8250)
lbergelson Mar 21, 2023
3880421
Intro to Cosmos Spike [VS-845] (#8254)
mcovarr Mar 24, 2023
c0535f2
Rc vs 778 spike import limit (#8197)
RoriCremer Mar 27, 2023
5e19ec0
Update variants base image [VS-866] (#8262)
mcovarr Mar 27, 2023
2dd76f7
Rename tieout WDL input to not match name of output [VS-860] (#8265)
mcovarr Mar 28, 2023
529b078
VS-849 - Use the annotation 'AS_MQ' for indels. (#8261)
gbggrant Mar 29, 2023
8217073
Tidy up ExtractCohortTest so it can be run in IntelliJ cleanly [VS-86…
mcovarr Mar 30, 2023
23a64a7
VS-838. Remove the unneeded SCORE field from the filter_set_info_vqsr…
gbggrant Apr 5, 2023
e2b2e5c
VS-775 vat validation shards (#8175)
RoriCremer Apr 10, 2023
4ab6bde
VS-883 - Add tests for extract cohort lite (#8284)
gbggrant Apr 12, 2023
0f24625
Fix broken gsutil in Variants Docker image [VS-888] (#8289)
mcovarr Apr 14, 2023
ce3a5c7
Adding the uber_monitor.py script (#8268)
gbggrant Apr 16, 2023
dc8b800
VS-885 - Fix VAT TSV (#8286)
gbggrant Apr 18, 2023
17afee4
Sanity check variantstore images before publishing [VS-889] (#8291)
mcovarr Apr 18, 2023
f4a355c
VS-776. Update to latest version of VQSR Lite. (#8269)
gbggrant Apr 21, 2023
a2ffeb8
Azure SQL Database Ingest [VS-879] (#8293)
mcovarr Apr 24, 2023
daeae13
VS-895. Add VQSR lite support to 'extract cohort by sample names' (#8…
gbggrant Apr 26, 2023
fc2c7f7
Move monitoring script to public bucket [VS-908] (#8303)
mcovarr May 1, 2023
6dc27f0
remove GvsAoUReblockGVCF.wdl and references to it (#8300)
mmorgantaylor May 2, 2023
0b4e305
Add python script to our repo (#8282)
RoriCremer May 5, 2023
834e6da
Refactor to put 'Lite' functionality into ExtractCohort. (#8295)
gbggrant May 10, 2023
928ffe9
Serverless Azure SQL Database costs [VS-907] (#8309)
mcovarr May 10, 2023
6a0b1a4
VS-917. Update override jar in GvsJointCalling wdl to fix support iss…
gbggrant May 11, 2023
316d013
Fix potential bug in VCF Integration test (#8316)
gbggrant May 12, 2023
deadfc8
Bulk Ingest (#8301)
RoriCremer May 17, 2023
b64a252
RWB Search Export [VS-909] (#8311)
mcovarr May 17, 2023
ebe4835
No empty alts if you please [VS-918] (#8326)
mcovarr May 18, 2023
fe981fa
Process Sample gVCF headers during Ingest [VS-836] (#8321)
rsasch May 22, 2023
0b83d1d
VS-802. Add support for VQSR Lite to VDS generation. (#8322)
gbggrant May 23, 2023
85205fc
replace spaces with url-encoded in docker build documentation (#8333)
rsasch May 23, 2023
55a471a
VS-924 - Have pipeline automatically differentiate between VQSR Lite …
gbggrant May 25, 2023
40947ed
Update to the latest import_gvs() code (#8330)
RoriCremer May 26, 2023
24b0ca1
VS-943. Fix bug in summarize_monitoring_logs script (#8345)
gbggrant May 30, 2023
b24b508
VS-910 Handling ploidy issues in GVS (#8334)
koncheto-broad May 31, 2023
0b41b51
updating beta to use the latest code changes (#8349)
koncheto-broad Jun 1, 2023
c575ff8
bq --apilog=false [VS-944] (#8348)
mcovarr Jun 2, 2023
0feb524
VS-914 Add support for vqsr lite to integration test (#8324)
gbggrant Jun 2, 2023
ed338e4
GVS BigQuery Traversal Artifact [VS-927] (#8344)
mcovarr Jun 5, 2023
e0d408a
Docker build script --branch and --release options [VS-935] (#8354)
mcovarr Jun 8, 2023
a191f2d
wiring through the option for VQSR-Lite (#8353)
koncheto-broad Jun 13, 2023
5b72ceb
Even Quickerstart (3 samples, chr20, chrX, chrY only) [VS-682] (#8350)
mcovarr Jun 13, 2023
1dafc33
VQSR Lite by default except for beta users [VS-700] (#8360)
mcovarr Jun 14, 2023
9691e72
Update gvs-quickstart.md (#8364)
kbergin Jun 14, 2023
d1c76a9
Add ability to run S&P workflow on arbitrary multiple chromosomes [VS…
rsasch Jun 15, 2023
af4f273
Remove unused parameter. (#8370)
gbggrant Jun 21, 2023
37fe914
VS-987 out of date docker (#8374)
gbggrant Jun 22, 2023
1ce25db
ah_var_store housekeeping (#8377)
mcovarr Jun 22, 2023
e3f2d8a
updated GvsJointVariantCalling.wdl with latest override_gatk (#8376)
rsasch Jun 23, 2023
517da3d
Bulk Ingest with sample sets (#8325)
RoriCremer Jun 25, 2023
d7d1737
Use portable Python arg parsing from Docker build script [VS-995] (#8…
mcovarr Jun 26, 2023
901644f
add dockstore to ah_var_store (#8383)
RoriCremer Jun 26, 2023
500b88a
doh (#8384)
RoriCremer Jun 26, 2023
8813ed0
VS-942. Fix an error where NO variants were found (#8388)
gbggrant Jun 28, 2023
90aa0fd
Fix argument parsing test [VS-995] (#8389)
mcovarr Jun 28, 2023
c13b8c9
VS-1000 - Migrate VQSR Classic Code to own WDL (#8393)
gbggrant Jun 29, 2023
17b0493
Pinning joblib to v1.1.1. (#8392)
mcovarr Jun 30, 2023
09989de
Move BigQuery classes [VS-971] (#8375)
mcovarr Jun 30, 2023
e90d90e
BigQuery storage v1beta2 to v1 [VS-1001] (#8399)
mcovarr Jun 30, 2023
d42630f
VS-1001 update dockers for google storage library update (#8401)
gbggrant Jul 5, 2023
b240348
VS-950 - Developer Documentation for running Exomes through GVS (#8398)
gbggrant Jul 6, 2023
5580ca0
Update documentation to point to newly released BGE exome calling lis…
gbggrant Jul 11, 2023
7a144d3
VS-1007 - Use default names for VQSR-Lite generated code. (#8412)
gbggrant Jul 14, 2023
d37e34b
VS-1009 Add SCORE to VQSR Lite VCF (#8423)
gbggrant Jul 20, 2023
154bee2
Copy/paste cleanup magic for Github Action CI (#8437)
mcovarr Jul 25, 2023
f839eb0
pin hail version in the integration test (#8424)
RoriCremer Aug 3, 2023
189242c
Kd GVS bulk ingest docs (#8397)
kbergin Jul 13, 2023
597f75e
Bulk ingest: single FOFN + Quickstart + Beta [VS-962] [VS-984] [VS-98…
mcovarr Jul 15, 2023
b96dce0
Update logic around missing sample data in TDM [VS-997] (#8422)
rsasch Jul 18, 2023
65aea7f
dockstore
rsasch Jul 19, 2023
94c86dc
more dockstore
rsasch Jul 19, 2023
3cb94c7
Fix AssignIds pipefail with `head` [VS-1011] (#8441)
mcovarr Jul 27, 2023
5ba5b89
VS-991. Wire interval list from Joint Calling through to other workfl…
gbggrant Jul 28, 2023
434206d
All sample inputs as files instead of strings [VS-1019] (#8444)
rsasch Aug 1, 2023
2c9534d
VS-1004. Increasing max auto batch size to 25K to allow for larger be…
gbggrant Aug 1, 2023
118ede9
Update P + S WDL to get around input limits [VS-970] (#8449)
mcovarr Aug 2, 2023
4a209b8
Update override jar in bulk ingest staging (#8451)
gbggrant Aug 2, 2023
c837ad7
VS-931 Exome integration test on bulk ingest staging (#8448)
gbggrant Aug 3, 2023
c5724cb
VS-1004 Changes to make benchmarking VQSR Classic possible. (#8452)
gbggrant Aug 7, 2023
1190e80
Stop mysterious sample drops in curation, improve performance [VS-102…
mcovarr Aug 7, 2023
a7585bf
Centralize and standardize Docker image versioning [VS-965] (#8457)
mcovarr Aug 10, 2023
2eda6c5
Centralized Docker fixups (#8469)
mcovarr Aug 11, 2023
3d48f01
Update GVS docs with VETS information (#8466)
kbergin Aug 11, 2023
98a22c4
update Dockers
mcovarr Aug 12, 2023
c1efb6f
Fix copy/paste errors with Docker image variable names (#8474)
mcovarr Aug 15, 2023
020f934
Bump up some disk sizes for P&S (#8480)
mcovarr Aug 16, 2023
c08d4c7
Internal bulk ingest docs [VS-985] (#8479)
mcovarr Aug 21, 2023
dc1b3e9
Coaelesce 4 bulk ingest tasks into one (#8478)
mcovarr Aug 21, 2023
95dd23f
GVS versioning, Beta WDLs can be pinned to a version [VS-1032] (#8493)
mcovarr Aug 25, 2023
3b44722
Add more disk space to extract (#8473)
RoriCremer Aug 25, 2023
f30597a
Update Beta workspace [VS-969] (#8496)
mcovarr Aug 25, 2023
82568b9
Make extract backwards compatible with the new schema (#8488)
RoriCremer Aug 25, 2023
da42d34
Initial updates to bulk ingest docs with 25k run (#8475)
kbergin Aug 28, 2023
6cc161c
Notes from the hail meeting to track improvements (#8498)
RoriCremer Aug 29, 2023
8ee10e8
Merge remote-tracking branch 'origin/master' into vs_1012_master_finally
mcovarr Aug 29, 2023
f2b8edd
GATKTool.java from master, adapt to GATKTool reference being package …
mcovarr Aug 29, 2023
e26a3dc
Annotation adjustments for ExtractTool
mcovarr May 3, 2023
08dd603
Call new ReferenceConfidenceVariantContextMerger#mergeWithRemapping m…
mcovarr May 17, 2023
771df9a
Java 17 for most spots, Java 8 remains for Hail
mcovarr May 11, 2023
414849a
Dockstore
mcovarr Aug 29, 2023
960ef17
Adapt for Avro 1.11 behavior of throwing on get()s of non-existent fi…
mcovarr Mar 28, 2023
5c3d571
copy script
mcovarr Aug 29, 2023
b2ef240
Update GATK Docker image to get required Java 17 JRE
mcovarr Aug 29, 2023
7155e0c
VS-1029 Fix naming errors in hail_gvs_import script (#8487)
gbggrant Aug 30, 2023
34ed02e
Changes after creating 190K Exome Callset [VS-189] (#8459)
rsasch Aug 31, 2023
5eb4528
PR feedback
mcovarr Aug 31, 2023
085efa4
Merge remote-tracking branch 'origin/ah_var_store' into vs_1012_maste…
mcovarr Aug 31, 2023
41f19e8
update Dockers
mcovarr Aug 31, 2023
a4bed50
Make VETS default in beta workflow [VS-1039] (#8511)
rsasch Sep 1, 2023
f8427ad
Update gvs workspace description [VS-1042] (#8509)
kbergin Sep 1, 2023
9af6e15
Improve Beta UX with a default `extract_output_gcs_dir` [VS-1040] (#8…
mcovarr Sep 1, 2023
3a2adec
Add support for ingest from requester pays bucket [VS-1038] (#8514)
rsasch Sep 7, 2023
0c8cd41
Update hail version to 120 in Integration test VS 1025 (#8502)
RoriCremer Sep 11, 2023
2dc324c
VS-1024 beta workflow for exomes (#8515)
gbggrant Sep 11, 2023
689ad30
Adding CHANGELOG.md entry for new Beta branch 0.3.2 (#8518)
gbggrant Sep 11, 2023
bfdcf12
VS-1047 remove --location=US in bq command calls (#8519)
gbggrant Sep 18, 2023
62e56c8
Make a VDS (how to / how do we even) for VS-1026 (#8513)
RoriCremer Sep 19, 2023
878dcba
Vs-327 (#8507)
koncheto-broad Sep 22, 2023
269bd95
VS-974 - Better error on GvsCreateFilterSet.PopulateFilterSetInfo (#8…
gbggrant Sep 26, 2023
fae98a2
Hail Dataproc from WDL, cluster tracking [VS-1013] [VS-1021] (#8525)
mcovarr Sep 27, 2023
d27c412
VS-1056 support import of PS (#8531)
gbggrant Sep 28, 2023
c28b058
Close header line creator once ingest is complete [VS-1045] (#8533)
mcovarr Sep 29, 2023
44054fb
Add benchmark exome run details [VS-1005] (#8532)
rsasch Sep 29, 2023
6d48223
VS-1059 Add Phasing information to Avro files (#8536)
gbggrant Oct 2, 2023
b1dece8
Merge remote-tracking branch 'origin/master' into vs_1065_merge_master
mcovarr Oct 2, 2023
9ac0bef
Merge master, fixups for changes in JointVariantCalling.wdl [VS-1065]
mcovarr Oct 2, 2023
a5d8dac
Documentation change on quota increases.
gbggrant Oct 6, 2023
062776e
Revert Documentation change on quota increases. (#8541)
gbggrant Oct 6, 2023
80034d3
Restore Beta throttling [VS-1072] (#8540)
mcovarr Oct 6, 2023
6529d8e
Documentation change on quota increases. (#8542)
gbggrant Oct 6, 2023
f1b9553
Throttling integration test [VS-1076] (#8545)
mcovarr Oct 10, 2023
e09afef
Do not cache workspace-reading GenerateImportFofnFromDataTable [VS-10…
mcovarr Oct 10, 2023
f64f9b1
Finalizing CHANGELOG.md for GVS 0.4.0 [VS-1074] (#8547)
mcovarr Oct 10, 2023
4f75a5b
VS-857 - Add ability to drop GQ0 ref state and no calls. (#8538)
gbggrant Oct 10, 2023
2f91ed2
Release docs fixup [VS-1088] (#8548)
mcovarr Oct 11, 2023
ceadce0
VS-857. Change default behavior of GvsBulkIngestGenomes.wdl to drop s…
gbggrant Oct 12, 2023
0750dc5
Vs 1063 bit pack ref ranges data into a more efficient representation…
koncheto-broad Oct 12, 2023
ec49042
Always have git hash in Docker tags to avoid collisions [VS-1086] (#8…
mcovarr Oct 16, 2023
206efcb
Make a Quickstart VDS in a WDL [VS-1064] (#8534)
RoriCremer Oct 16, 2023
8245d48
Use `gcloud storage cp` to sidestep composite object issues [VS-1090]…
mcovarr Oct 18, 2023
8da467b
Fixing LoadData error discovered in delcho run (#8556)
koncheto-broad Oct 18, 2023
5addbba
Bitpacking fixup [VS-1104] (#8559)
mcovarr Oct 23, 2023
90687e7
VS-1105. Write missing ref ranges as ZERO state, unless dropping that…
gbggrant Oct 24, 2023
0b4cb75
AOU deliverables doc updates [VS-1073] (#8551)
mcovarr Oct 24, 2023
838e57a
Integration testing tables are not forever [VS-1049] (#8563)
mcovarr Oct 26, 2023
c00d459
VS-1115 - Updating CHANGELOG.md for new missing behavior (#8565)
gbggrant Oct 26, 2023
2635ade
VS-1119 remove unused call (#8566)
gbggrant Nov 3, 2023
12846a0
VS-1092 - Fix for GvsCreateFilterSet OOD (#8573)
gbggrant Nov 3, 2023
fbf53db
More memory for PopulateFilterSetInfo (#8575)
gbggrant Nov 6, 2023
162d9fe
VS-1090 - Update AoU Documentation for GvsImportGenomes to reflect ne…
gbggrant Nov 7, 2023
9d297c0
VS 1102 add vds creation wdl to integration tests (#8554)
RoriCremer Nov 7, 2023
8e1c160
Make BigQuery command line calls more consistent across WDLs [VS-458]…
rsasch Nov 7, 2023
803515b
Phasing and GT removal in VDS [VS-1060] [VS-1122] [VS-1123] (#8553)
mcovarr Nov 8, 2023
20e9d49
Add logging export to VDS creation VS 1139 (#8583)
RoriCremer Nov 15, 2023
0fcf2fd
AoU Echo Precision and Sensitivity Updates [VS-1093] (#8578)
mcovarr Nov 15, 2023
7498a6d
VS 923 add validation to cluster creation (#8581)
RoriCremer Nov 15, 2023
6212cd3
Plumb through disk and memory overrides for ExtractTask [VS-1124] (#8…
rsasch Nov 16, 2023
e43e81b
Compressed reference smarts [VS-1138] (#8586)
mcovarr Nov 20, 2023
89ff60f
Revert phasing changes [VS-1144] (#8590)
mcovarr Nov 21, 2023
22d2b7b
Update docs for callset stats [VS-1094] (#8591)
rsasch Nov 21, 2023
e219cfc
Add parameter to give tables created by prepare step a two week TTL …
rsasch Dec 1, 2023
b9d6afb
VS-1159 - Enhance GVSWithdrawSamples (#8599)
gbggrant Dec 5, 2023
307c7ca
VS 678 - adding optional interval list to the subcohort extract workf…
koncheto-broad Dec 7, 2023
f8d5f2d
Echo Scale Test VDS Updates [VS-1095] [VS-1112] (#8602)
mcovarr Dec 15, 2023
816d644
Adjust ingest/extract scatter calculations for exomes [VS-1146] (#8615)
rsasch Dec 18, 2023
655ac15
VS-1158 Address precision and sensivity probs (#8633)
gbggrant Jan 4, 2024
ba1c460
VS-1163 - Set default to enable TTL on prepare ranges tables. (#8640)
gbggrant Jan 4, 2024
da8fcb0
Updates to CHANGELOG.md in preparation for release 0.5.1 (#8641)
gbggrant Jan 8, 2024
a9380ee
Small doc change before releasing exomes [VS-932] (#8646)
rsasch Jan 9, 2024
463c6cf
Search for and possibly clean up Hail refuse [VS-1022] (#8644)
mcovarr Jan 9, 2024
907e928
Address ExtractTask OOMs [VS-1127] (#8648)
mcovarr Jan 16, 2024
8edbc46
Fix GvsExtractCohortFromSampleNames.wdl [VS-1197]
mcovarr Jan 17, 2024
c4da917
Updates in preparation for release 0.5.2 [VS-1196] (#8659)
mcovarr Jan 17, 2024
2fc9c9c
VS-1103 GatherVcfs logging too much (#8650)
gbggrant Jan 18, 2024
573c915
Standardize bash options [VS-460] (#8658)
mcovarr Jan 18, 2024
82e96db
Workspace updates in release docs [VS-1202] (#8661)
mcovarr Jan 18, 2024
f7c3953
Replace LGT with GT within hail_create_vat_inputs.py [VS-1195] (#8660)
rsasch Jan 18, 2024
c2933e1
Fix population typo [VS-1155] (#8665)
rsasch Jan 29, 2024
7d9343c
Add datetime modified check to IsUsingCompressedReferences for call c…
rsasch Jan 30, 2024
fe782dc
Add support for interval list or bed file to Prepare Step (within lim…
rsasch Jan 30, 2024
bfd313e
Add CheckForNullColumns to and fixed ClinvarSignificance in VAT valid…
rsasch Jan 31, 2024
aef44a2
Minor changes for updated VAT reference testing [VS-1054] (#8670)
mcovarr Feb 2, 2024
3ac378c
VS-1206 (#8679)
koncheto-broad Feb 6, 2024
0dc4e8a
VS-1058 add phasing to vcf (#8655)
gbggrant Feb 8, 2024
4ba7b40
Change format of sample Avro files to speed up large callset VDS crea…
rsasch Feb 8, 2024
07057b6
Make run_in_hail_cluster.py more generic and use it to create the VAT…
rsasch Feb 8, 2024
0e4e6d3
Only run certain VAT tests on larger callsets [VS-823] (#8685)
rsasch Feb 12, 2024
9629428
VDS optional args fixes [VS-1226] (#8687)
mcovarr Feb 12, 2024
01b571f
Update documentation for VDS/VAT [VS-1125] (#8688)
rsasch Feb 14, 2024
3d3d10d
Allow for running Hail via a wheel [VS-1214] (#8692)
mcovarr Feb 14, 2024
bbc829a
small doc change (#8695)
rsasch Feb 15, 2024
fe1bc63
uploading basic scaffolding code
koncheto-broad Feb 15, 2024
ed98c09
uploading basic scaffolding code... again
koncheto-broad Feb 15, 2024
ace7eab
testing structure of the bash loop when embedded in a wdl
koncheto-broad Feb 15, 2024
649f773
testing structure of the bash loop when embedded in a wdl 2
koncheto-broad Feb 15, 2024
0f351b6
VS-1232 SelectVariants extension fix (#8696)
gbggrant Feb 16, 2024
3f96025
testing inserting db rows for the moved tables
koncheto-broad Feb 16, 2024
aa28c75
weird bash error
koncheto-broad Feb 16, 2024
f2a1512
weird stuff v2: removing things
koncheto-broad Feb 16, 2024
3ced854
ugh v1
koncheto-broad Feb 16, 2024
f4f0d2b
ugh v2
koncheto-broad Feb 16, 2024
b06fc84
ugh v3--found it
koncheto-broad Feb 16, 2024
96f6867
ugh v4--fixing SQL issue
koncheto-broad Feb 16, 2024
c816df2
changing table
koncheto-broad Feb 16, 2024
6f1e4d9
changing table v2
koncheto-broad Feb 16, 2024
25c788a
changing table v2
koncheto-broad Feb 16, 2024
833408b
testing util for getting table names, including exclusion filtering
koncheto-broad Feb 20, 2024
18ccbde
forgot runtime clause
koncheto-broad Feb 20, 2024
7965a44
missing single quote in conditional bash
koncheto-broad Feb 20, 2024
764fbd2
VS 1036 documenting quotas (#8649)
RoriCremer Feb 20, 2024
4a51ee0
Headers only ingest [VS-1187] (#8674)
mcovarr Feb 20, 2024
da4f056
adding timestamps for efficient storage
koncheto-broad Feb 20, 2024
3098ff5
making debugging slightly easier
koncheto-broad Feb 21, 2024
040bf21
making debugging slightly easier v2
koncheto-broad Feb 21, 2024
936d031
making debugging slightly easier v2.5
koncheto-broad Feb 21, 2024
b50b0d5
changes to reuse existing tables
koncheto-broad Feb 21, 2024
80f9d03
changes to reuse existing tables v2
koncheto-broad Feb 21, 2024
98a062a
bash syntax issues are fun
koncheto-broad Feb 21, 2024
66fdc9f
oops, used wrong variable substitution
koncheto-broad Feb 21, 2024
89f09a6
sloppy, sloppy
koncheto-broad Feb 21, 2024
bbe53c1
oops, break -> continue
koncheto-broad Feb 21, 2024
1d8d61c
Attempting a real callset to try it for real this time
koncheto-broad Feb 21, 2024
b68614f
dockstore
koncheto-broad Feb 21, 2024
3827e68
keep wdl task alias nice and distinct
koncheto-broad Feb 21, 2024
8e236ae
wdl not very smart
koncheto-broad Feb 21, 2024
48854cf
wdl task sequencing Is A Thing
koncheto-broad Feb 22, 2024
695d57e
wdl task sequencing Is A Thing Part 2: Sequential Boogaloo
koncheto-broad Feb 22, 2024
e4f86b0
testing snapshot restoration
koncheto-broad Feb 22, 2024
1f2e479
silly bash error
koncheto-broad Feb 22, 2024
2b6c529
testing results comparison
koncheto-broad Feb 27, 2024
7199f69
tweaking output for comparison
koncheto-broad Feb 27, 2024
c34adc2
VS-1129 Remove Duplicates from VAT upon creation. (#8700)
gbggrant Feb 27, 2024
bd09760
add link to README (#8705)
rsasch Feb 27, 2024
0fc84a5
Added a note to AoU documentation describing possible need for more m…
gbggrant Feb 27, 2024
38a4be3
parallel run experiment
koncheto-broad Feb 27, 2024
8896475
typo in dockstore
koncheto-broad Feb 28, 2024
e897ce8
changing name of parallel task
koncheto-broad Feb 28, 2024
eb26750
Merge branch 'ah_var_store' into VS-299-spike-speed-up-integration-tests
koncheto-broad Feb 29, 2024
fb18c3d
pushing schema of table_mappings table
koncheto-broad Feb 29, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
299 changes: 297 additions & 2 deletions .dockstore.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ workflows:
testParameterFiles:
- /scripts/cnv_cromwell_tests/germline/cnv_germline_case_scattered_workflow.json
filters:
branches:
branches:
- master
tags:
- /.*/
Expand All @@ -33,7 +33,7 @@ workflows:
- name: cnv_somatic_pair_workflow
subclass: WDL
primaryDescriptorPath: /scripts/cnv_wdl/somatic/cnv_somatic_pair_workflow.wdl
testParameterFiles:
testParameterFiles:
- /scripts/cnv_cromwell_tests/somatic/cnv_somatic_pair_wes_do-gc_workflow.json
filters:
branches:
Expand Down Expand Up @@ -88,6 +88,301 @@ workflows:
- master
tags:
- /.*/
- name: GvsAssignIds
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsAssignIds.wdl
testParameterFiles:
- /scripts/variantstore/wdl/GvsAssignIds.example.inputs.json
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: GvsBenchmarkExtractTask
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsBenchmarkExtractTask.wdl
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: GvsRescatterCallsetInterval
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsRescatterCallsetInterval.wdl
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: GvsCreateFilterSet
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsCreateFilterSet.wdl
testParameterFiles:
- /scripts/variantstore/wdl/GvsCreateFilterSet.example.inputs.json
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: GvsPopulateAltAllele
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsPopulateAltAllele.wdl
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: GvsCreateTables
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsCreateTables.wdl
testParameterFiles:
- /scripts/variantstore/wdl/GvsCreateTables.example.inputs.json
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: GvsCreateVDS
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsCreateVDS.wdl
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: GvsExtractCallset
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsExtractCallset.wdl
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: GvsImportGenomes
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsImportGenomes.wdl
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: GvsBulkIngestGenomes
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsBulkIngestGenomes.wdl
filters:
branches:
- master
- ah_var_store
- vs_1187_headers_only_ingest
tags:
- /.*/
- name: GvsPrepareRangesCallset
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsPrepareRangesCallset.wdl
testParameterFiles:
- /scripts/variantstore/wdl/GvsPrepareRangesCallset.example.inputs.json
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: GvsCreateVATfromVDS
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsCreateVATfromVDS.wdl
filters:
branches:
- master
- ah_var_store
- gg_VS-1229_DeDupTheVatUponCreation
tags:
- /.*/
- name: GvsCreateVATFilesFromBigQuery
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/variant_annotations_table/GvsCreateVATFilesFromBigQuery.wdl
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: GvsValidateVat
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/variant_annotations_table/GvsValidateVAT.wdl
testParameterFiles:
- /scripts/variantstore/variant_annotations_table/GvsValidateVat.example.inputs.json
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: GvsExtractCohortFromSampleNames
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsExtractCohortFromSampleNames.wdl
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: GvsWithdrawSamples
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsWithdrawSamples.wdl
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: GvsJointVariantCalling
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsJointVariantCalling.wdl
filters:
branches:
- master
- ah_var_store
- VS-299-spike-speed-up-integration-tests
tags:
- /.*/
- name: GvsBeta
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsJointVariantCalling.wdl
filters:
branches:
# `master` is here so there will be *something* under the branches filter, but GVS WDLs are not on `master`
# so this shouldn't show up in the list of available versions in Dockstore or Terra.
- master
- vs_1099_composite_object_inputs
tags:
- /.*/
- name: GvsJointVariantCallsetCost
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsJointVariantCallsetCost.wdl
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: GvsCalculatePrecisionAndSensitivity
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsCalculatePrecisionAndSensitivity.wdl
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: GvsQuickstartVcfIntegration
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsQuickstartVcfIntegration.wdl
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: GvsQuickstartHailIntegration
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsQuickstartHailIntegration.wdl
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: GvsQuickstartIntegration
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsQuickstartIntegration.wdl
filters:
branches:
- master
- ah_var_store
- vs_1187_headers_only_ingest
tags:
- /.*/
- name: GvsIngestTieout
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsIngestTieout.wdl
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: GvsCallsetCost
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsCallsetCost.wdl
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: GvsExtractAvroFilesForHail
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsExtractAvroFilesForHail.wdl
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: GvsCallsetStatistics
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsCallsetStatistics.wdl
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: GvsFindHailCruft
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsFindHailCruft.wdl
filters:
branches:
- master
- ah_var_store
tags:
- /.*/
- name: HailFromWdl
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/HailFromWdl.wdl
filters:
branches:
- master
tags:
- /.*/
# Destroy this later once testing is done
- name: GvsSnapshotRun
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsSnapshotRun.wdl
filters:
branches:
- master
- VS-299-spike-speed-up-integration-tests
tags:
- /.*/
- name: GvsParallelIntegrationTest
subclass: WDL
primaryDescriptorPath: /scripts/variantstore/wdl/GvsParallelIntegrationTest.wdl
filters:
branches:
- master
- VS-299-spike-speed-up-integration-tests
tags:
- /.*/
- name: MitochondriaPipeline
subclass: WDL
primaryDescriptorPath: /scripts/mitochondria_m2_wdl/MitochondriaPipeline.wdl
Expand Down
4 changes: 4 additions & 0 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -613,6 +613,10 @@ task collectBundleIntoDir(type: Copy) {
from("scripts/sv", { into("scripts/sv") })
from("scripts/cnv_wdl/", { into("scripts/cnv_wdl") })
from("scripts/mutect2_wdl/", { into("scripts/mutect2_wdl") })
from("scripts/variantstore/") {
include 'wdl/extract/curate_input_array_files.py'
into("scripts/variantstore")
}
from("scripts/dataproc-cluster-ui", { into("scripts/")})
into "$buildDir/bundle-files-collected"
}
Expand Down
Loading