AoU cleanup docs, round 1 [VS-671] #8104
Conversation
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## ah_var_store #8104 +/- ##
================================================
Coverage ? 86.246%
Complexity ? 35202
================================================
Files ? 2173
Lines ? 165045
Branches ? 17793
================================================
Hits ? 142345
Misses ? 16375
Partials ? 6325 |
|
Github actions tests reported job failures from actions build 3533317748
|
|
Github actions tests reported job failures from actions build 3535155196
|
|
Github actions tests reported job failures from actions build 3535257980
|
|
Github actions tests reported job failures from actions build 3566605662
|
|
Github actions tests reported job failures from actions build 3578066764
|
|
|
||
| * VDS storage cost: $500 / month. Note AoU should have exact copies of the two VDSes generated for Delta. | ||
| * VDS regeneration cost: $1000 (~10 hours @ ~$100 / hour cluster cost) + $3000 to regenerate Avro files if necessary. | ||
|
|
There was a problem hiding this comment.
what about the intermediate VDS files? dont they not get cleaned up?
There was a problem hiding this comment.
Those live in the gvs import temporary directory. The current expectation is that we would nuke that directory manually upon successful completion of the import.
mcovarr
force-pushed
the
vs_671_aou_cleanup
branch
from
835e278 to
e2f80c0
Compare
|
Github actions tests reported job failures from actions build 3587567328
|
mcovarr
force-pushed
the
vs_671_aou_cleanup
branch
3 times, most recently
from
e2774d7 to
4b321f3
Compare
mcovarr
force-pushed
the
vs_671_aou_cleanup
branch
from
4b321f3 to
2033c0c
Compare
| Approximate figures for the ~250K samples Delta callset: | ||
|
|
||
| * VDS storage cost: ~$500 / month. Note AoU should have exact copies of the VDSes we have delivered for Delta, though | ||
| it's not certain that these copies will remain accessible to the Variants team in the long term. The Variants team has |
There was a problem hiding this comment.
This is the Brian Freeman question? Is there someone from AoU that can verify?
There was a problem hiding this comment.
I believe we decided yesterday that we're just going to hang on to the first and last Delta VDS for now.
| GvsPrepareRangesCallset: [$1,803.14](https://docs.google.com/spreadsheets/d/1fcmEVWvjsx4XFLT9ZUsruUznnlB94xKgDIIyCGu6ryQ/edit#gid=0) | ||
| * Storing prepare data: $878.39 / month |
There was a problem hiding this comment.
I think it would be more helpful to have the way to calculate these numbers (e.g. number of GB * $6) in addition to the specific costs from the last run (I picked that number out of thin air).
| * ~~AI/AN filtering round one `gs://fc-secure-fb908548-fe3c-41d6-adaf-7ac20d541375/delta_ai_an_filtered_2022_11_15.vds/`~~ | ||
| * ~~AI/AN filtering round two, controls removed `gs://fc-secure-fb908548-fe3c-41d6-adaf-7ac20d541375/delta_ai_an_control_filtered_2022_12_13.vds/`~~ |
There was a problem hiding this comment.
Why are we keeping these in the doc if they have been deleted?
No description provided.