-
Notifications
You must be signed in to change notification settings - Fork 94
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
COST-953: GCP processing window fix #2622
Conversation
Codecov Report
@@ Coverage Diff @@
## master #2622 +/- ##
========================================
- Coverage 94.8% 94.7% -0.0%
========================================
Files 281 281
Lines 21527 21529 +2
Branches 2445 2445
========================================
- Hits 20397 20393 -4
- Misses 679 682 +3
- Partials 451 454 +3 |
WHERE DATE(_PARTITIONTIME) >= '{scan_start}' | ||
AND DATE(_PARTITIONTIME) <= '{scan_end}' | ||
WHERE usage_start_time >= '{scan_start}' | ||
AND usage_start_time < '{scan_end}' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sweet glad to see the switch to usage_start_time < '{scan_end}'
worked!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Awesome work on this @dccurtis!
The GCP downloader
_generate_monthly_pseudo_manifest
function was generating a scan window that was one day off. Additionally there was an assumption that the partitioning date is the same as the usage date. This did not turn out to be the case and we are now doing the BigQuery query based on usage_start so that it aligns with the summary processorTesting
Test Results
cost_593_ut.txt