Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

COST-953: GCP processing window fix #2622

Merged
merged 9 commits into from
Feb 2, 2021
Merged

Conversation

dccurtis
Copy link
Contributor

@dccurtis dccurtis commented Feb 1, 2021

The GCP downloader _generate_monthly_pseudo_manifest function was generating a scan window that was one day off. Additionally there was an assumption that the partitioning date is the same as the usage date. This did not turn out to be the case and we are now doing the BigQuery query based on usage_start so that it aligns with the summary processor

Testing

  1. Set DATE_OVERRIDE to Jan-29 and ingest GCP. Verify data comes in for 1-29 of January
  2. Set DATE_OVERRIDE to Jan-30 and ingest GCP. Verify data comes in for 1-30 of January. Verify 1-29 does not change.
  3. Set DATE_OVERRIDE to Jan-31 and ingest GCP. Verify data comes in for 1-31 of January. Verify 1-30 does not change.
  4. Set DATE_OVERRIDE to Feb 1 and ingest GCP. Verify data comes in for 1 January - 1 Feb. Verify January data does not change.

Test Results
cost_593_ut.txt

@dccurtis dccurtis self-assigned this Feb 1, 2021
@codecov
Copy link

codecov bot commented Feb 1, 2021

Codecov Report

Merging #2622 (ec372d4) into master (b8fbc68) will decrease coverage by 0.0%.
The diff coverage is 100.0%.

@@           Coverage Diff            @@
##           master   #2622     +/-   ##
========================================
- Coverage    94.8%   94.7%   -0.0%     
========================================
  Files         281     281             
  Lines       21527   21529      +2     
  Branches     2445    2445             
========================================
- Hits        20397   20393      -4     
- Misses        679     682      +3     
- Partials      451     454      +3     

@dccurtis dccurtis marked this pull request as ready for review February 2, 2021 00:35
WHERE DATE(_PARTITIONTIME) >= '{scan_start}'
AND DATE(_PARTITIONTIME) <= '{scan_end}'
WHERE usage_start_time >= '{scan_start}'
AND usage_start_time < '{scan_end}'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sweet glad to see the switch to usage_start_time < '{scan_end}' worked!

Copy link
Contributor

@myersCody myersCody left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome work on this @dccurtis!

@dccurtis dccurtis merged commit c360293 into master Feb 2, 2021
@dccurtis dccurtis deleted the gcp_processing_window_fix branch February 2, 2021 13:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants