Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add order by to bigquery queries (for manifest creation datetimes) #5074

Merged
merged 4 commits into from
Apr 30, 2024

Conversation

lcouzens
Copy link
Contributor

@lcouzens lcouzens commented Apr 30, 2024

Jira Ticket

COST-4975

Description

This change will add an order by to the GCP bigquery we make so when creating manifests for a new provider they are created in sequential order. There is a future code side affect in cluster info because these are created some what randomly in order

Testing

  1. Checkout main
  2. Restart Koku
  3. Add REAL GCP source (Running this QE test is best test_api_ocp_on_gcp_source_cluster_info_infra)
  4. If multiple months are ingested this test will likely fail do to the manifest creation order (since latest is previous month when we do lookups for cluster info)
  5. Check sources endpoint for status summary (see examples below)
  6. Checkout Branch
  7. Rerun the above test and it should now pass.
  8. check sources endpoint for completed summary state

Example main:
"summary": {
"state": "Pending"
}

Branch Example:
"summary": {
"end": "2024-04-30T10:42:45.011783+00:00",
"start": "2024-04-30T10:42:43.755820+00:00",
"state": "complete"
}

Release Notes

  • proposed release note
* [COST-4975](https://issues.redhat.com/browse/COST-4975) Add ordering to bigquery to create manifests in asc order

@lcouzens lcouzens added the gcp-smoke-tests pr_check will build the image and run gcp + ocp on gcp smoke tests label Apr 30, 2024
@lcouzens lcouzens requested review from a team as code owners April 30, 2024 11:32
@lcouzens lcouzens changed the title Add order by to bigquery queries (for manifest creation start times) Add order by to bigquery queries (for manifest creation datetimes) Apr 30, 2024
Copy link

codecov bot commented Apr 30, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.1%. Comparing base (80d964b) to head (5c9adf1).

Additional details and impacted files
@@           Coverage Diff           @@
##            main   #5074     +/-   ##
=======================================
- Coverage   94.1%   94.1%   -0.0%     
=======================================
  Files        377     377             
  Lines      31344   31344             
  Branches    3715    3715             
=======================================
- Hits       29502   29501      -1     
  Misses      1176    1176             
- Partials     666     667      +1     

@lcouzens
Copy link
Contributor Author

Smoke test failures are failing on main in ephemeral.

@lcouzens lcouzens added hot-fix-smoke-tests pr_check label to run minimal smoke tests for fast moving bug-fix and removed gcp-smoke-tests pr_check will build the image and run gcp + ocp on gcp smoke tests labels Apr 30, 2024
@lcouzens lcouzens enabled auto-merge (squash) April 30, 2024 16:54
@lcouzens lcouzens merged commit 103c876 into main Apr 30, 2024
11 checks passed
@lcouzens lcouzens deleted the gcp-add-order-by-to-BQ-queries branch April 30, 2024 17:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
hot-fix-smoke-tests pr_check label to run minimal smoke tests for fast moving bug-fix smokes-required
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants