Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[COST-4741] Add UsageType and Operation to daily data #5216

Merged
merged 3 commits into from
Jul 17, 2024

Conversation

samdoran
Copy link
Contributor

@samdoran samdoran commented Jul 12, 2024

Jira Ticket

COST-3761
COST-4741

Description

Add additional columns to the AWS parquet files needed to determine node network costs.

Testing

  1. Checkout Branch
  2. Restart Koku
  3. Drop external tables
python koku/manage.py migrate_trino_tables --drop-tables aws_line_items,aws_line_items_daily,aws_openshift_daily
  1. Download data so that the external tables are recreated
  2. Check that the columns exist with expected data
trino:org1234567> SELECT DISTINCT lineitem_usagetype,lineitem_operation from aws_line_items;

Release Notes

  • proposed release note
* [COST-4741](https://issues.redhat.com/browse/COST-4741) Add AWS usage type and operation columns to the Trino tables

@samdoran samdoran requested review from a team as code owners July 12, 2024 20:59
@samdoran samdoran added aws-smoke-tests pr_check will build the image and run aws + ocp on aws smoke tests trino migration Change requires a Trino migration labels Jul 12, 2024
@bacciotti
Copy link
Contributor

Maybe I'm missing something, but I don't get anything here
trino:org1234567> SELECT DISTINCT data_transfer_direction from aws_line_items;

But when I try the new columns, I have proper results:
SELECT DISTINCT lineitem_usagetype, lineitem_operation from aws_line_items;

@samdoran
Copy link
Contributor Author

samdoran commented Jul 15, 2024

Oops, I put the wrong column in the testing directions. I fixed it, but you were already looking for the right columns.

Copy link

codecov bot commented Jul 15, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.1%. Comparing base (64bdae1) to head (0d525cf).

Additional details and impacted files
@@           Coverage Diff           @@
##            main   #5216     +/-   ##
=======================================
- Coverage   94.1%   94.1%   -0.0%     
=======================================
  Files        371     371             
  Lines      30941   30941             
  Branches    4545    4545             
=======================================
- Hits       29114   29113      -1     
  Misses      1166    1166             
- Partials     661     662      +1     

@samdoran samdoran merged commit 9289c51 into main Jul 17, 2024
11 checks passed
@samdoran samdoran deleted the COST-3761/COST-4741-aws-post-processor-extra-columns branch July 17, 2024 14:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
aws-smoke-tests pr_check will build the image and run aws + ocp on aws smoke tests smokes-required trino migration Change requires a Trino migration
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants