[COST-4741] - Separate AWS node network costs into a Network unattributed namespace

[samdoran]

Jira Ticket

COST-4741

Description

Separates out AWS node network costs into a separate project for later distribution.

Testing

1. Load OCP on AWS test data.
   This is currently broken. You will need to run the second command twice.

   make create-test-customer
   make load-test-customer-data test_source=aws
   

2. Inspect data in Trino. Where product_family is Data Transfer and product_code is AmazonEC2, there should be a data transfer direction

   SELECT namespace,product_code,product_family,data_transfer_direction,usage_amount FROM reporting_ocpawscostlineitem_project_daily_summary;
   

   namespace                 |  product_code  |   product_family   | data_transfer_direction |     usage_amount
   --------------------------+----------------+--------------------+-------------------------+-----------------------
    cost-management          | AmazonEC2      | Compute Instance   | NULL                    |                   8.0
    Network unattributed     | AmazonEC2      | Data Transfer      | IN                      |   0.05788159130273985
    Network unattributed     | AmazonEC2      | Data Transfer      | IN                      |   0.08385133566183305
    Network unattributed     | AmazonEC2      | Data Transfer      | OUT                     | 4.0166646410467517E-4
    Network unattributed     | AmazonEC2      | Data Transfer      | IN                      |  0.013915647933780239
    Platform unallocated     | AmazonEC2      | Data Transfer      | NULL                    |   0.04199137044014825
    Network unattributed     | AmazonEC2      | Data Transfer      | OUT                     |  0.050977435375557656
   ...
   

3. Verify there are records for Network unattributed in PostgreSQL AWS daily summary table

   SELECT * FROM reporting_ocpawscostlineitem_project_daily_summary_p WHERE namespace = 'Network unattributed';
   

4. Verify OpenShift costs are correct

   curl http://localhost:8000/api/cost-management/v1/reports/openshift/costs/
   

5. Verify AWS filtered by OpenShift costs are correct

   http://localhost:8000/api/cost-management/v1/reports/openshift/infrastructures/aws/costs/
   

6. Compare Trino and PostgreSQL data to make sure costs add up:

   postgres=# SELECT sum(unblended_cost) from reporting_aws_cost_summary_p
   sum
   -----------------
    23785.362736418
   
   trino:org1234567> SELECT sum(lineitem_unblendedcost) from aws_line_items;
   _col0
   --------------------
    23785.362736451105
   (1 row)
   

Release Notes

• proposed release note

* [COST-3761](https://issues.redhat.com/browse/COST-3761) Separate AWS EC2 instance network costs into a `Network unattributed` project
  Network costs were previously combined with the AWS EC2 instance costs. Now those costs presented in a new `Network unattributed` project.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.1%. Comparing base (ddfeb17) to head (1db894e).

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #5104     +/-   ##
=======================================
- Coverage   94.1%   94.1%   -0.0%     
=======================================
  Files        373     373             
  Lines      31217   31217             
  Branches    4591    4591             
=======================================
- Hits       29378   29374      -4     
- Misses      1172    1175      +3     
- Partials     667     668      +1     

[cgoodfred]

Just a few quick questions/comments

[cgoodfred]

This SQL is grouped by these items below, are these enough to ensure a unique data_transfer_direction or should we be grouping by the data_transfer_direction result as well?

GROUP BY aws.lineitem_usagestartdate,
    aws.lineitem_resourceid,
    4, -- CASE satement
    aws.product_productfamily,
    aws.product_instancetype,
    aws.lineitem_availabilityzone,
    aws.product_region,
    aws.resourcetags,
    aws.costcategory

[samdoran]

We do probably need to group by the direction, but we can't use an aggregation in a group by. Grouping by aws.lineitem_usagetype may work.

[cgoodfred]

You can also use numbers to group by items in the SELECT statement, for example the 4 in the group by there is based on the CASE statement in the SQL