COST-1075: implement improvements to ranking logic.

[blentz]

This PR fixes a bug that is present for queries that use resolution=daily, as described in COST-1075.

Previously, the behavior of filter[limit]=LIMIT&group_by[GROUP]=* did not constrain the number of groups in the whole response. There would be LIMIT number of elements in each per-day list, but the total number of groups in the response could exceed the LIMIT.

With this change, the response now ensures that only LIMIT number of groups are present in the whole response being returned.

Testing

Azure

Example query: LIMIT=5; curl -s -g "http://localhost:8000/api/cost-management/v1/reports/azure/costs/?filter[limit]=$LIMIT&filter[offset]=0&group_by[subscription_guid]=*&order_by[cost]=desc&end_date=2021-02-28&start_date=2021-02-01&filter[resolution]=daily" | jq '.'

Example output:
azure_query.txt

AWS

Example query: LIMIT=5; curl -s -g "http://localhost:8000/api/cost-management/v1/reports/azure/costs/?filter[limit]=$LIMIT&filter[offset]=0&group_by[region]=*&order_by[cost]=desc&end_date=2021-02-28&start_date=2021-02-01&filter[resolution]=daily" | jq '.'

Example output:
aws_query.txt

OCP + AWS

Example query: LIMIT=5; curl -s -g "http://localhost:8000/api/cost-management/v1/reports/openshift/infrastructures/aws/costs/?filter[limit]=$LIMIT&filter[offset]=0&group_by[project]=*&order_by[cost]=desc&end_date=2021-02-28&start_date=2021-02-01&filter[resolution]=daily" | jq '.'

Example output:
ocp_aws_query.txt

OCP

Test Data: 9 projects

Functional testing:

$ for LIMIT in $(seq 1 10); do curl -s -g "http://localhost:8000/api/cost-management/v1/reports/openshift/costs/?filter[limit]=$LIMIT&filter[offset]=0&group_by[project]=*&order_by[cost]=desc&end_date=2021-02-28&start_date=2021-02-01&filter[resolution]=daily" | jq '[.data[].projects[].project] | unique | length'; done
1
2
3
4
5
6
7
8
9
9

Example query: $ LIMIT=5; curl -s -g "http://localhost:8000/api/cost-management/v1/reports/openshift/costs/?filter[limit]=$LIMIT&filter[offset]=0&group_by[project]=*&order_by[cost]=desc&end_date=2021-02-28&start_date=2021-02-01&filter[resolution]=daily" | jq '.'

Example response:
query_json.txt

[codecov]

Codecov Report

Merging #2708 (9ac421f) into master (da715dc) will decrease coverage by 0.0%.
The diff coverage is 95.5%.

@@           Coverage Diff            @@
##           master   #2708     +/-   ##
========================================
- Coverage    94.6%   94.6%   -0.0%     
========================================
  Files         289     289             
  Lines       22484   22490      +6     
  Branches     2571    2583     +12     
========================================
+ Hits        21278   21280      +2     
- Misses        732     733      +1     
- Partials      474     477      +3     

[dlabrecq]

LGTM

[adberglund]

We have a handful of smoke tests failing on this. Essentially when we're returning daily data, and a particular day doesn't have an entry for one of the ranks, we need to add in a zeroed out or blank entry for it. This will be important for the table in the cost explorer so we always have an entry, even if it is 0. Happy to chat about this too.

[dccurtis]

If everything is set for this one. Lets run smokes locally one more time before merging.

[dccurtis]

Thanks for slogging through this one Brett!