r.2024.07.18.0

What's Changed

• [Tech-Debt] Remove dead code from Trino upgrade. by @myersCody in #5153
• COST-5252 cache schema/table exists by @maskarb in #5213
• COST-5093 Update mapped tags in EC2 summary table by @djnakabaale in #5147
• COST-4912 Calculate node based unattributed storage (Azure) by @myersCody in #5066
• COST-5198 Only store user object on write requests by @chambridge in #5214
• COST-5125 Trino migration command improvements by @samdoran in #5191
• COST-4741 Add UsageType and Operation to daily data by @samdoran in #5216
• COST-5114 Add type ahead support for the new EC2 fields by @djnakabaale in #5159
• COST-5287 Prevent azure cost deduplication on disk-id resources by @myersCody in #5219
• COST-4283 OCP: set start-date based on data in file being processed by @maskarb in #5212

Full Changelog: r.2024.07.12.0...r.2024.07.18.0

r.2024.07.12.0

Summary:

Handle exception when querying Azure for VM ID, add csi_volume_handle back in SQL, enable using readreplica db, dependency updates.

What's Changed

• COST-5198 - split read traffic to read replica db using nginx proxy by @chambridge in #5188
• remove unused method by @maskarb in #5208
• Bump certifi from 2024.6.2 to 2024.7.4 in the pip group across 1 directory by @dependabot in #5207
• chore(image): update base image by @github-actions in #5203
• Handle case when resource ID cannot be obtained by @chambridge in #5209
• COST-5148 update EC2 compute insert sql by @djnakabaale in #5206
• Unpause the csi volume handle sql by @myersCody in #5175
• [COST-5244, COST-5245, COST-5246, COST-5247] dependency update by @maskarb in #5211
• COST-4981 reset local unleash by @maskarb in #5148

Full Changelog: r.2024.07.08.0...r.2024.07.12.0

r.2024.07.08.0

Summary:

Upgrade to Python 3.11, adjustment of logs, and logging of S3 batch deletes. Corrections in VM usage reporting, and processing of OCPGCP network data.

What's Changed

• COST-5076 upgrade to python 3.11 by @maskarb in #4444
• COST-5228 log outside for loop by @djnakabaale in #5202
• log s3 batch deletes by @lcouzens in #5204
• COST-5219 Correctly report VM usage for metering when billing record is split by @cgoodfred in #5201
• COST-4745 OCPGCP Network data processing SQL by @cgoodfred in #5058

Full Changelog: r.2024.07.04.0...r.2024.07.08.0

r.2024.07.04.0

Summary:

Fix and improve S3 operations, Docker build improvements, and filtering optimization changes.

What's Changed

• COST-5213 - fix S3 prepare by @lcouzens in #5194
• COST-5214 Remove directive to make the build work with docker by @samdoran in #5196
• COST-5216 Delete filtering optimization by @myersCody in #5197
• Revert "[COST-5216] Delete filtering optimization" by @myersCody in #5200
• COST-5226 - Skip S3 delete (daily flow) if we have marked deletion complete. by @lcouzens in #5198
• COST-5213 - Improve collection for delete from S3. by @lcouzens in #5194

Full Changelog: r.2024.06.28.0...r.2024.07.04.0

r.2024.06.28.0

Summary:

Prevent downloads queuing for a provider when a current download is already in progress

What's Changed

• COST-4844 Tweak serializer for storageclass order by @myersCody in #5174
• COST-5078 Switch to RHEL 8 VMs for CI jobs by @samdoran in #5193
• COST-5180 Only attempt to fetch new reports if current processing is complete
  by @lcouzens in #5181
• COST-5214 Update containerfile to build with podman by @samdoran in #5195

Full Changelog: r.2024.06.27.0...r.2024.06.28.0

r.2024.06.27.0

Summary:

Made a few changes to data processing to ensure tasks are sent to the correct workers and that task caching works as expected.

What's Changed

• Improve processing times by using queues more efficiently by @cgoodfred in #5186
• COST-5196 Improve processing times by using queues more efficiently by @lcouzens in #5187
• COST-5176 Correctly pass context dictionary within log_json function call by @djnakabaale in #5182
• COST-5178 Improve processing S3 deletions by @lcouzens in #5180
• Bump urllib3 from 1.26.18 to 1.26.19 in the pip group across 1 directory by @dependabot in #5172
• Add flower as a development dependency by @samdoran in #5189

Full Changelog: r.2024.06.24.0...r.2024.06.27.0

r.2024.06.24.0

Summary:

Optimizations in SQL and cost model task, and other operational improvements.

What's Changed

• [COST-4916] add v3.3.0 operator commits by @maskarb in #5143
• [COST-5124] Improve Trino migration management command by @samdoran in #5163
• Filter accounts by matching criteria during subs processing by @cgoodfred in #5184
• Update tasks.py by @lcouzens in #5185
• clean up grafana dashboard by @lcouzens in #5183

Full Changelog: r.2024.06.21.0...r.2024.06.24.0

r.2024.06.21.0

Summary:

Improvements to Trino migrations, a new data transfer direction column added to Trino tables for GCP and AWS, and a new queue to better distribute processing tasks.

What's Changed

• [COST-5141] Add command to remove expired hive partitions by @myersCody in #5166
• Update GitHub actions by @samdoran in #5165
• [COST-5141] Add group by to the outer query by @myersCody in #5170
• [COST-5141] Fix management command to use continue instead of return. by @myersCody in #5173
• [COST-5128] Process new subs tag to identify non-converted instances by @cgoodfred in #5162
• [COST-4745] Add data_transfer_direction to GCP network costs to Trino tables by @samdoran in #5130
• [COST-4741] Add data_transfer_direction for AWS network costs to Trino tables by @samdoran in #5129
• [COST-5168] - Adding new penalty pipeline by @lcouzens in #5176
• Improve our logging readability by @lcouzens in #5178
• add prometheus metrics for new queues by @lcouzens in #5179

Full Changelog: r.2024.06.17.0...r.2024.06.21.0

r.2024.06.17.0

Summary:

One new enhancement to enable saving plan costs for OCP on AWS calculations. Other minor bug fixes and enhancements detailed beow.

What's Changed

• Bump the pip group across 1 directory with 2 updates by @dependabot in #5161
• COST-5072 Check schema exists prior to querying it during HCS processing by @cgoodfred in #5137
• COST-5097 Format OCP data transfer fields like other usage fileds by @samdoran in #5152
• Remove suffix from masu trino endpoint by @myersCody in #5154
• catch None object by @maskarb in #5138
• COST-4966 Enable Savings plan costs for OCP on AWS calculations
• masu - Account for intermediate None values when getting cost model by @samdoran in #5167
• Update disable_source to work for tasks already in the download queue. by @myersCody in #5168

Full Changelog: r.2024.06.11.0...r.2024.06.17.0

r.2024.06.11.0

Summary:

Work around an issue in Azure with V2 cost exports by specifying the API version in the client.
Disable use of a new column in Trino tables due to incomplete data migrations. This will be reenabled once the Trino migrations complete successfully.

What's Changed

• Use newer API version for Azure Cost Management client by @samdoran, @lcouzens in #5158
• COST-5124 Pause use of csi_volume_handle in SQL by @myersCody in #5160

Full Changelog: r.2024.06.10.0...r.2024.06.11.0