[Work_Item] Revise glossary definition for FOCUS dataset to include provider columns

[ijurica]

1. Problem Statement *

Describe the problem, issue, use case, or opportunity that this work item addresses.
Include practitioner quotes illustrating real examples a) of questions being asked by practitioners and b) value unlocked by answering these questions, if available.

• What is the problem?: Explain the context and why it needs resolution.
• Impact: Describe how the problem affects users, systems, or the project.

The current FOCUS dataset specification does not clarify whether custom/native provider-specific columns (prefixed with x_) that contain essential information not covered by standard FOCUS columns should be included in the FOCUS dataset.

Due to this omission, some providers who support FOCUS might continue to provide key data only in separate provider-specific cost and usage datasets. If that happens, practitioners will encounter challenges, as there is no reliable way to correlate charge records between the FOCUS dataset and native datasets.

Use Case

Follow instructions on how to craft a use case

2. Objective *

State the objective of this work item. What outcome is expected?

• Success Criteria: Define how success will be measured (e.g. metrics and KPIs).

The objective is to revise the FOCUS dataset specification to clarify that custom/native columns (prefixed with x_), which provide crucial information unavailable in standard FOCUS columns, should be included in the FOCUS dataset. This will ensure consistency across providers' FOCUS datasets.

Success Criteria:

• Clear specification: The FOCUS dataset glossary will be updated to explicitly state that such columns should be included where they add value beyond standard FOCUS columns.
• Inclusion of custom columns: Custom provider-specific columns (x_ prefixed) that offer essential unavailable in standard FOCUS columns data will be integrated into the FOCUS dataset produced by all providers.

3. Supporting Documentation *

Include links to supporting documents such as:

• Data Examples: [Link to data or relevant files; DO NOT share proprietary information]
• Related Use Cases or Discussion Documents: [Link to discussion]
• PRs or Other References: [Link to relevant references]

For example, in Oracle Cloud Infrastructure (OCI), key dimensions such as lineItem/referenceNo, lineItem/backReference, and product/compartmentId provide valuable insights. However, these fields are not included in the OCI FOCUS dataset and are only available in the native OCI Cost and Usage reports.

4. Proposed Solution / Approach

Outline any proposed solutions, approaches, or potential paths forward. Do not submit detailed solutions; please keep suggestions high-level.

• Initial Ideas: Describe potential solution paths, tools, or technologies.
• Considerations: Include any constraints, dependencies, or risks.
• Feasibility: Include any information that helps quantify feasibility, such as perceived level of effort to augment the spec, or existing fields in current data generator exports.
• Benchmarks: Are there established best practices for solving this problem available to practitioners today (e.g. mappings from existing CSP exports that are widely used)?

Potential solution:

• Update the FOCUS specification, particularly the FOCUS dataset glossary term, to
  • Clarify that custom or native provider columns (prefixed with x_) should be included in the FOCUS dataset when they provide additional information not captured by the existing FOCUS columns.
  • Specify that if the introduction of a custom column might result in splitting original charge records into multiple entries, providers must consider the specified aggregation-related requirements for metric columns (such as costs and quantities). The values in the resulting charge records must be recalculated appropriately to prevent aggregation errors. Providers must ensure that the FOCUS dataset conforms to the specified aggregation-related requirements for metric columns, particularly costs and quantities.

Considerations:

• This change may require updates to the data generation processes of providers that already support the FOCUS specification.

Feasibility:

• Discussing this topic and introducing this change (assuming we reach consensus) into the specification shouldn't require too much time or effort.

5. Epic or Theme Association

This section will be completed by the Maintainers.

• Epic: [Epic Name]
• Theme: [Theme Name, if applicable]

TBD

6. Stakeholders *

List the main stakeholders for this issue.

• Primary Stakeholder: [Name/Role]
• Other Involved Parties: [Names/Roles]

TBD

[ijurica]

Adding link to a comment by @ahullah that was originally posted on the related issue: #602 (comment)

[shawnalpay]

Maybe this is tough to phrase, but: is there a use case you can think of for this one, @ijurica?

[jpradocueva]

Notes from the Maintainers' call on November 4:

Context: Updating block-three definitions to include additional provider-specific columns allows the spec to better accommodate unique provider requirements and reporting structures.
Level of Effort Required: Trivial — Expanding block-three definitions involves restructuring the spec to integrate new columns, requiring coordination with providers to ensure accuracy and relevance.

[jpradocueva]

Summary from the Maintainers' call on Nov 25:

Context:
This work item seeks to expand the glossary definitions to explicitly cover provider-specific columns, improving clarity and alignment with the dataset structure.
Maintainers Assigned:
Irena.
Task Force Assigned:
Task Force 1 (TF1).

[jpradocueva]

Action Items from the Members' call on December 5:

• [#617] Irena @ijurica : Draft initial glossary revisions for provider columns and circulate for review.
• [#617] Irena @ijurica : Ensure alignment of glossary definitions with broader normative requirement updates under [Work_Item] Augment support for correction handling #556.
• [#617] All members: Review and provide feedback on the draft revisions before the next meeting.
• [#617] Chris @cnharris10 : Collaborate with CSP representatives to incorporate specific examples of provider column usage.

[jpradocueva]

Action Items from the Maintainers' call on December 16:

• [#617] Irena @ijurica : Draft an initial version of the updated glossary definition when discussions resume.

[jpradocueva]

Action Items from the Maintainers' call on Jan 6:

• [[Work_Item] Revise glossary definition for FOCUS dataset to include provider columns #617] Irena - Complete the revision of the glossary definition to include provider columns within the next two weeks.

[ijurica]

Note:
During one of the meetings, it was suggested that, in addition to defining this in the FOCUS dataset glossary term, we should consider mentioning it explicitly in the Introduction and/or adding an appendix highlighting that custom/native provider-specific columns (prefixed with x_) containing essential information not covered by standard FOCUS columns should also be included in the FOCUS dataset.

[ijurica]

The objective of this Work Item is to update the FOCUS specification, clarifying expectations regarding the inclusion of custom/native provider-specific columns (prefixed with x_) in the FOCUS dataset, while ensuring compliance with aggregation-related requirements for metric columns (e.g., costs, quantities) to prevent aggregation errors.

1. Min requirement:
   The update to the FOCUS dataset glossary term, addressing the inclusion of x_ columns, while ensuring compliance with aggregation-related requirements for metric columns, is covered in PR FOCUS #617: FOCUS dataset - glossary term update #682.

2. Nice to have:
   A clarification regarding the inclusion of custom/native provider-specific columns (prefixed with x_), while ensuring compliance with aggregation-related requirements, may be included in the Introduction and/or a new appendix as well.
   Note: This requires further discussion by the TF-1 team to determine whether to proceed with this addition or if the Work Item can be closed once the glossary term update (point 1) is finalized.

[jpradocueva]

Reopen after merging PR #682