Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

otelcol.exporter.prometheus: Make histogram counts cumulative for Prometheus #4196

Merged
merged 7 commits into from
Jun 21, 2023

Conversation

grcevski
Copy link
Contributor

@grcevski grcevski commented Jun 16, 2023

PR Description

This PR attempts to fix the problem with non-cumulative histograms exported to Prometheus, from the otel metrics collector. The code change follows the approach taken by the opentelemetry-prometheus-exporter as shown here:
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/prometheusexporter/collector.go#LL224C76-L224C76

Given a number of histogram buckets, we first sort the buckets by the bucket bounds value, then we calculate the cumulative value for each bucket, instead of using the exact bucket count provided by OTel.

Which issue(s) this PR fixes

Fixes #4193

Notes to the Reviewer

I'd appreciate any feedback on where to add tests for this issue.

PR Checklist

  • CHANGELOG updated
  • Documentation added
  • Tests updated

@grcevski grcevski requested a review from a team as a code owner June 16, 2023 20:46
@CLAassistant
Copy link

CLAassistant commented Jun 16, 2023

CLA assistant check
All committers have signed the CLA.

@tpaschalis
Copy link
Member

fix the problem with non-cumulative histograms exported to Prometheus, from the otel metrics collector

So to better understand, this is for cumulative temporality histograms (we're dropping delta temporality histograms for now as we haven't implemented the delta-to-cumulative conversion), but you mean we're fixing the cumulative distribution inside of the buckets, right?

Copy link
Member

@tpaschalis tpaschalis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Surprising from the OTel side, but it is what it is; LGTM at a first glance.

Let's fix the test and I'll leave the final ✅ to @rfratto who wrote the converter


sort.Float64s(bounds)

// Calculate cumulative count value as Prometheus wants it
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think there's a way to reword here so that it's clearer we don't refer to cumulative temporality but so that we make it clear it's about the way that Prometheus uses cumulative buckets for histograms?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done! Thanks.

Copy link
Member

@rfratto rfratto left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, I see where I got my logic wrong here.

Can you add a bugfix entry to the CHANGELOG before we merge this? Thanks!

@grcevski
Copy link
Contributor Author

Thanks @tpaschalis and @rfratto! I've updated the changelog and resolved conflicts with main.

@tpaschalis tpaschalis merged commit 432127e into grafana:main Jun 21, 2023
captncraig added a commit that referenced this pull request Jun 27, 2023
* otelcol.exporter.prometheus: Make histogram counts cumulative for Prometheus (#4196)

* Use sha256 for rpm signing to allow installation on fips-enabed systems (#4268)

* Use sha256 for rpm digest signing to allow installation on fips-enabled systems

* add changelog

* fix startup order for prometheus.operator.servicemonitors. Fixes #4142 (#4146)

* update versions for 0.34.3

* build: update to go 1.20.5

---------

Co-authored-by: Nikola Grcevski <[email protected]>
Co-authored-by: kfriedrich123 <[email protected]>
@github-actions github-actions bot added the frozen-due-to-age Locked due to a period of inactivity. Please open new issues or PRs if more discussion is needed. label Feb 23, 2024
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Feb 23, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
frozen-due-to-age Locked due to a period of inactivity. Please open new issues or PRs if more discussion is needed.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

otelcol.exporter.prometheus: Histograms are not exported with cumulative counts
4 participants