Prometheus export from OTLP receiver not functional

[jmacd]

Describe the bug
Discussed under open-telemetry/opentelemetry-go#875, we have evidence that OTLP metrics data are being transformed incorrectly, such that the Prometheus exporter will:

• report Counter data as "gauge"
• drop Summary data (which the OTel-Go SDK is exporting for ValueRecorder, ValueObsever)

Steps to reproduce
The steps are described in open-telemetry/opentelemetry-go#875.

What did you expect to see?
Counter data should report as counter, ValueRecorder data should report as gauge.

What did you see instead?
Counters are labeled gauges, and no gauges appear.

What version did you use?
0.4

What config did you use?
This is also linked in open-telemetry/opentelemetry-go#875

Environment
Dockerized alpine

Additional context
I will self-assign this. I am working on it, just want it recorder here that Prometheus exporter is not functional with OTLP data due to a translation problem that I am investigating.

[huyan0]

I noticed the underlying prometheus-go-metric-exporter that the Collector's Prometheus exporter uses drop Summary when converting to Prometheus type. For some reason I thought it was a feature. Also, it doesn't care about the semantic difference and reports instantaneous adding scalar as a gauge.

Seems like the original prometheus-go-metric-exporter is moved to a different repo and support for Summary type is in progress .

[jmacd]

The big issue with prometheus-go-metric-exporter is that it uses the OpenCensus format, and we're migrating to OTLP. I am not certain what the best option is. A bigger question is which library will support Prometheus push: see #1150.

[jmacd]

I don't think we can sort this out until OTLP is settled, but it's fine to assign this to me.

[tigrannajaryan]

This does not appear to be a dealbreaker for 1.0, I am removing from the milestone, feel free to object.

[alolita]

Hi @tigrannajaryan Support of the summary metric type in OTLP is essential for supporting Prometheus metric types. The Prometheus pipeline is not complete without this support. Currently the default OTLP behavior is to just drop the metric if it is a summary type and the information gets lost at this point. See https://github.com/open-telemetry/opentelemetry-collector/blob/master/translator/internaldata/oc_to_metrics.go#L227-L228 Also please see https://prometheus.io/docs/concepts/metric_types/#summary
Since summary is not supported in OTLP, any summary type metric gets dropped - see code example below
https://github.com/open-telemetry/opentelemetry-collector/blob/master/exporter/prometheusremotewriteexporter/exporter.go#L126

This is a blocker for the Prometheus remote write exporter and the Prometheus exporter. I would consider it a show-stopper for OTLP.

[jmacd]

I think we can solve this. We do not have a single data point type that represents summaries, but we can generate several gauges and counters to represent a summary so that the Prometheus remote-write output data will be correct. Prometheus uses a metric name suffix ("_count" an integer, "_sum" a floating point) for two counters, and it uses a gauge for point each quantile (e.g., "quantile=0.5", "quantile=0.9"). This representation is not efficient compared with a compact Summary data point, but no matter how we represent this data in OTLP, two counters and one gauge per quantile are what a Prometheus user expects to receive.

[mtwo]

Following up from the specs SIG meeting. @alolita from AWS will create a proposal for this.

@jmacd can you re-assign to @alolita?

[tigrannajaryan]

@alolita as per @mtwo 's comment I assigned this to you.

[alolita]

@tigrannajaryan Will follow up with an implementation proposal shortly.

[amanbrar1999]

@jmacd Resolving the summary part of this issue is currently happening, but does the issue of counter data being reported as gauge still exist?

[amanbrar1999]

My understanding of this is that counter data being reported as gauge is a direct consequence of prometheus exporter exporting delta metrics, when it currently only functions correctly when exporting cumulative metrics. This specific part of this issue should be partially resolved when this spec issue is resolved: open-telemetry/opentelemetry-specification#731

It will only be completely resolved when there is functionality to convert delta metrics to cumulative metrics: https://github.com/open-telemetry/opentelemetry-collector/issues/1422

[andrewhsu]

from the metrics sig mtg today, discussed this issue and decided to close