Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

failed to translate metric from otel-python histogram using prometheusexporter #13443

Open
jaronoff97 opened this issue Aug 20, 2022 · 21 comments
Labels
bug Something isn't working exporter/prometheus never stale Issues marked with this label will be never staled and automatically removed priority:p2 Medium

Comments

@jaronoff97
Copy link
Contributor

Seeing this error message:

otel-collector_1  | 2022-08-20T22:03:26.430Z	error	[email protected]/accumulator.go:105	failed to translate metric	{"kind": "exporter", "data_type": "metrics", "name": "prometheus", "data_type": "\u0000", "metric_name": "graphql.api.request.time"}
otel-collector_1  | github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*lastValueAccumulator).addMetric
otel-collector_1  | 	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/accumulator.go:105
otel-collector_1  | github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*lastValueAccumulator).Accumulate
otel-collector_1  | 	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/accumulator.go:82
otel-collector_1  | github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).processMetrics
otel-collector_1  | 	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/collector.go:66
otel-collector_1  | github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*prometheusExporter).ConsumeMetrics
otel-collector_1  | 	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/prometheus.go:88

And it looks like this is due to a histogram metric being reported by the PeriodicExportingMetricReader exporting my histogram metric which hasn't had any data reported in the past period. Below is the output from a file exporter for some more debugging information. I believe this will be fixed by #9006 but let me know if that is incorrect.

Metrics dump
[
    {
        "resourceMetrics":
        [
            {
                "resource":
                {
                    "attributes":
                    [
                        {
                            "key": "telemetry.sdk.language",
                            "value":
                            {
                                "stringValue": "python"
                            }
                        },
                        {
                            "key": "telemetry.sdk.name",
                            "value":
                            {
                                "stringValue": "opentelemetry"
                            }
                        },
                        {
                            "key": "telemetry.sdk.version",
                            "value":
                            {
                                "stringValue": "1.12.0"
                            }
                        },
                        {
                            "key": "service.name",
                            "value":
                            {
                                "stringValue": "test"
                            }
                        }
                    ]
                },
                "scopeMetrics":
                [
                    {
                        "scope":
                        {
                            "name": "graphql-api",
                            "version": "1.0.0"
                        },
                        "metrics":
                        [
                            {
                                "name": "graphql.api.request.time",
                                "description": "Request time metrics for GraphQL API.",
                                "unit": "ms",
                                "histogram":
                                {
                                    "dataPoints":
                                    [
                                        {
                                            "attributes":
                                            [
                                                {
                                                    "key": "status",
                                                    "value":
                                                    {
                                                        "stringValue": "Success"
                                                    }
                                                }
                                            ],
                                            "startTimeUnixNano": "1660706168040247000",
                                            "timeUnixNano": "1660706170294892000",
                                            "count": "2",
                                            "sum": 0.0000030994415283203125,
                                            "bucketCounts":
                                            [
                                                "0",
                                                "2",
                                                "0",
                                                "0",
                                                "0",
                                                "0",
                                                "0",
                                                "0",
                                                "0",
                                                "0",
                                                "0"
                                            ],
                                            "explicitBounds":
                                            [
                                                0,
                                                5,
                                                10,
                                                25,
                                                50,
                                                75,
                                                100,
                                                250,
                                                500,
                                                1000
                                            ],
                                            "min": 0.0000011920928955078125,
                                            "max": 0.0000019073486328125
                                        }
                                    ],
                                    "aggregationTemporality": "AGGREGATION_TEMPORALITY_CUMULATIVE"
                                }
                            },
                            {
                                "name": "graphql.api.requests",
                                "description": "Usage metrics for GraphQL API.",
                                "unit": "1",
                                "sum":
                                {
                                    "dataPoints":
                                    [
                                        {
                                            "attributes":
                                            [
                                                {
                                                    "key": "status",
                                                    "value":
                                                    {
                                                        "stringValue": "Success"
                                                    }
                                                }
                                            ],
                                            "startTimeUnixNano": "1660706168040307000",
                                            "timeUnixNano": "1660706170294892000",
                                            "asInt": "2"
                                        }
                                    ],
                                    "aggregationTemporality": "AGGREGATION_TEMPORALITY_CUMULATIVE",
                                    "isMonotonic": true
                                }
                            }
                        ]
                    }
                ]
            }
        ]
    },
    {
        "resourceMetrics":
        [
            {
                "resource":
                {
                    "attributes":
                    [
                        {
                            "key": "telemetry.sdk.language",
                            "value":
                            {
                                "stringValue": "python"
                            }
                        },
                        {
                            "key": "telemetry.sdk.name",
                            "value":
                            {
                                "stringValue": "opentelemetry"
                            }
                        },
                        {
                            "key": "telemetry.sdk.version",
                            "value":
                            {
                                "stringValue": "1.12.0"
                            }
                        },
                        {
                            "key": "service.name",
                            "value":
                            {
                                "stringValue": "test"
                            }
                        }
                    ]
                },
                "scopeMetrics":
                [
                    {
                        "scope":
                        {
                            "name": "graphql-api",
                            "version": "1.0.0"
                        },
                        "metrics":
                        [
                            {
                                "name": "graphql.api.request.time",
                                "description": "Request time metrics for GraphQL API.",
                                "unit": "ms"
                            },
                            {
                                "name": "graphql.api.requests",
                                "description": "Usage metrics for GraphQL API.",
                                "unit": "1",
                                "sum":
                                {
                                    "dataPoints":
                                    [
                                        {
                                            "attributes":
                                            [
                                                {
                                                    "key": "status",
                                                    "value":
                                                    {
                                                        "stringValue": "Success"
                                                    }
                                                }
                                            ],
                                            "startTimeUnixNano": "1660706168040307000",
                                            "timeUnixNano": "1660706185345493000",
                                            "asInt": "2"
                                        }
                                    ],
                                    "aggregationTemporality": "AGGREGATION_TEMPORALITY_CUMULATIVE",
                                    "isMonotonic": true
                                }
                            }
                        ]
                    }
                ]
            }
        ]
    }
]
@jaronoff97
Copy link
Contributor Author

code example

    temporality_cumulative = {
        Counter: AggregationTemporality.CUMULATIVE,
        UpDownCounter: AggregationTemporality.CUMULATIVE,
        Histogram: AggregationTemporality.CUMULATIVE,
    }
    exporter = OTLPMetricExporter(insecure=True, preferred_temporality=temporality_cumulative)
    reader = PeriodicExportingMetricReader(
        exporter,
        export_interval_millis=15000,
    )
    provider = MeterProvider(metric_readers=[reader])
    set_meter_provider(provider)
...
        self.histogram = meter.create_histogram(
            "graphql.api.request.time",
            unit="ms",
            description="Request time metrics for GraphQL API.",
        )

if this is an issue with the python SDK please let me know :)

@jmacd
Copy link
Contributor

jmacd commented Aug 23, 2022

Note the error message {"kind": "exporter", "data_type": "metrics", "name": "prometheus", "data_type": "\u0000", "metric_name": "graphql.api.request.time"} contains two entries for "data_type", the second one being unrecognized. Seems like prometheusexporter is falling through into "failed to translate metric" where it might better print "unrecognized data type".

@jaronoff97
Copy link
Contributor Author

oh interesting! How can a map contain two entries for the same key? Either way, the issue seems to be that no data is associated with the second histogram point. Definitely some better error messages would be helpful though.

@jmacd
Copy link
Contributor

jmacd commented Aug 23, 2022

Maps are (for performance reasons) represented as lists in the OTLP protocol; and the duplicate is meant to be resolved by taking the last value, which is how protobuf defines the situation for maps as well. There's something incorrect about the data type here, but I wouldn't want to guess how come.

@github-actions
Copy link
Contributor

Pinging code owners: @Aneurysm9. See Adding Labels via Comments if you do not have permissions to add labels yourself.

@TylerHelmuth TylerHelmuth added bug Something isn't working priority:p2 Medium labels Aug 26, 2022
@zisom-hc
Copy link

zisom-hc commented Oct 11, 2022

I wanted to mention, in the case it may help in any way, that i'm also seeing this failed to translate metric error when using a different version of the prometheusexporter against metric data coming through an otlp-grpc receiver that's being provided data from an app through go.opentelemetry.io/otel/exporters/otlp/otlpgrpc.

If this should be reported through a new issue, or there's any other details y'all may be interested in let me know.

Here's an example of the error:

2022-10-07T19:47:36.788Z        error   [email protected]/accumulator.go:105   failed to translate metric      {"kind": "exporter", "data_type": "metrics", "name": "prometheus", "data_type": "\u0000", "metric_name": "tfc-agent.update-status.milliseconds"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*lastValueAccumulator).addMetric
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/accumulator.go:105
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*lastValueAccumulator).Accumulate
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/accumulator.go:82
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).processMetrics
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/collector.go:58
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*prometheusExporter).ConsumeMetrics
        github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/prometheus.go:88
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsRequest).export
        go.opentelemetry.io/[email protected]/exporter/exporterhelper/metrics.go:65
go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send
        go.opentelemetry.io/[email protected]/exporter/exporterhelper/common.go:225
go.opentelemetry.io/collector/exporter/exporterhelper.(*retrySender).send
        go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:147
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
        go.opentelemetry.io/[email protected]/exporter/exporterhelper/metrics.go:132
go.opentelemetry.io/collector/exporter/exporterhelper.(*queuedRetrySender).send
        go.opentelemetry.io/[email protected]/exporter/exporterhelper/queued_retry.go:83
go.opentelemetry.io/collector/exporter/exporterhelper.NewMetricsExporter.func2
        go.opentelemetry.io/[email protected]/exporter/exporterhelper/metrics.go:112
go.opentelemetry.io/collector/consumer.ConsumeMetricsFunc.ConsumeMetrics
        go.opentelemetry.io/[email protected]/consumer/metrics.go:36
go.opentelemetry.io/collector/processor/processorhelper.NewMetricsProcessor.func1
        go.opentelemetry.io/[email protected]/processor/processorhelper/metrics.go:69
go.opentelemetry.io/collector/consumer.ConsumeMetricsFunc.ConsumeMetrics
        go.opentelemetry.io/[email protected]/consumer/metrics.go:36
go.opentelemetry.io/collector/processor/batchprocessor.(*batchMetrics).export
        go.opentelemetry.io/[email protected]/processor/batchprocessor/batch_processor.go:297
go.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).sendItems
        go.opentelemetry.io/[email protected]/processor/batchprocessor/batch_processor.go:176
go.opentelemetry.io/collector/processor/batchprocessor.(*batchProcessor).startProcessingCycle
        go.opentelemetry.io/[email protected]/processor/batchprocessor/batch_processor.go:133

@github-actions
Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@mplachter
Copy link

mplachter commented May 23, 2023

Any update on this?

Still getting this error when using a GRPC Revicer and a Prometheus Exporter

standalone  | 2023-05-23T14:35:36.385Z  error   [email protected]/accumulator.go:105   failed to translate metric      {"kind": "exporter", "data_type": "metrics", "name": "prometheus", "data_type": "\u0000", "metric_name": "http.client.duration"}

@github-actions github-actions bot removed the Stale label May 26, 2023
@thesayyn
Copy link

thesayyn commented Jul 7, 2023

I am having the same issue with statsd receiver and Prometheus exporter.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 6, 2023

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@crobert-1
Copy link
Member

This is still an issue, as reported in #26725 for v0.85.0 of the collector.

@thesayyn
Copy link

I believe I have found the cause of this issue. looks like prometheus exporter doesn't support exponential histograms.

@vpmedia
Copy link

vpmedia commented Sep 29, 2023

Hi!

I've upgraded to v0.86.0 collector contrib version. Still having this issue:

otel_collector-1         | 2023-09-29T16:29:29.924Z	error	[email protected]/accumulator.go:94	failed to translate metric	{"kind": "exporter", "data_type": "metrics", "name": "prometheus", "data_type": "\u0000", "metric_name": "http.server.duration"}
otel_collector-1         | github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*lastValueAccumulator).addMetric
otel_collector-1         | 	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/accumulator.go:94
otel_collector-1         | github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*lastValueAccumulator).Accumulate
otel_collector-1         | 	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/accumulator.go:71
otel_collector-1         | github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*collector).processMetrics
otel_collector-1         | 	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/collector.go:92
otel_collector-1         | github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*prometheusExporter).ConsumeMetrics
otel_collector-1         | 	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/prometheus.go:85
otel_collector-1         | go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsRequest).Export
otel_collector-1         | 	go.opentelemetry.io/collector/[email protected]/exporterhelper/metrics.go:60
otel_collector-1         | go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send
otel_collector-1         | 	go.opentelemetry.io/collector/[email protected]/exporterhelper/timeout_sender.go:41
otel_collector-1         | go.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send
otel_collector-1         | 	go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:40
otel_collector-1         | go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
otel_collector-1         | 	go.opentelemetry.io/collector/[email protected]/exporterhelper/metrics.go:176
otel_collector-1         | go.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send
otel_collector-1         | 	go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:40
otel_collector-1         | go.opentelemetry.io/collector/exporter/exporterhelper.(*baseExporter).send
otel_collector-1         | 	go.opentelemetry.io/collector/[email protected]/exporterhelper/common.go:202
otel_collector-1         | go.opentelemetry.io/collector/exporter/exporterhelper.NewMetricsExporter.func1
otel_collector-1         | 	go.opentelemetry.io/collector/[email protected]/exporterhelper/metrics.go:100
otel_collector-1         | go.opentelemetry.io/collector/consumer.ConsumeMetricsFunc.ConsumeMetrics
otel_collector-1         | 	go.opentelemetry.io/collector/[email protected]/metrics.go:25
otel_collector-1         | go.opentelemetry.io/collector/internal/fanoutconsumer.(*metricsConsumer).ConsumeMetrics
otel_collector-1         | 	go.opentelemetry.io/[email protected]/internal/fanoutconsumer/metrics.go:69
otel_collector-1         | go.opentelemetry.io/collector/processor/batchprocessor.(*batchMetrics).export
otel_collector-1         | 	go.opentelemetry.io/collector/processor/[email protected]/batch_processor.go:442
otel_collector-1         | go.opentelemetry.io/collector/processor/batchprocessor.(*shard).sendItems
otel_collector-1         | 	go.opentelemetry.io/collector/processor/[email protected]/batch_processor.go:256
otel_collector-1         | go.opentelemetry.io/collector/processor/batchprocessor.(*shard).start
otel_collector-1         | 	go.opentelemetry.io/collector/processor/[email protected]/batch_processor.go:218

I'm happing to give additional information if required.
My stack is: Collector-Contrib running inside a Docker container, My server is using the Python SDK / Flask-Instrumentation.

@crobert-1
Copy link
Member

Another user has hit this in #25146.

@deathsaber
Copy link

I am using gauges and I am running into the same issue.

@mdio
Copy link

mdio commented Dec 14, 2023

I'm getting the same with opentelemetry-collector:0.91.0 and Node.js with @opentelemetry/* latest (1.18.1/0.45.1) when trying to use an exponential histogram (ExponentialHistogramAggregation).

error   [email protected]/accumulator.go:94    failed to translate metric      {"kind": "exporter", "data_type": "metrics", "name": "prometheus", "data_type": "\u0004", "metric_name": "http_request_duration_seconds"}

@rockyburt
Copy link

rockyburt commented Dec 18, 2023

I am also seeing this problem with several different metrics. Tried upgrading to 0.91.0 and observing the following on local docker deployments as well as Kubernetes deployments on AWS/EKS..

error   [email protected]/accumulator.go:94    failed to translate metric      {"kind": "exporter", "data_type": "metrics", "name": "prometheus", "data_type": "\u0000", "metric_name": "http.server.request.size"}
 github.com/open-telemetry/opentelemetry-collector-contrib/exporter/prometheusexporter.(*lastValueAccumulator).addMetric
         github.com/open-telemetry/opentelemetry-collector-contrib/exporter/[email protected]/accumulator.go:94

Copy link
Contributor

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

See Adding Labels via Comments if you do not have permissions to add labels yourself.

@github-actions github-actions bot added the Stale label Feb 19, 2024
@crobert-1 crobert-1 added never stale Issues marked with this label will be never staled and automatically removed and removed Stale labels Feb 26, 2024
@galihputera
Copy link

getting same error with @mdio "data_type": "\u0004"

@MilosMoravac
Copy link

getting same error with @mdio "data_type": "\u0004"

Same here!

@pedrodiascoelho
Copy link

I believe I have found the cause of this issue. looks like prometheus exporter doesn't support exponential histograms.

Yup I have realized the same at least for python. I got confused by this talk https://www.youtube.com/watch?v=W2_TpDcess8 where they say that they have it implemented, although I was only able to find such implementation in go here: https://github.com/open-telemetry/opentelemetry-go-contrib/blob/6e79f7ca22d58345e2ccfe3f9e8bb4ad71633ab3/bridges/prometheus/producer.go#L170C6-L170C33. It seems that the python exporter does not have such implementation: https://github.com/open-telemetry/opentelemetry-python/blob/main/exporter/opentelemetry-exporter-prometheus/src/opentelemetry/exporter/prometheus/__init__.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working exporter/prometheus never stale Issues marked with this label will be never staled and automatically removed priority:p2 Medium
Projects
None yet
Development

No branches or pull requests