Automatically generated metrics and labels by prometheus are dropped by the prometheus receiver

[crsandeep]

Describe the bug
Labels and metrics that are automatically generated by prometheus are dropped by the prometheus receiver. Example - metric up and scrape_duration_seconds (few other scape_ metrics) and labels job and instance. Is this intentional or a potential bug?

Steps to reproduce
I set up a pipeline with prometheus receiver and prometheusremotewrite. Config file is provided below.

What did you expect to see?
I expected to see the metrics and labels mentioned here

What did you see instead?
Above metrics and labels were dropped.

What version did you use?
Version: 0.17.0

What config did you use?

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-collector'
          scrape_interval: 10s
          static_configs:
            - targets: [ '0.0.0.0:8888' ]

exporters:
  prometheusremotewrite:
    endpoint: "http://localhost:1234/receive"

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      exporters: [prometheusremotewrite]

Additional context
I briefly looked into the code and I observed below

• Prometheus generated labels seems to be dropped here
• Prometheus generated metrics seems to be dropped here

[crsandeep]

@alolita If these labels and metrics are being dropped intentionally then is it fine if I file a PR to make it config driven to not block these when a particular config is set?

[gillg]

In fact the receiver doesn't drop job and instance but it not appends them to metrics in the scapping process.
The use case can make sense because you can use the prometheus exporter and the final prometheus server will add them at the end (with less granularity).

In my use case, I would at least expose the job, to avoid metrics conflicts (for example go_memstats_alloc_bytes is common to almost all exporters, so the value collected by otel collector make no sense without job) and to continue to use "standard" grafana dashboards. Most of them ar based on job name.

For now my workaround is :

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-collector'
          scrape_interval: 5s
          static_configs:
            - targets: ['otel-collector:8888']
          relabel_configs:
          - action: replace
            replacement: otel-collector
            target_label: job
        - job_name: node-exporter
          static_configs:
            - targets: ['node-exporter:9100']
          relabel_configs:
          - action: labelmap
            regex: __(.+)
          # Trick because otel collector not expose the job
          - action: replace
            replacement: node-exporter
            target_label: job
        - job_name: es-exporter
          static_configs:
            - targets: ['es-exporter:9114']
          relabel_configs:
          # Trick because otel collector not expose the job
          - action: replace
            replacement: es-exporter
            target_label: job

By this way I federate 3 exporters on OTEL Collector and I expose all metrics with the prometheus exporter in only one endpoint.

What could be great :

• Add labels __meta_otel_job_name and __meta_otel_instance which contains the same values that prometheus put in job and instance during its scrapping.
  With this, we could write relabeling_rules as we want, for some jobs or all.
• Add a global option receivers.prometheus.expose_job: (tue|false) which could by default add the job as label for all scrap configs
  The instance label takes less sense in my opinion to be added at the otel receiver level. And with the __meta_otel_instance we are free. It's really common to relabel the instance during the scrapping (see https://prometheus.io/docs/guides/multi-target-exporter/ as exemple).
• Why not... add a global option receivers.prometheus.job_label: xxxxx to change the default name "job" to something else. It could be usefull if you want avoid the option honor_labels: true at the prometheus server side.

EDIT: In another task, it could be usefull to append automaticaly the metric up (0|1) depending on the scrapping success or not ! Don't you think ?

[gillg]

For information, the native job adding is done here : https://github.com/prometheus/prometheus/blob/df80dc4d3970121f2f76cba79050983ffb3cdbb0/scrape/target.go#L328

[gillg]

In fact... I'm wrong about something in my previous assumption 🤔
I don't undestand why but, if I relabel to the target label "job" it's not displayed in metrics, but if I call it "job2" or "job_name" it appears. So, yes otel-collector seems drop by any way this label. But I don't understand where in code.

[rakyll]

Duplicate of open-telemetry/prometheus-interoperability-spec#37.