label dimensions inconsistent with previously collected metrics

[boazjohn]

I'm getting the following error when I try localhost:9122/metrics

Guess this is because of empty or conflicting values?

* collected metric processes_unknown label:<name:"dc" value:"ap" > label:<name:"host" value:"ip-172-16-4-63" > untyped:<value:0 >  has label dimensions inconsistent with previously collected metrics in the same metric family
* collected metric net_err_in label:<name:"dc" value:"ap" > label:<name:"host" value:"ip-172-16-33-127" > label:<name:"interface" value:"eth0" > label:<name:"rack" value:"logs" > untyped:<value:0 >  has label dimensions inconsistent with previously collected metrics in the same metric family
* collected metric kernel_interrupts label:<name:"dc" value:"ap" > label:<name:"host" value:"ip-172-16-3-243" > untyped:<value:2.2162444e+07 >  has label dimensions inconsistent with previously collected metrics in the same metric family
* collected metric procstat_cpu_time_system label:<name:"dc" value:"ap" > label:<name:"host" value:"ip-172-16-34-34" > label:<name:"pidfile" value:"/var/run/tomcat7.pid" > label:<name:"process_name" value:"java" > label:<name:"rack" value:"logs" > untyped:<value:23071.55 >  has label dimensions inconsistent with previously collected metrics in the same metric family

[boazjohn]

Found a fix for this. Made changes similar to this:
https://github.com/kubernetes/kube-state-metrics/pull/194/files#diff-1c14cc7ce5782cee84227eef5b3358f6

This is working now by removing checkMetricConsistency

[matthiasr]

Yes, this can happen if the same metric is submitted with different label sets. Is this something that happens persistently? Could it be avoided by coupling the exporter lifetime to the lifetime of the source (e.g. in Kubernetes, I would run them in one pod)? That's generally the best way to run exporters and avoids issues if you add or remove a label in your code.

[matthiasr]

I labeled this "enhancement" because I don't think the current behavior is a bug per se. Whether we should support this use case is still to be discussed.

[boazjohn]

I had a use-case where I was trying to export data from some legacy influxdb systems into prometheus. Would be hard to change the configs elsewhere.

[lukasmrtvy]

I have a same problem:

An error has occurred during metrics gathering:

collected metric system_uptime label:<name:"host" value:"proxmox" > label:<name:"object" value:"nodes" > untyped:<value:1.280685e+07 >  has label dimensions inconsistent with previously collected metrics in the same metric family

Metrics are sent from Proxmox:
https://pve.proxmox.com/wiki/External_Metric_Server

[matthiasr]

I'm going to close this because of age. We've updated the Prometheus client library since then, which changes some behaviors. In general I'm okay with being relaxed about label dimensions.