Version 1.13+ Prometheus module does not export all Consul plug-in metrics. #7387
Comments
|
Can you add your prometheus_client output plugin configuration? |
danielnelson
added
area/prometheus
bug
|
|
Looking into this a bit closer, and the issue appears to be that labels starting with a 0-9 are illegal in Prometheus format and are rejected by the official library, in Telegraf 1.13 we updated the library and it has become more strict preventing these. If you switch to I think the best way forward is to adjust the consul input to avoid these types of tags. What if you disable the |
|
cat consul.conf A possible solution would be to introduce an additional option into the consul module that will rename the metrics to a template that will be correct for the prometheus library. |
|
Having UUIDs as the tagkey is not an ideal setup for any output, so I think we can come up with a better strategy for creating metrics. Can you show the output of |
|
|
Can you run this query against the consul HTTP api in order to get the raw JSON for one of the check_id that produces a UUID tagkey: |
|
Quick follow-up, what I'm expecting to see is that you have ServiceTags like: I'm far from a Consul expert, so to me tags like this seem a bit odd. Can you tell me a bit about how you use this type of tag? |
|
|
I think what will be best in your case is to exclude these tags. The information is contained in the [[inputs.consul]]
tagexclude = ["[!0-9]*"]As a more general fix, perhaps we should add a new option that matches only ServiceTags, similar to how the docker plugin is structured: [[inputs.consul]]
service_tag_include = []
service_tag_exclude = ["[0-9]*"] |
|
Hello! But Prometheus can't take labels with first digit: Maybe should have an option, which shows if we wants to see tags as labels or not? Or any regex for including this tags, not all of them. At this moment i see that valid consul meta configuration can affect on some metrics(!!) not even labels disappear. Same theme was mentioned in several topics: |
|
@ekbfh What do you think about if we add the |
|
@danielnelson It might work, if you plan enable them by default. Cause as i say, i may have this naming in consul and cannot in prom. Could you also add an option to choose what tags i want to gather? |
|
You would be able to exclude all service tags with We should also make sure that the prometheus output just removes tags that it cannot encode as labels, without removing the output. |
|
Yes, just removing tags is a good idea |
|
Any update? |
After upgrade 1.12.6 to 1.13+ (1.14.1 affected too) i see this picture
Grafana quiery:
sum by (service_name)(consul_health_checks_passing{service_name!=""})telegraf --config telegraf.conf --config ./telegraf.d/basic_inputs.conf --config ./telegraf.d/consul.conf --test | grep a-B
curl http://127.0.0.1:9273/metrics | grep -i a-B
System info:
Telegraf 1.12.6 and 1.13.0
Consul v1.6.2
Prometheus 2.15.2
Centos 7.8
Steps to reproduce:
upgrade 1.12.6 to 1.13+
Expected behavior:
Actual behavior:
Additional info:
The text was updated successfully, but these errors were encountered: