-
Notifications
You must be signed in to change notification settings - Fork 146
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Prometheus feature native-histograms
breaks targets with info
metric types from being scraped.
#14
Comments
Seems like simply switching to Gauge type produces an error on kube-state-metrics:
|
I was able to setup the gauges after looking at the kube-state-metrics docs And have setup gauges based on the following. The metric name is changed to gotk_resource_status since not sure it is 1:1 with the suggested alerts in the flux 2.1 docs. |
@defenestration thanks for reporting this and doing all the investigation. I see that it's being worked on in upstream KSM and started some discussions in prometheus. I think we can wait for the on-going KSM work to conclude before making any change for prometheus native-histograms support. My guess is that we may not need to change anything and KSM will handle the issue internally, based on the KSM discussions. |
I was updating fluxv2 and using this repo as a template, but i noticed in prometheus, that the kube-state-metrics target was showing the error
"invalid metric type "info"
. After some some local testing with a fork of the repo, I'm finding thenative-histograms
feature of prometheus seems to cause the problem. I have a branch here for testing this out.https://github.com/defenestration/flux2-monitoring-example/tree/native-histograms
Heres a pr with the diffs from this repo from when i forked it if it helps https://github.com/defenestration/flux2-monitoring-example/pull/1/files
Realize this is probably an upstream issue, but wondering what the best work-around for this could be? Seems like the custom metric types may be best if set as
Gauge
instead perhaps?I will continue to look on my own but figured it was worth filing an issue in case someone else had this problem too.
The text was updated successfully, but these errors were encountered: