How to scrape and use kube-state-metrics

[fischerman]

I haven't found a good reference on scraping from kube-state-metrics. kube-prometheus uses a service monitor which scrapes all the endpoints of the service. Hence, if kube-state-metrics runs replicated the metrics are scraped multiple times with different instance labels. Even with one replica the instance label changes over time. How do I write queries and alerts without having to constantly (pseudo-)aggregate over all instances?

kube-state-metrics exposes service-level metrics and therefore I would also scrape them through a service (i.e. one instance per Kubernetes cluster backed by multiple pods for HA). Or is there a good reason to scrape directly from the pods? Scraping from services is possible in Prometheus but AFAIK not that easy using the Prometheus operator. Despite it's name a service monitor doesn't scrape a service.

[lilic]

Hey 👋 Do you mean scrapping kube-state-metrics if it's shared with multiple replicas/instances? Maybe you can clarify the use case here, thank you!

[fischerman]

Let me give you an example which illustrates it with plain kube-prometheus (i.e. one replica scraped by a service monitor). Suppose I want to query the number of nodes with a certain label. I might write something like that.

count(kube_node_labels{job="kube-state-metrics", label_foo="bar"})

In most cases this query will give me the correct number n. Now suppose I update kube-state-metrics. During the rolling release there are two pods (and everything is eventually consistent). Suddenly, the query returns 2*n although the number of pods is still n. If I were to replicate kube-state-metrics this problem would be more permanent.

So in order to prevent that my query would have to look something like this

count(avg without(instance) (kube_node_labels{job="kube-state-metrics", label_foo="bar"}))

I think this is not obvious and becomes cumbersome if I want to join multiple kube-state-metrics. So intuitively I would only want to have one instance (in the Prometheus sense) to scrape (i.e. an instance resembles a Kubernetes cluster). Without the operator I would use a service role. My primary questions is, is there a disadvantage with that approach? I hope the advantages are clear. If that is the case, it is still not that easy to implement with the operator because the service monitor relies on the endpoint role rather than the service role.

[fischerman]

The Prometheus operator has some docs in that regard and names kube-state-metrics explicitly as an example for non-instance exporters:

Cluster-state exporters: These expose metrics about an entire system. For example these could be the number of 3D objects in a game, or metrics about a Kubernetes deployment. These exporters are typically deployed as a normal Kubernetes deployment, but can vary depending on the nature of the particular exporter. A real life example of this is the kube-state-metrics exporter, which exposes metrics about the cluster state of a Kubernetes cluster. [1]

Unfortunately, it does not go into details on how to achieve this with the Operator.

[1] https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/exposing-metrics.md

[lilic]

Okay got it now, its when two replicas of same instance of kube-state-metrics are running during rollout/upgrade. Is this issue mainly a problem for you with querying/recording rules (e.g. many-to-many errors) or also are you also worried about ingestion of duplicated series due to too much resources?

[fischerman]

It's mainly about convenient and readable queries. Only ingesting the metrics once would be one solution that also reduces resource usage.
In my mind kube-state-metrics is stateless (except caching) and should be able to run with multiple replicas for high availability. It shouldn't matter which replica is scraped for metrics.