Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kubernetes Apiserver metricset collects deprecated metrics #31834

Closed
MichaelKatsoulis opened this issue Jun 6, 2022 · 0 comments · Fixed by #31973
Closed

Kubernetes Apiserver metricset collects deprecated metrics #31834

MichaelKatsoulis opened this issue Jun 6, 2022 · 0 comments · Fixed by #31973
Assignees
Labels
Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team

Comments

@MichaelKatsoulis
Copy link
Contributor

MichaelKatsoulis commented Jun 6, 2022

Kubernetes API server metricset collects metrics that are deprecated(apiserver_request_latencies, apiserver_request_count) in newer versions of kubernetes.
At the same time we don't collect some really important metrics(apiserver_request_duration_seconds_sum, apiserver_request_duration_seconds_count).

Currently we support k8s versions starting from 1.21.x so there is no need to keep searching for older versions' fields in current/newer versions of beats.

  1. All API server exposed metrics should be investigated and the one we collect should be reconsider.

  2. The API server dashboard is half broken because it tries to make use of the deprecated fields. It should be updated as well with more meaningful panels.

  3. Some metrics exposed from API server prometheus endpoint are of histogram type.
    apiserver_request_duration_seconds is one of them. We should find a way to store them as object_type: histogram in Elasticsearch so that we can leverage percentiles aggregation in Kibana for meaningful dashboard creation. This has been achieved in the past in lightweight modules like istio(link). It makes use of the use_types config option which allows prometheus collector metricset to map prometheus histogram types to elasticsearch histograms (PR). We need to find a way to leverage that feature(or the logic behind it) at a non-lightweight module for specific fields. (This bullet is moved to new issue Use Elasticsearch histogram type to store Prometheus histograms #32006)

Kubernetes Controller Manager metricset appears to have same issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Team:Cloudnative-Monitoring Label for the Cloud Native Monitoring team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant