Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Customize label selector for prometheus-operator-managed ServiceMonitors #2199

Closed
gdemonet opened this issue Jan 21, 2020 · 0 comments · Fixed by #3356
Closed

Customize label selector for prometheus-operator-managed ServiceMonitors #2199

gdemonet opened this issue Jan 21, 2020 · 0 comments · Fixed by #3356
Assignees
@alexandre-allard
Labels
kind:debt Technical debt topic:deployment Bugs in or enhancements to deployment stages topic:docs Documentation topic:monitoring Everything related to monitoring of services in a running cluster topic:solutions Anything related to external "Solutions" integration with the platform

Comments

@gdemonet
Copy link
Contributor

gdemonet commented Jan 21, 2020 •
edited
Loading

Component: monitoring, salt

Why this is needed:

When Solutions (and most likely, their Operators) set up metrics scraping by the Prometheus provided by MetalK8s, they can either:

Our current deployment of prometheus-operator only looks at ServiceMonitors that match the release=prometheus-operator label. This can be too generic if other Prometheus Operators are deployed in the cluster, and doesn't make much sense outside of the Helm chart context.

What should be done:

Instead, we should define and use a custom label for these, e.g. metalk8s.scality.com/monitor="".
We should also document this label clearly in Solution developer docs (see #1994).

Implementation proposal (strongly recommended):

Test plan:

Epic: #1994
@gdemonet gdemonet added topic:docs Documentation topic:deployment Bugs in or enhancements to deployment stages moonshot topic:solutions Anything related to external "Solutions" integration with the platform topic:monitoring Everything related to monitoring of services in a running cluster labels Jan 21, 2020
This was referenced May 13, 2020
Merged
Closed
@gdemonet gdemonet mentioned this issue May 3, 2021
Merged
alexandre-allard added a commit that referenced this issue May 4, 2021
@alexandre-allard
charts: Add custom selector for Prometheus
05dceb8 
The selector is for Probe, ServiceMonitor and
PrometheusRule objects and it must match the label
`metalk8s.scality.com/monitor: ''`

We also add this labels to all resources deployed
by the kube-prometheus-stack chart as there is
actually no way to only add it to specific
resources (except for PrometheusRule).

Refs: #2199
alexandre-allard added a commit that referenced this issue May 4, 2021
@alexandre-allard
salt,charts: Use custom label selector for Prometheus
844478e 
We now use custom label selector
`metalk8s.scality.com/monitor` instead of the
default one `release` and `app` to target
ServiceMonitors, PrometheusRules and Probes.

kube-prometheus chart render command:
```
./charts/render.py prometheus-operator \
  charts/kube-prometheus-stack.yaml \
  charts/kube-prometheus-stack/ \
  --namespace metalk8s-monitoring \
  --service-config grafana \
  metalk8s-grafana-config \
  metalk8s/addons/prometheus-operator/config/grafana.yaml \
  metalk8s-monitoring \
  --service-config prometheus \
  metalk8s-prometheus-config \
  metalk8s/addons/prometheus-operator/config/prometheus.yaml \
  metalk8s-monitoring \
  --service-config alertmanager \
  metalk8s-alertmanager-config \
  metalk8s/addons/prometheus-operator/config/alertmanager.yaml \
  metalk8s-monitoring \
  --service-config dex \
  metalk8s-dex-config \
  metalk8s/addons/dex/config/dex.yaml.j2 metalk8s-auth \
  --drop-prometheus-rules charts/drop-prometheus-rules.yaml \
  > salt/metalk8s/addons/prometheus-operator/deployed/chart.sls
```

Refs: #2199
alexandre-allard added a commit that referenced this issue May 4, 2021
@alexandre-allard
salt, charts: Use custom Prometheus label
192e5e3 
Refs: #2199
@alexandre-allard alexandre-allard added the kind:debt Technical debt label May 5, 2021
alexandre-allard added a commit that referenced this issue May 5, 2021
@alexandre-allard
salt, charts: Use custom Prometheus label
117f584 
Refs: #2199
alexandre-allard added a commit that referenced this issue May 5, 2021
@alexandre-allard
docs: Update PrometheusRule required labels
f56e8b6 
Refs: #2199
@alexandre-allard alexandre-allard mentioned this issue May 6, 2021
Merged
@alexandre-allard alexandre-allard self-assigned this May 6, 2021
alexandre-allard added a commit that referenced this issue May 7, 2021
@alexandre-allard
charts: Add custom selector for Prometheus
76aed09 
The selector is for Probe, ServiceMonitor and
PrometheusRule objects and it must match the label
`metalk8s.scality.com/monitor: ''`

We also add this labels to all resources deployed
by the kube-prometheus-stack chart as there is
actually no way to only add it to specific
resources (except for PrometheusRule).

Refs: #2199
alexandre-allard added a commit that referenced this issue May 7, 2021
@alexandre-allard
salt,charts: Use custom label selector for Prometheus
9af9857 
We now use custom label selector
`metalk8s.scality.com/monitor` instead of the
default one `release` and `app` to target
ServiceMonitors, PrometheusRules and Probes.

kube-prometheus chart render command:
```
./charts/render.py prometheus-operator \
  charts/kube-prometheus-stack.yaml \
  charts/kube-prometheus-stack/ \
  --namespace metalk8s-monitoring \
  --service-config grafana \
  metalk8s-grafana-config \
  metalk8s/addons/prometheus-operator/config/grafana.yaml \
  metalk8s-monitoring \
  --service-config prometheus \
  metalk8s-prometheus-config \
  metalk8s/addons/prometheus-operator/config/prometheus.yaml \
  metalk8s-monitoring \
  --service-config alertmanager \
  metalk8s-alertmanager-config \
  metalk8s/addons/prometheus-operator/config/alertmanager.yaml \
  metalk8s-monitoring \
  --service-config dex \
  metalk8s-dex-config \
  metalk8s/addons/dex/config/dex.yaml.j2 metalk8s-auth \
  --drop-prometheus-rules charts/drop-prometheus-rules.yaml \
  > salt/metalk8s/addons/prometheus-operator/deployed/chart.sls
```

Refs: #2199
alexandre-allard added a commit that referenced this issue May 7, 2021
@alexandre-allard
salt, charts: Use custom Prometheus label
948f10b 
Refs: #2199
alexandre-allard added a commit that referenced this issue May 7, 2021
@alexandre-allard
docs: Update PrometheusRule required labels
bf06485 
Refs: #2199
alexandre-allard added a commit that referenced this issue May 7, 2021
@alexandre-allard
Add CHANGELOG entry for new Prometheus label selector
530683c 
Refs: #2199
@gdemonet gdemonet mentioned this issue Feb 7, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alexandre-allard alexandre-allard

Labels
kind:debt Technical debt topic:deployment Bugs in or enhancements to deployment stages topic:docs Documentation topic:monitoring Everything related to monitoring of services in a running cluster topic:solutions Anything related to external "Solutions" integration with the platform
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Customize Prometheus label selector scality/metalk8s
2 participants
@alexandre-allard @gdemonet