Add Alertmanager config and templates in Helm chart #188

TheoBrigitte · 2024-12-10T09:05:24Z

This PR does couple of things to get the Alertmanager into a Secret in the Helm chart:

Helm
- Add secret resource, embedding raw and templated alertmanager files
- Expose alertmanager templates values as helm chart values
Alertmanager
- Remove all Mimir related conditions from templates
- Escape template in template
- Split template into url and notification templates, to reduce template in template escaping
- Re-use slack actions, to reduce template in template escaping
- Drop template directive, dynamically set by the operator

How I generated the new Alertmanager config and notification template

Alertmanager config

wget https://raw.githubusercontent.com/giantswarm/prometheus-meta-operator/refs/heads/main/files/templates/alertmanager/alertmanager.yaml
sed -i -e 's/\[\[/{{/g' -e 's/\]\]/}}/g' alertmanager.yaml
patch alertmanager.yaml < <following patch>

13,15d12
< templates:
< - '/etc/alertmanager/config/*.tmpl'
<
24d20
<   {{- if .MimirEnabled }}
34d29
<   {{- end }}
173d167
< {{- if .MimirEnabled }}
187d180
< {{- end }}
206c199
<     actions:
---
>     actions: &slack-actions
209,210c202,203
<       url: '{{ template "__runbookurl" . }}'
<       style: '{{ if eq .Status "firing" }}primary{{ else }}default{{ end }}'
---
>       url: {{`{{ template "__runbookurl" . }}`}}
>       style: {{`{{ if eq .Status "firing" }}primary{{ else }}default{{ end }}`}}
213c206
<       url: '{{ template "__alert_linked_postmortems" . }}'
---
>       url: {{`{{ template "__alert_linked_postmortems" . }}`}}
216c209
<       url: '{{ template "__alerturl" . }}'
---
>       url: {{`{{ template "__alerturl" . }}`}}
219c212
<       url: '{{ template "__dashboardurl" . }}'
---
>       url: {{`{{ template "__dashboardurl" . }}`}}
222,223c215,216
<       url: '{{ template "__alert_silence_link" .}}'
<       style: '{{ if eq .Status "firing" }}danger{{ else }}default{{ end }}'
---
>       url: {{`{{ template "__alert_silence_link" .}}`}}
>       style: {{`{{ if eq .Status "firing" }}danger{{ else }}default{{ end }}`}}
242,259c235
<     actions:
<     - type: button
<       text: ':green_book: OpsRecipe'
<       url: '{{ template "__runbookurl" . }}'
<       style: '{{ if eq .Status "firing" }}primary{{ else }}default{{ end }}'
<     - type: button
<       text: ':coffin: Linked PMs'
<       url: '{{ template "__alert_linked_postmortems" . }}'
<     - type: button
<       text: ':mag: Query'
<       url: '{{ template "__alerturl" . }}'
<     - type: button
<       text: ':grafana: Dashboard'
<       url: '{{ template "__dashboardurl" . }}'
<     - type: button
<       text: ':no_bell: Silence'
<       url: '{{ template "__alert_silence_link" .}}'
<       style: '{{ if eq .Status "firing" }}danger{{ else }}default{{ end }}'
---
>     actions: *slack-actions
278,295c254
<     actions:
<     - type: button
<       text: ':green_book: OpsRecipe'
<       url: '{{ template "__runbookurl" . }}'
<       style: '{{ if eq .Status "firing" }}primary{{ else }}default{{ end }}'
<     - type: button
<       text: ':coffin: Linked PMs'
<       url: '{{ template "__alert_linked_postmortems" . }}'
<     - type: button
<       text: ':mag: Query'
<       url: '{{ template "__alerturl" . }}'
<     - type: button
<       text: ':grafana: Dashboard'
<       url: '{{ template "__dashboardurl" . }}'
<     - type: button
<       text: ':no_bell: Silence'
<       url: '{{ template "__alert_silence_link" . }}'
<       style: '{{ if eq .Status "firing" }}danger{{ else }}default{{ end }}'
---
>     actions: *slack-actions
314,331c273
<     actions:
<     - type: button
<       text: ':green_book: OpsRecipe'
<       url: '{{ template "__runbookurl" . }}'
<       style: '{{ if eq .Status "firing" }}primary{{ else }}default{{ end }}'
<     - type: button
<       text: ':coffin: Linked PMs'
<       url: '{{ template "__alert_linked_postmortems" . }}'
<     - type: button
<       text: ':mag: Query'
<       url: '{{ template "__alerturl" . }}'
<     - type: button
<       text: ':grafana: Dashboard'
<       url: '{{ template "__dashboardurl" . }}'
<     - type: button
<       text: ':no_bell: Silence'
<       url: '{{ template "__alert_silence_link" . }}'
<       style: '{{ if eq .Status "firing" }}danger{{ else }}default{{ end }}'
---
>     actions: *slack-actions
350,367c292
<     actions:
<     - type: button
<       text: ':green_book: OpsRecipe'
<       url: '{{ template "__runbookurl" . }}'
<       style: '{{ if eq .Status "firing" }}primary{{ else }}default{{ end }}'
<     - type: button
<       text: ':coffin: Linked PMs'
<       url: '{{ template "__alert_linked_postmortems" . }}'
<     - type: button
<       text: ':mag: Query'
<       url: '{{ template "__alerturl" . }}'
<     - type: button
<       text: ':grafana: Dashboard'
<       url: '{{ template "__dashboardurl" . }}'
<     - type: button
<       text: ':no_bell: Silence'
<       url: '{{ template "__alert_silence_link" . }}'
<       style: '{{ if eq .Status "firing" }}danger{{ else }}default{{ end }}'
---
>     actions: *slack-actions
382,399c307
<     actions:
<     - type: button
<       text: ':green_book: OpsRecipe'
<       url: '{{ template "__runbookurl" . }}'
<       style: '{{ if eq .Status "firing" }}primary{{ else }}default{{ end }}'
<     - type: button
<       text: ':coffin: Linked PMs'
<       url: '{{ template "__alert_linked_postmortems" . }}'
<     - type: button
<       text: ':mag: Query'
<       url: '{{ template "__alerturl" . }}'
<     - type: button
<       text: ':grafana: Dashboard'
<       url: '{{ template "__dashboardurl" . }}'
<     - type: button
<       text: ':no_bell: Silence'
<       url: '{{ template "__alert_silence_link" .}}'
<       style: '{{ if eq .Status "firing" }}danger{{ else }}default{{ end }}'
---
>     actions: *slack-actions
418,435c326
<     actions:
<     - type: button
<       text: ':green_book: OpsRecipe'
<       url: '{{ template "__runbookurl" . }}'
<       style: '{{ if eq .Status "firing" }}primary{{ else }}default{{ end }}'
<     - type: button
<       text: ':coffin: Linked PMs'
<       url: '{{ template "__alert_linked_postmortems" . }}'
<     - type: button
<       text: ':mag: Query'
<       url: '{{ template "__alerturl" . }}'
<     - type: button
<       text: ':grafana: Dashboard'
<       url: '{{ template "__dashboardurl" . }}'
<     - type: button
<       text: ':no_bell: Silence'
<       url: '{{ template "__alert_silence_link" . }}'
<       style: '{{ if eq .Status "firing" }}danger{{ else }}default{{ end }}'
---
>     actions: *slack-actions
450,467c341
<     actions:
<     - type: button
<       text: ':green_book: OpsRecipe'
<       url: '{{ template "__runbookurl" . }}'
<       style: '{{ if eq .Status "firing" }}primary{{ else }}default{{ end }}'
<     - type: button
<       text: ':coffin: Linked PMs'
<       url: '{{ template "__alert_linked_postmortems" . }}'
<     - type: button
<       text: ':mag: Query'
<       url: '{{ template "__alerturl" . }}'
<     - type: button
<       text: ':grafana: Dashboard'
<       url: '{{ template "__dashboardurl" . }}'
<     - type: button
<       text: ':no_bell: Silence'
<       url: '{{ template "__alert_silence_link" .}}'
<       style: '{{ if eq .Status "firing" }}danger{{ else }}default{{ end }}'
---
>     actions: *slack-actions
482,499c356
<     actions:
<     - type: button
<       text: ':green_book: OpsRecipe'
<       url: '{{ template "__runbookurl" . }}'
<       style: '{{ if eq .Status "firing" }}primary{{ else }}default{{ end }}'
<     - type: button
<       text: ':coffin: Linked PMs'
<       url: '{{ template "__alert_linked_postmortems" . }}'
<     - type: button
<       text: ':mag: Query'
<       url: '{{ template "__alerturl" . }}'
<     - type: button
<       text: ':grafana: Dashboard'
<       url: '{{ template "__dashboardurl" . }}'
<     - type: button
<       text: ':no_bell: Silence'
<       url: '{{ template "__alert_silence_link" .}}'
<       style: '{{ if eq .Status "firing" }}danger{{ else }}default{{ end }}'
---
>     actions: *slack-actions
504c361
<     tags: "{{ (index .Alerts 0).Labels.alertname }},{{ (index .Alerts 0).Labels.cluster_type }},{{ (index .Alerts 0).Labels.severity }},{{ (index .Alerts 0).Labels.team }},{{ (index .Alerts 0).Labels.area }},{{ (index .Alerts 0).Labels.service_priority }},{{ (index .Alerts 0).Labels.provider }},{{ (index .Alerts 0).Labels.installation }},{{ (index .Alerts 0).Labels.pipeline }},{{ (index .Alerts 0).Labels.customer }}"
---
>     tags: {{`{{ (index .Alerts 0).Labels.alertname }},{{ (index .Alerts 0).Labels.cluster_type }},{{ (index .Alerts 0).Labels.severity }},{{ (index .Alerts 0).Labels.team }},{{ (index .Alerts 0).Labels.area }},{{ (index .Alerts 0).Labels.service_priority }},{{ (index .Alerts 0).Labels.provider }},{{ (index .Alerts 0).Labels.installation }},{{ (index .Alerts 0).Labels.pipeline }},{{ (index .Alerts 0).Labels.customer }}`}}

Notification template

wget https://raw.githubusercontent.com/giantswarm/prometheus-meta-operator/refs/heads/main/files/templates/alertmanager/notification-template.tmpl
patch notification-template.tmpl < <following patch>

3,21d2
< {{ define "__alerturl" }}
< [[- if .MimirEnabled -]]
< [[ .GrafanaAddress ]]/alerting/Mimir/{{ .CommonLabels.alertname }}/find
< [[- else -]]
< {{ .ExternalURL }}/#/alerts?receiver={{ .Receiver }}&silenced=false&inhibited=false&active=true&filter=%7Balertname%3D%22{{ .CommonLabels.alertname }}%22%7D
< [[- end -]]
< {{ end }}
<
< {{ define "__dashboardurl" -}}{{ if match "^https://.+" (index .Alerts 0).Annotations.dashboard }}{{ (index .Alerts 0).Annotations.dashboard }}{{ else }}[[ .GrafanaAddress ]]/d/{{ (index .Alerts 0).Annotations.dashboard }}{{ end }}{{- end }}
< {{ define "__runbookurl" -}}https://intranet.giantswarm.io/docs/support-and-ops/ops-recipes/{{ (index .Alerts 0).Annotations.opsrecipe }}{{- end }}
<
< {{ define "__queryurl" }}
< [[- if .MimirEnabled -]]
< [[ .GrafanaAddress ]]/alerting/Mimir/{{ .CommonLabels.alertname }}/find
< [[- else -]]
< {{ (index .Alerts 0).GeneratorURL }}
< [[- end -]]
< {{ end }}
<
59d39
< [[- if .MimirEnabled ]]
61,64d40
< [[- else ]]
< 🔔 Alertmanager {{ template "__alerturl" . }}
< 👀 Query: {{ template "__queryurl" . }}
< [[- end ]]

I would like some opinions before I continue in this direction, because I feel there are a lot of workaround here to get this config into a Secret and it could be easier to have it directly in code. Also does anyone remember what the ProxyURL is used for ? It seems to be OpsGenie related but I gladly ignored it and I have a feeling this could be important.

helm/observability-operator/files/alertmanager/alertmanager.yaml.helm-template

QuentinBisson · 2024-12-10T10:42:29Z

helm/observability-operator/files/alertmanager/alertmanager.yaml.helm-template

+    send_resolved: true
+    actions: *slack-actions
+
+- name: team_turtles_slack


This maybe should go away

QuentinBisson · 2024-12-10T10:45:15Z

helm/observability-operator/files/alertmanager/url-template.tmpl.helm-template

@@ -0,0 +1,18 @@
+{{`
+{{ define "__alerturl" }}
+`}}{{ .alertmanager.grafanaAddress }}{{`/alerting/Mimir/{{ .CommonLabels.alertname }}/find


I think we should be able to replace this with the grafanaExploreURL or this queryFromGeneratorURL instead https://grafana.com/docs/mimir/latest/references/architecture/components/alertmanager/#templating right?

Yes I'll look into this later

I'm trying to see if we can get rid of this whole file. that would make things simpler imo

When using those we go back to having long urls containing the alert query. I don't think we want to go back there.

We can discuss this sure :)

helm/observability-operator/templates/alertmanager/_helpers.tpl

helm/observability-operator/files/alertmanager/url-template.tmpl.helm-template

helm/observability-operator/values.yaml

helm/observability-operator/files/alertmanager/notification-template.tmpl

- Add secret resource, embedding raw and templated alertmanager files - Expose alertmanager templates values as helm chart values - Remove all Mimir related conditions - Split template into url and notification templates - Drop template directive, dynamically set by the operator - Escape template in template - Re-use slack actions

…asepath from values inside tpl function

This fixes the infamous: error calling tpl: cannot retrieve Template.Basepath from values inside tpl function It does use .Values in templates to access values and pass $ root context to tpl

Co-authored-by: Quentin Bisson <[email protected]>

helm/observability-operator/Chart.yaml

helm/observability-operator/values.yaml

This reverts commit 283c100.

QuentinBisson · 2024-12-17T13:10:36Z

helm/observability-operator/files/alertmanager/notification-template.tmpl

+* Team: {{ (index .Alerts 0).Labels.team }}
+* Area: {{ (index .Alerts 0).Labels.area }} / {{ (index .Alerts 0).Labels.topic }}
+* Instances:{{ range .Alerts.Firing }}
+🔥 {{ if .Labels.instance }}{{ .Labels.instance }}: {{ end }}{{ .Annotations.description }}{{ end }}
 {{- end }}

 # This builds the silence URL.  We exclude the alertname in the range


I think we need to remove this silence URL right?

TheoBrigitte requested a review from a team as a code owner December 10, 2024 09:05

TheoBrigitte self-assigned this Dec 10, 2024

TheoBrigitte changed the base branch from main to alertmanager-config December 10, 2024 09:06

QuentinBisson reviewed Dec 10, 2024

View reviewed changes

helm/observability-operator/files/alertmanager/alertmanager.yaml.helm-template Outdated Show resolved Hide resolved

QuentinBisson reviewed Dec 10, 2024

View reviewed changes

helm/observability-operator/templates/alertmanager/_helpers.tpl Outdated Show resolved Hide resolved

QuentinBisson reviewed Dec 10, 2024

View reviewed changes

helm/observability-operator/files/alertmanager/url-template.tmpl.helm-template Show resolved Hide resolved

QuentinBisson reviewed Dec 10, 2024

View reviewed changes

helm/observability-operator/values.yaml Outdated Show resolved Hide resolved

QuentinBisson reviewed Dec 10, 2024

View reviewed changes

helm/observability-operator/files/alertmanager/notification-template.tmpl Show resolved Hide resolved

QuentinBisson reviewed Dec 10, 2024

View reviewed changes

helm/observability-operator/files/alertmanager/notification-template.tmpl Show resolved Hide resolved

TheoBrigitte force-pushed the alertmanager-config-helm branch from 4b4c161 to 11abd5f Compare December 10, 2024 15:18

TheoBrigitte mentioned this pull request Dec 10, 2024

Alertmanager test branch #192

Closed

TheoBrigitte changed the base branch from alertmanager-config to main December 10, 2024 17:45

TheoBrigitte and others added 8 commits December 10, 2024 18:48

use .helm-template instead of .tpl

9eea3f8

re-use existing values

27b62fa

normalize values names

49b5a1a

try to make helm happy: error calling tpl: cannot retrieve Template.B…

2ea1101

…asepath from values inside tpl function

helm 3.10.3 I hate you deeply

801cadd

This fixes the infamous: error calling tpl: cannot retrieve Template.Basepath from values inside tpl function It does use .Values in templates to access values and pass $ root context to tpl

Release v0.10.0 (#191)

8cbdbfb

Co-authored-by: Quentin Bisson <[email protected]>

quote the tmp_string

5d320b9

TheoBrigitte force-pushed the alertmanager-config-helm branch from 9d0c548 to 5d320b9 Compare December 10, 2024 17:50

Add alertmanagerURL

4cfbb20

TheoBrigitte mentioned this pull request Dec 10, 2024

Investigation: Migrate the existing alerts to mimir alertmanager giantswarm/roadmap#3746

Closed

9 tasks

QuentinBisson reviewed Dec 10, 2024

View reviewed changes

helm/observability-operator/Chart.yaml Outdated Show resolved Hide resolved

TheoBrigitte added 2 commits December 13, 2024 12:47

Merge remote-tracking branch 'origin/main' into alertmanager-config-helm

492a1d7

rename alertmanager to alerting in values.yaml

9591a92

TheoBrigitte mentioned this pull request Dec 15, 2024

Implement handling of Alertmanager config #180

Merged

QuentinBisson reviewed Dec 16, 2024

View reviewed changes

helm/observability-operator/values.yaml Outdated Show resolved Hide resolved

TheoBrigitte added 8 commits December 16, 2024 19:51

set alertmanager controller name

283c100

Revert "set alertmanager controller name"

ec33d05

This reverts commit 283c100.

use mimir url generators;kill the new lines

cd01322

add templates

0b081dc

Use correct datasource, with uid :/

1a42e53

Attempt at making dashboard url a single line

eda8f3a

Align with new notification-template changes

9e09827

fix explore link new line

52901c4

QuentinBisson reviewed Dec 17, 2024

View reviewed changes

TheoBrigitte added 5 commits December 17, 2024 14:47

Merge remote-tracking branch 'origin/main' into alertmanager-config-helm

2b3730c

Add the silence link

253a288

update silence link (again)

51d0f0b

Use proxy_from_environment in Alertmanager

270467c

Merge branch 'main' into alertmanager-config-helm

76d694d

QuentinBisson approved these changes Jan 6, 2025

View reviewed changes

TheoBrigitte added 6 commits January 6, 2025 11:23

remove proxyURL from values

1a5e321

add comments in url-template

cb2a4b3

Merge branch 'main' into alertmanager-config-helm

78b8e4a

fix templates comments

f5af73a

only use alert link

c6b897c

Merge branch 'main' into alertmanager-config-helm

c9fc15b

TheoBrigitte enabled auto-merge (squash) January 15, 2025 12:06

TheoBrigitte disabled auto-merge January 15, 2025 12:09

Merge branch 'main' into alertmanager-config-helm

932efb3

TheoBrigitte enabled auto-merge (squash) January 15, 2025 12:44

TheoBrigitte merged commit 510b5f4 into main Jan 15, 2025
9 checks passed

TheoBrigitte deleted the alertmanager-config-helm branch January 15, 2025 12:52

TheoBrigitte added a commit that referenced this pull request Jan 15, 2025

Add changelog entry for #188

703437b

TheoBrigitte mentioned this pull request Jan 15, 2025

Add changelog entry for #188 #227

Merged

TheoBrigitte added a commit that referenced this pull request Jan 15, 2025

Add changelog entry for #188 (#227)

fdd2fca

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Alertmanager config and templates in Helm chart #188

Add Alertmanager config and templates in Helm chart #188

TheoBrigitte commented Dec 10, 2024 •

edited

Loading

QuentinBisson Dec 10, 2024

TheoBrigitte Dec 10, 2024

QuentinBisson Dec 10, 2024

TheoBrigitte Dec 10, 2024

QuentinBisson Dec 10, 2024

TheoBrigitte Jan 6, 2025

QuentinBisson Jan 7, 2025

QuentinBisson Dec 17, 2024

Add Alertmanager config and templates in Helm chart #188

Add Alertmanager config and templates in Helm chart #188

Conversation

TheoBrigitte commented Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheoBrigitte commented Dec 10, 2024 •

edited

Loading