Add RabbitMQ Alerts 🚨

[ansd]

Add Prometheus rules for RabbitMQ and cluster operator.
Add Alertmanager config for Slack notifications.
Add RabbitMQ-Alerts Grafana dashboard.

Blog post on https://blog.rabbitmq.com/ will follow soon.

In future, we might want to move all dashboards from https://github.com/rabbitmq/rabbitmq-server/tree/master/deps/rabbitmq_prometheus/docker/grafana/dashboards into this repository and create GitHub actions to publish dashboards on https://grafana.com/orgs/rabbitmq.

[coro]

Looks amazing!

I love that you include remediation/next steps n each of the alerts, I think that's very useful for someone hitting these alerts and not knowing how to fix them.

I'm curious what you would/did use to test these alerts, and if it's worth sharing that tooling.

[coro]

If I understand correctly, this provides additional metadata about this specific alert if it fires. So if I have lots of different alerts, only one type will populate this column, and the rest will have an empty entry:

I'm fine with this for now, but there may come a time when we want to display metadata for many more alerts in the future. I wonder if there's any other way of displaying just opaque metadata, which can be filled with whatever is relevant to the alert in question.

As I say, only a thought, curious to hear your opinions. No need to change this now if you'd like.

[ansd]

Good point. The renaming from alert label (i.e. the metadata) to table column name takes place in

cluster-operator/observability/grafana/dashboards/rabbitmq-alerts.yml

Lines 380 to 392 in ae6e74e

	"renameByName": {
	"Time": "Time of Query",
	"Value": "Number of Times Alert started",
	"alertname": "Alert Name",
	"container": "",
	"instance": "Instance (scraped endpoint)",
	"namespace": "Namespace",
	"persistentvolumeclaim": "PersistentVolumeClaim",
	"pod": "Pod",
	"rabbitmq_cluster": "RabbitMQ Cluster",
	"rabbitmq_node": "RabbitMQ node",
	"severity": "Severity"
	}

. In your table output above only some alerts have a label persistentvolumeclaim. If an alert doesn't have such a label, the cell in the table will be empty.

I think if we observe that the table contains too many columns, we can either remap or omit certain columns or introduce an opaque metadata label as you suggest. As of now, the user can also filter the table by alert type (via the drop down at the top).

[coro]

I'm confused about the purpose of this tutorial (admittedly going in rather fresh-faced). Perhaps a README one level up would be useful in explaining the purpose of the three directories are?

Essentially, my user experience has been run the quickstart, then investigate the other directories under observability/prometheus. My prometheus spec doesn't have a podMonitorSelector, or a serviceMonitorSelector, but a probeSelector, so as a user I'm unclear how this README relates to what I've just set up.

[ansd]

Installation experience is different depending on where you start:

1. Prometheus is not installed, or
2. Prometheus is already installed via Prometheus Operator, or
3. Prometheus is already installed, but not via Prometheus Operator

By doing the quickstart, you started from option 1.

Your Prometheus spec has an empty podMonitorSelector and serviceMonitorSelector:

kubectl -n kube-prometheus get prometheus -o yaml

- apiVersion: monitoring.coreos.com/v1
  kind: Prometheus
...
  spec:
    ...
    podMonitorNamespaceSelector: {}
    podMonitorSelector: {}
    ...
    probeNamespaceSelector: {}
    probeSelector:
      matchLabels:
        release: prom
    ...
    serviceMonitorNamespaceSelector: {}
    serviceMonitorSelector: {}
    ...
    version: v2.26.0

I left these selectors empty on purpose via

cluster-operator/observability/quickstart.sh

Lines 31 to 32 in 279e6a6

	--set "prometheus.prometheusSpec.podMonitorSelectorNilUsesHelmValues=false" \
	--set "prometheus.prometheusSpec.serviceMonitorSelectorNilUsesHelmValues=false" \

to keep things as simple as possible in the quickstart. So, you don't need to add any labels in option 1.

However, if you were to apply these yaml files starting from option 2, you'll likely need to add these labels.

The blog post (currently in review) will make these 3 options clearer.

For now, I added probeSelectorNilUsesHelmValues=false to the quickstart to not have any matchLabels selector at all in the Prometheus spec.

I also included a comment in the YAML files that the labels only need to be added if there is a matchLabels selector defined (e.g. in

cluster-operator/observability/prometheus/monitors/rabbitmq-servicemonitor.yml

Line 6 in 279e6a6

# If labels are defined in spec.serviceMonitorSelector.matchLabels of your deployed Prometheus object, make sure to include them here.

).

I can add more READMEs if you think it makes it clearer?
I personally would rather want to link to the blog post from within https://github.com/rabbitmq/cluster-operator/blob/alerts/observability/README.md once it has been reviewed.

[coro]

Happy to wait on the blog post.