Push notifications on metrics

[einari]

child of #71

We need to be able to get push notifications to our mobile phones based on different metrics.
Grafana has an extensive alert system that should enable this

[woksin]

@einari I need to know which metrics we want push notifications on, and for which namespaces

[einari]

Could we start with memory - as a test.
Set it to 1GB right now - we'll be setting a memory quota for everything that is way less than that, but interesting to gain some more experience

[woksin]

1gb quota for what? "User namespace", node, pod or what were you thinking of?

[einari]

Per Pod to start with.

[woksin]

Grafana's alert system is really flaky grafana/grafana#10412 (at least at the version that we are currently using).

Effectively means that I cannot set up alerts as of now

• Edit: Scratch that, I found out what was wrong

[einari]

This sits in developed; probably needs some clarification.
The idea was to be able to push to mobile devices - using something like notification hubs in Azure:
https://azure.microsoft.com/en-us/services/notification-hubs/

The goal is to have a variety of options for people to be able to subscribe to events. For our internal staff

[woksin]

@einari push notifications are apparently not that easy to set up.

However, Prometheus Alertmanager comes with "out-of-the-box" integrations for SMS and email. I think that setting the alertmanager up to send some specified alerts by sms and/or email would be our best bet for now.

[einari]

We can do SMS - but from what I can see in the documentation, it shouldn't be hard to do Notification Hubs:
https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-notification-hubs

https://docs.microsoft.com/en-us/azure/notification-hubs/

https://azure.microsoft.com/en-us/resources/samples/event-hubs-dotnet-user-notifications/

There should be a HTTPS endpoint to be able to do this. Can Prometheus post to a Web URL?

[einari]

There is another alternative as well - Azure Alerts: https://azure.microsoft.com/en-us/resources/videos/introduction-to-webhooks-for-azure-alerts/

[woksin]

Alternatively, there are lots of out-of-the-box support for alerting. Including possible integrations with WebHook, there are support for services like:
PagerDuty, Slack, VictorOps, OpsGenie, and more...

[woksin]

I think the easiest and best solution would be to use either PagerDuty or VictorOps for this, though they are paid services.

Or, perhaps Pushover for just mobile notifications

[woksin]

Or, we could use slack. Slack can give us mobile push notifications

[einari]

We have Teams - so that will effectively do the same in that regards, don't want to introduce Slack to the mix. Let's investigate further. It should be very simple to use Notification Hubs. The idea was to then govern this from Azure AD with groups

[einari]

We will park this issue for noe.