[Observability] Basic alerting meta issue

[jasonrhodes]

We need to implement basic alerting features for all of our observability apps. We should take advantage of the new alerting and actions plugins in Kibana to do this.

This meta issue will track work being done on this effort.

Important Links

• Alerting plugin documentation
  
• Actions plugin documentation

Stage 1: Requirements gathering

• Gather requirements across observability for what we need in an alerting solution.

Stage 2: Proof of concept

Build a simple proof of concept that uses the alerting and action plugins. The POC should do the following:

• Create an alert
• Prove the created alert fires a slack (and email?) action
• Document what features are available on the fired alert (does it fire once only? do you have to "ack" it? can you snooze it or mute it? etc)
• Display a list of created alerts
• Update an alert (what can be updated?)
• Delete an alert

@graphaelli is going to look into whether this can be built into the APM UI as a POC replacement for the watcher integration. If that doesn't work for any reason, we'll choose a different way to implement the POC in Kibana.

Stage 3: Implementation plan

While the POC is being built, we should begin to have a better outline of the product we want to build, assuming it's possible to build it. What features are most important? What should be the priority for alert implementations across the observability solutions?

[jasonrhodes]

Replaced by working group meta issue