[Research] Consolidate alert management

[jbrown-xentity]

User Story

In order to have 1 location for all alerts, data.gov O&M personnel wants to have a one-stop-shop for all alerts to manage the O&M roles.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• WHEN an alert is fired in Snyk, New Relic, Cloud.gov, etc
  THEN the data.gov team is notified in an effective way
  AND a dashboard is viewable to see all outstanding non-triaged issues/alerts

Background

Raised in our retro. With various systems, it can be easy for alerts to slip through the cracks with constant checking. It would be helpful to have a one-stop shop for all alerts that can be triaged, cleared, and revisited as necessary.
Currently that system is kind of slack, but some things are only sent to email. Currently the Slack channel datagov-alerts does not adequately highlight high importance alerts. We probably don't want to set up anything new/additional.

Security Considerations (required)

System needs to be secure.

Sketch

Should do research into this, how other TTS teams manage alerts, what products are available to be utilized, and cost.

• Can everything go into New Relic?

[FuhuXia]

Slack is good at being an alert hub that can receive alerts from all sources. We have our channel datagov-alert for it. But it lacks a way to manage those alerts, i.e., a dashboard. That is where a GitHub repo excels at. If we can build a private GH repo, with some python code and slack_sdk module to monitor datagov-alert channel messages via a github action cron, and do some filtering, and create an issue for each alert. That will combine the best features of two products.

[FuhuXia]

A GitHub gist is created to showcase a private Github repo datagov-alerts with a python script slack-to-github.py and a GitHub action create-issue.yml can be linked to our #datagov-alerts channel. Any new alerts going to the Slack channel will generate GitHub issue within 5 mins.

[update]
The files are installed to a public repo at https://github.com/FuhuXia/slack-to-github. Issues were created from messages in a test slack workspace.