As a cloud.gov operator, I want production prometheus alerts in slack.

[spgreenberg]

Email is so 2005. Our email is overloaded with alerts. History of response to alerts is often lost.

It appears prometheus supports slack (I don't know if our version does): https://prometheus.io/docs/alerting/latest/notification_examples/.

It would be much more useful to have relevant production alerts in slack with icons. This way, the support pair could discuss and respond. We would have a history of relevant alerts.

Acceptance Criteria

• Production alerts are posted in slack in a private channel

---

Security considerations

This hopefully helps us to respond to alerts and potential issues more quickly.

[bengerman13]

FWIW - when we were looking at OpsGenie, one of the problems we encountered was that OpsGenie is not authorized for FedRAMP Medium, so we were supposed to remove certain types of information from our alerts (which we never did and is why we're not using OpsGenie)
I'm not sure what level Slack is authorized for - I believe it was a LISaaS, but that may have changed with the switch to Enterprise Grid or whatever it's called.

[spgreenberg]

Good point. With Slack templates, I think we can effectively filter and still have more timely alerts.