Grafana - setup IRM alerting (Grafana Cloud)

In this setup we will use Grafana Cloud IRM (incident response and management) functionality.

In Grafana, navigate to “Alerts & IRM” on the left panel and expand.
1. Expand “IRM” and and select “Integrations”. Click “Add integration” and select “web hook”
2. Give the integration a name. For now you can skip the remaining fields. Click “Create integration”.
3. On the next screen (see screenshot 1 below) click the box highlighted in red to get your alert web hook from Grafana. Copy the web hook and store it in a text editor.
4. Now you must register that Grafana web hook with Eyer. We recommend that you register it with the “updates” coming from Eyer, as it will then be able to capture any escalations on anomaly alerts.
  1. Register your web hook with Eyer with “alert-update” status, see the following article on how to register Web hooks - Anomaly Alerts
  2. Eyer will now push all anomaly alerts including updates to the registered web hook
5. Now, select the “Templates” section from your integration (see screenshot 1 below) and click the edit button.
  1. On the top section called “Grouping” (see screenshot 2 below), edit it to say {{ payload.id }}
  2. We want Grafana to only alert us on relevant alerts, so we want to use the “Autoresolution” section (screenshot 2). The example in the bullet below auto-resolves all alerts that have metrics not properly learnt yet by the ML (learning_status > 1), alerts that are not “severe”, alerts that impacts less than 5 nodes (components) and finally it auto-resolves all alerts that contains the wording “storage”.
  3. {{ (payload.learning_status > 1) or (payload.nodes_affected < 5) or (payload.severity != "severe") or ("storage" in payload|string) }}
  4. All the parameters above you should adjust to fit your alerting needs, including the free-text part. You can add multipe words it should search for (components, metrics etc) and auto-resolve.
  5. Close the window(s) so you get back to the “Integrations” window.
Next up, you define the users that should be able to receive alerts. In the left Grafana panel, select “IRM → Users”
1. Add the users and notification channels you would like to use. Pay attention to the default and important notification rules and make sure you have these set per user.
Final step is to define an “Escalation chain”. In the left Grafana panel, select “IRM → Escalation chains”.
1. Click “New escalation chain”
2. Add escalation steps according to your requirements. You can start the notification chain by just alerting a single person via email, then later if you receive multiple updates on the same alert you can notify more people via sms or some other channel.
3. Once you are happy with your escalation chain, go back to “Integrations” and add the escalation chain to your integration by clicking “Add route” (see screenshot 1 below).

Screenshot 1

Screenshot 2