Grafana - IRM alerts
This article is just one example out of many on how you can setup Grafana alerts based on the anomaly alerts coming from Eyer. Grafana can be highly customized with alert templates and contact points to accommodate a wide range of alerting scenarios.
The Grafana IRM (Incident response and management, read more about the feature here) alert rules can be used to process anomaly alerts coming from Eyer, and apply logic based on anomaly alert severity, correlations, learning status and which nodes that take part in the alert.
The Eyer API used in the example below is described in Webhooks - Anomaly Alerts (deprecated) and we recommend that you subscribe to the “updates” to capture the full history of the alerts.
Under “IRM → Integrations” you select “Add integrations” and then select “Webhook generic”. Set an integration name and then click “Create integration”.
Register the Grafana web hook with the Eyer API listed above, and subscribe to “updates”
Select your created integration and then click edit on the “Templates” section. You should now see something similar as the screenshot below.
For “Grouping” you want to set it to “{{ payload.id }}” as this will group all the individual alerts and their updates.
For “Autoresolution” we try to set a rule so that all alerts we are not interested in gets auto-resolved. In our example, the “Autoresolution” use multiple parameters in the alert to qualify if the alert should be resolved automatically: “
{{ (payload.learning_status > 1) or (payload.nodes_affected < 5) or (payload.severity != "severe") }}"In the example, we only want to be alerted if the alert has a solid learning status (baselines are solid, Eyer knows a lot about the metrics that are alerted), and that the ongoing situation shows an impact from the initial metric deviation (nodes_affected) and of “severe” status.
