/
Grafana - Alert investigation flow

Grafana - Alert investigation flow

Owned by Marius Wergeland Storsten
Last updated: Nov 27, 2024
2 min read

The dashboard template can be downloaded here.

Make sure you follow the instructions below to populate the dashboard with the correct keys before you import it to Grafana

This dashboard allows the user to select an ongoing alert, and then get more details on affected components and metrics.

image-20241127-105825.png

On top of the dashboard, the user can see an overview of

  • Dropdown menus to select alert & node (some alerts include multiple nodes affected). Once the alert is selected, select the node dropdown to chart data from different nodes in the alert.

  • Total active alerts (not qualified)

  • Total active alerts (qualified) - every alert from Eyer also contains a numeric status value in the range from 0 to 5, that says how accurate the alert is. The numeric status is calculated based on how good the data quality is for every single metric (is it enough data? Any data gaps? Are data points coming very seldom?). Eyer considers a value that is 0 or 1 as good quality, and these alerts are the ones highlighted in the dashboard.

  • Affected nodes - how many nodes out of the total are affected by the ongoing qualified alerts.

  • Affected metrics - how many metrics out of the total are affected by the ongoing qualified alerts.

  • Alerts treemap - this component highlights which alerts that are most critical and affects the most metrics. The user can hover the treemap for more information.

  • Info panel - when the user selects an alert from the top dropdown menu, detailed information on the alert will be shown here.

  • Alert history chart - the chart shows how the alert evolves from start to end

  • Metrics section - based on the alert (and node in the alert) selected by the user, the metrics in the alert will be charted here.

 

Import instructions

  1. Add the Influx connector to Grafana as described in step 1-4 in this article Grafana - visualize the metrics from Eyer

  2. Open the alert investigation dashboard template you downloaded from the top of this article. Search for “bucket”. Insert your environment id (in the format “97db480f-4566-4e6f-8ea5-124dd466fbc2”) from step to in front of the /RP_DetectionCorridor.

  3. Do Step 1 in the following article to enable the Grafana http connector Grafana - visualize anomaly alerts from Eyer

  4. Open the alert investigation dashboard template again. Replace all

    "value": " "

    with

    "value": "your_read_API_token"

  5. Save the template. In Grafana, select “New → Import” to import the dashboard template.