Grafana IRM
Fire and resolve a Grafana IRM alert from Observer via an incoming webhook.
Observer sends incidents to Grafana IRM (Incident Response and Management) through a custom incoming-webhook integration. One Observer incident maps to one Grafana IRM alert.
Set up
Create a Webhook integration
In Grafana IRM, add an Integration of type Webhook. Grafana issues a
unique URL for it, for example
https://<stack>.grafana.net/integrations/v1/webhook/<token>/.
Add the integration in Observer
In the console, open Alerts, choose Add alert, pick Grafana IRM, and paste the integration URL. The URL embeds a token, so it is stored encrypted. Choose org-wide or a single page, then save.
Test it
Use Test on the row. A test alert should appear in Grafana IRM. Resolve it there when you are done.
Lifecycle mapping
| Observer event | Grafana IRM action |
|---|---|
| Incident published | Fire an alert (grouped by alert id) |
| Incident update posted | Fire on the same alert id |
| Incident resolved | Resolve the grouped alert |
Correlation is by alert_uid, derived from the Observer incident id
(observer-incident-<id>). New events group onto that alert; resolution posts
to the integration URL's resolve action for the same id, so the lifecycle
produces one alert that fires and then resolves.
Maintenance windows do not alert
Scheduled maintenance events do not fire a Grafana IRM alert. Use a chat or email channel if you want maintenance announcements.
Troubleshooting
- No alert appears: confirm the URL is the full integration webhook URL, including the trailing token segment, and that the integration is enabled in Grafana IRM. A 4xx is surfaced in the Observer delivery result.
- Resolve does not close the alert: the resolve posts to the same URL with a
resolveaction. Make sure the integration is a Webhook integration that accepts incoming alerts.