Splunk On-Call
Open and recover a Splunk On-Call incident from Observer.
Observer integrates with Splunk On-Call (formerly VictorOps) through its generic alert endpoint. One Observer incident maps to one Splunk On-Call incident by entity id.
Set up
Enable the generic integration
In Splunk On-Call, open Integrations and enable the Generic (REST) integration. Note the API key in the integration URL, and create or choose a routing key that maps to the right escalation policy.
Add the integration in Observer
In the console, open Alerts, choose Add alert, pick Splunk On-Call, and enter the API key and routing key. Choose org-wide or a single page, then save.
Test it
Use Test on the row. A test incident should open in Splunk On-Call. Recover it when you are done.
Lifecycle mapping
| Observer event | Splunk On-Call message type |
|---|---|
| Incident published | CRITICAL or WARNING (by severity) |
| Incident update posted | same, on the same entity |
| Incident resolved | RECOVERY |
The entity id is derived from the Observer incident id
(observer-incident-<id>). Splunk On-Call correlates messages with the same
entity id, so updates land on the open incident and the recovery closes the
right one.
Severity mapping
Critical and major map to CRITICAL; minor maps to WARNING. Scheduled
maintenance does not page.
Troubleshooting
- No incident appears: confirm both the API key and the routing key are correct, and that the routing key maps to an active escalation policy.
- Wrong escalation: the routing key controls who is paged. Adjust it in Splunk On-Call, not in Observer.