Webhook deliveries failing
Diagnose a webhook subscription whose deliveries do not reach the receiver, or whose receiver rejects them.
A failing webhook subscription presents in one of three ways: the delivery log shows non-2xx responses from the receiver, the log shows network errors before the receiver was reached, or the log is empty when the operator expected events.
Step 1: read the delivery log
Open Webhooks, the subscription in question, then Delivery log. Each entry shows:
event_typeandevent_id.attempted_at.response_status(or a network-level error string).response_body(truncated).
If the log is empty, the events the subscription is bound to have not fired since the subscription was created. Trigger a test event by changing a metric's threshold to flip status, or wait for an organic event.
Step 2: non-2xx from the receiver
| Status | Probable cause |
|---|---|
400 | The receiver expects a different payload schema. Compare against Webhook payload reference. |
401 / 403 | Authentication required. Receivers like generic Slack apps or HMAC-protected endpoints require headers Observer does not set by default. |
404 | URL is wrong. Re-paste from the receiver's documentation. |
429 | Receiver is rate-limiting. Reduce subscription scope or contact the receiver's vendor. |
5xx | Receiver is failing. The delivery worker retries with exponential backoff up to a fixed cap; deliveries are then moved to the dead-letter view. |
The delivery worker retries non-2xx responses. If retries exhaust, the entry moves to the Dead letter view; manual replay is available there.
Step 3: network errors
If response_status is missing and the log shows a network-level
error:
| Error substring | Probable cause |
|---|---|
ECONNREFUSED | Receiver host is not listening on the configured port. |
ENOTFOUND | DNS resolution failed. Verify the URL hostname. |
ETIMEDOUT | Receiver did not respond within the request timeout. |
CERT_HAS_EXPIRED / UNABLE_TO_VERIFY_LEAF_SIGNATURE | The receiver's TLS certificate is expired or untrusted by the public CA bundle. Renew the certificate; Observer does not accept self-signed certificates against a public endpoint. |
Step 4: signature verification on the receiver
If the receiver computes a signature mismatch but the URL and secret are correct:
- Confirm the secret is the value Observer's webhook subscription page shows, not a copy with whitespace.
- Confirm the receiver computes
HMAC-SHA-256(secret, raw_body)and compares against theX-Observer-Signatureheader value (after thesha256=prefix). - Confirm the receiver hashes the raw request body, not a re-serialised JSON. Some web frameworks parse and re-serialise request bodies on the way to the handler; the recomputed signature does not match.
Step 5: subscription is disabled
A subscription whose enabled flag is off does not deliver events. The subscription edit page exposes the toggle. If a subscription was disabled while debugging, re-enable it to resume deliveries; new events fire from the moment it is re-enabled, not backfilled.
Reading the audit log
Webhook delivery actions write audit log rows under the
webhook category. Audit log rows carry the failing receiver
URL and the response status, which helps when correlating
across multiple subscriptions.