TroubleshootingWebhook deliveries failing

Webhook deliveries failing

Diagnose a webhook subscription whose deliveries do not reach the receiver, or whose receiver rejects them.

A failing webhook subscription presents in one of three ways: the delivery log shows non-2xx responses from the receiver, the log shows network errors before the receiver was reached, or the log is empty when the operator expected events.

Step 1: read the delivery log

Open Webhooks, the subscription in question, then Delivery log. Each entry shows:

  • event_type and event_id.
  • attempted_at.
  • response_status (or a network-level error string).
  • response_body (truncated).

If the log is empty, the events the subscription is bound to have not fired since the subscription was created. Trigger a test event by changing a metric's threshold to flip status, or wait for an organic event.

Step 2: non-2xx from the receiver

StatusProbable cause
400The receiver expects a different payload schema. Compare against Webhook payload reference.
401 / 403Authentication required. Receivers like generic Slack apps or HMAC-protected endpoints require headers Observer does not set by default.
404URL is wrong. Re-paste from the receiver's documentation.
429Receiver is rate-limiting. Reduce subscription scope or contact the receiver's vendor.
5xxReceiver is failing. The delivery worker retries with exponential backoff up to a fixed cap; deliveries are then moved to the dead-letter view.

The delivery worker retries non-2xx responses. If retries exhaust, the entry moves to the Dead letter view; manual replay is available there.

Step 3: network errors

If response_status is missing and the log shows a network-level error:

Error substringProbable cause
ECONNREFUSEDReceiver host is not listening on the configured port.
ENOTFOUNDDNS resolution failed. Verify the URL hostname.
ETIMEDOUTReceiver did not respond within the request timeout.
CERT_HAS_EXPIRED / UNABLE_TO_VERIFY_LEAF_SIGNATUREThe receiver's TLS certificate is expired or untrusted by the public CA bundle. Renew the certificate; Observer does not accept self-signed certificates against a public endpoint.

Step 4: signature verification on the receiver

If the receiver computes a signature mismatch but the URL and secret are correct:

  • Confirm the secret is the value Observer's webhook subscription page shows, not a copy with whitespace.
  • Confirm the receiver computes HMAC-SHA-256(secret, raw_body) and compares against the X-Observer-Signature header value (after the sha256= prefix).
  • Confirm the receiver hashes the raw request body, not a re-serialised JSON. Some web frameworks parse and re-serialise request bodies on the way to the handler; the recomputed signature does not match.

Step 5: subscription is disabled

A subscription whose enabled flag is off does not deliver events. The subscription edit page exposes the toggle. If a subscription was disabled while debugging, re-enable it to resume deliveries; new events fire from the moment it is re-enabled, not backfilled.

Reading the audit log

Webhook delivery actions write audit log rows under the webhook category. Audit log rows carry the failing receiver URL and the response status, which helps when correlating across multiple subscriptions.