Thresholds and dwell

A metric's status flips through three layers in order:

1. Threshold evaluation: the strict-operator rule applied to each pushed value (see threshold operators).
2. Dwell gating: a status only flips after holding the new state for the configured dwell period.
3. Shadow mode (optional): a metric can be marked as shadowed for a window so it does not affect status pages or fire webhook events while operators tune it.

This page covers steps 2 and 3.

Why dwell exists

A naive implementation would publish every status change the agent reports. In practice, metrics flap. A network blip pushes one bad sample, the next sample is fine, the on-call gets paged twice per minute.

Dwell gating requires the new status to hold for a minimum duration before propagating. Configure two values per metric:

• Dwell to breach: how long the metric must report the new status before flipping into a worse band (healthy → degraded or healthy → unhealthy).
• Dwell to recover: how long the metric must report the new status before flipping into a better band (unhealthy → healthy).

The defaults are conservative: 180 seconds to breach, 300 seconds to recover. Asymmetric values ("quick to flag, slow to recover") are appropriate for systems where premature recovery announcements have a higher cost than a delayed unhealthy alert.

Both values are editable per metric in the Thresholds step of the metric form, from 0 to 3600 seconds. A dwell of 0 flips on the next confirming sample. Configuration-as-code documents accept the same values through the optional dwell_seconds_to_breach and dwell_seconds_to_recover keys on each metric entry, and the API returns them on the metric detail endpoint.

Transitions involving no_data or unknown always use the recover dwell, in either direction. A single missed poll never declares a metric down or up.

Status sources

Three statuses come from the strict-operator rule:

• healthy
• degraded
• unhealthy

Two statuses come from collection-layer outcomes, not from values:

• no_data: the agent attempted a probe but produced no value. The reason code is recorded alongside (ECONNREFUSED, ETIMEDOUT, no_data_for_query, etc.).
• unknown: no recent push has arrived for the metric within the expected interval.

Both are operational signals: they surface in the agent dashboard and as metric.no_data webhook events. They are not free with respect to SLOs, though. A no_data sample burns error budget unless the metric's no-data treatment folds it to healthy, and a window with no samples at all burns under the default down unknown-data policy. See SLOs and error budgets.

Stale data tolerance

Dwell gating handles the small flaps. A separate read-time rule handles a much larger gap: what happens when no sample arrives at all, because the agent is crashed, the network between the agent and Observer Cloud is partitioned, or Observer Cloud itself is degraded.

A metric is stale when its last push timestamp is older than a cutoff derived from its push interval: three times the interval, capped at 15 minutes, except that the cap never falls below one minute past the next expected push.

cutoff = min(3 × push_interval, max(15 min, push_interval + 1 min))
stale  = (now - last_push_timestamp) > cutoff

Worked through:

A metric with no push interval set is treated as 15 minutes.

The 3× multiplier gives the agent one full retry-and-backoff window before a missing push is considered a problem. The 15-minute cap stops a moderately slow cadence from masking an outage. The push_interval + 1 term keeps a genuinely slow cadence from being declared stale before its next push is even due: a metric that pushes hourly is not blind at 15 minutes.

Staleness is enforced on both sides. A sweeper stamps the stored status to no_data with reason stale_agent_silent once the cutoff passes, and the status-page and embed renderers apply the same rule at read time, so a page loaded between sweeps still reads correctly.

A stale metric is excluded from the service rollup. It is not counted toward SLO burn. The metric.status_changed and metric.no_data webhooks are not fired on staleness transitions. What does fire: agent.lag_high and agent.offline, which speak to the actual cause. They are operator-facing.

When every metric on a service is stale, the service renders as monitoring_delayed with a "Last known: Operational" caption alongside. See Observer availability for the full trust contract.

Shadow mode

A metric can be marked shadowed until a future timestamp. While shadowed:

• The metric still pushes status to the cloud.
• Status pages do not consume the shadowed metric in the rolled-up page status.
• Webhook events for the shadowed metric are suppressed.
• The metric's history is still recorded for later inspection.

Use shadow mode when introducing a new metric, tuning its threshold, or rolling out a new probe runtime. Once the metric behaves as expected, clear the shadow timestamp and it joins the public status surface.

For the full picture of how these verdicts become the page badge, the history bars, and the SLO budgets, see How status is calculated.