Observability & SRE

One Alert Threshold Cannot Catch Both Fast and Slow Failures

Key takeaway: A single burn-rate window forces a choice between catching fast severe failures and catching slow gradual ones. Using several windows simultaneously catches both without the false-positive cost of over-tuning either alone.

The Single-Window Dilemma

An SLO burn-rate alert compares current error rate against the rate that would exhaust the monthly error budget by some point. A short window — checking burn rate over the last five minutes — reacts quickly to a severe outage and also reacts to any brief, harmless blip that happens to fall in that window, producing false pages for noise that resolved before anyone could act.

A long window — checking burn rate over six hours — smooths out that noise and also delays detection of a genuinely severe issue by the length of the window itself, since the average has to accumulate enough bad data points before it crosses the threshold, by which time a severe issue may have already consumed a dangerous fraction of the budget.

Neither window alone is correct for every failure shape, because failures genuinely occur at different speeds and severities.

Running Several Windows Together

The practical solution most mature SRE practices converge on uses multiple windows simultaneously, each with its own burn-rate threshold, so different failure shapes trigger the window best suited to detect them.

Window Detects Typical threshold
Short (5–15 min) Severe, fast-onset outages High burn rate required
Medium (1 hour) Moderate sustained degradation Moderate burn rate
Long (6–24 hours) Slow gradual erosion Lower burn rate, sustained

A severe outage burning through the budget rapidly trips the short window’s high threshold almost immediately. A milder but sustained degradation that would not trip the short window’s high bar accumulates enough burn over the medium or long window to trip those instead. Requiring the same event to trip both a short and a longer window before paging — rather than either alone — further reduces false positives from transient blips that briefly spike the short window without genuinely threatening the budget.

Why Requiring Multiple Windows to Agree Matters

A single short-window trigger alone still produces meaningful false-positive volume from genuine but brief anomalies — a deployment’s health check flapping for ninety seconds during a rollout, for instance. Requiring that the short window’s elevated burn rate also shows up, even modestly, in a concurrently checked longer window filters out these transient spikes while still reacting quickly to genuinely severe and sustained issues, since a truly severe outage will show elevated burn in both windows almost immediately.

Setting the Actual Thresholds

The threshold values themselves should derive from your error budget policy directly — specifically, from how much budget you are willing to have consumed before paging becomes necessary at each window’s timescale — rather than from an arbitrary round number. A monthly budget allowing a certain total error rate translates mathematically into an hourly burn rate that would exhaust it within a defined number of hours, and that calculation is what should set the threshold, not intuition about what number sounds reasonable.

The Bottom Line

Configure burn-rate alerting across multiple windows — short, medium and long — rather than a single threshold, and require agreement across at least two windows before paging to filter transient noise while still catching genuinely severe fast-onset failures. Derive the actual threshold values from your error budget policy’s maths rather than from an arbitrary number that happened to feel right.

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button