Alerting on Causes Produces Noise; Alerting on Symptoms Produces Pages
Key takeaway: Alert on what users experience, not on what a machine is doing. A metric that can be abnormal without anyone being affected should be a dashboard, not a page.
Why Cause-Based Alerts Overwhelm
A monitoring system is configured with thresholds: CPU above eighty percent, memory above ninety, disk above eighty-five, queue depth above a thousand.
Each fires regularly. Most times nothing is actually wrong — a batch job pushed CPU up, a cache warmed and used memory, a queue absorbed a burst and drained. The service was healthy throughout.
Responders learn that these alerts usually mean nothing. That learned dismissal is the real damage, because it applies equally to the one occurrence that mattered.
What Symptom-Based Alerting Looks Like
The question to ask is whether a user can tell. If the answer is no, it is not a page.
| Alert | Type | Should page? |
|---|---|---|
| CPU above 80% | Cause | No — dashboard |
| Memory above 90% | Cause | No — dashboard |
| Error rate above 2% for 5 min | Symptom | Yes |
| p99 latency above SLO | Symptom | Yes |
| Checkout success rate dropped | Symptom | Yes |
| Disk will fill in 4 hours | Predictive | Yes, as a ticket |
| Certificate expires in 14 days | Predictive | Yes, as a ticket |
The predictive category deserves separate handling. A disk that will fill in four hours is not currently affecting anyone, so it does not need a page at three in the morning — it needs a ticket that gets picked up promptly. Distinguishing “act now” from “act today” reduces night-time paging substantially without reducing coverage.
Error Budgets as the Alerting Basis
An SLO defines the reliability target, and the error budget is the permitted shortfall. Alerting on budget burn rate rather than on instantaneous thresholds solves two problems at once.
A brief spike that consumes a negligible fraction of the monthly budget does not page anyone, because it does not threaten the target. A sustained degradation that will exhaust the budget within hours pages immediately, because it does.
This gives the alert a meaningful severity derived from actual impact rather than from someone’s guess about a threshold. It also makes the alert defensible: the page happened because the service is on track to miss its commitment, which is a statement anyone can evaluate.
Use multiple burn-rate windows. A fast window catches severe sudden degradation; a slower one catches gradual erosion that a fast window would miss. Both reference the same budget.
Keeping the Set Healthy
Every page should have a documented action. If the responder’s only option is to acknowledge and wait, the alert is informational and belongs elsewhere.
Review alerts that fire and are resolved without action. Those are false positives regardless of whether the metric was accurate, and each one erodes trust in the whole set.
Track pages per on-call shift as an explicit metric. A rotation receiving more than a couple of actionable pages per shift is not sustainable, and the number tends to grow silently until someone measures it.
The Bottom Line
Page on user-visible symptoms measured against SLO burn rate, route predictive warnings to tickets rather than pages, keep cause metrics on dashboards for diagnosis, and delete any alert that fires without producing an action.
