Table of Contents
- The Single Cause That Was Not
- Why Human Error Is Never the Answer
- Blameless Means Something Specific
- What a Useful Postmortem Contains
- Counterfactuals Are a Trap
- Action Items That Actually Happen
- Incident Roles Under Pressure
- On-Call That Does Not Burn People Out
- Common Pitfalls
- Conclusion
- Frequently Asked Questions
Key takeaway: Incidents in complex systems result from multiple contributing conditions aligning. Identifying one root cause stops the investigation early and produces a fix for the last thing in the chain rather than for the conditions that made the chain possible.
The Single Cause That Was Not
A postmortem concludes: “Root cause: an engineer deployed a configuration change with an incorrect value.”
That is accurate and it is not the cause. It is the final event in a sequence that includes: the configuration format permitted a value that could not work, validation did not catch it, the deployment applied to all instances simultaneously rather than progressively, monitoring did not detect the resulting failure for eleven minutes, the alert routed to a rotation with no runbook for this failure, and rollback required a manual procedure nobody had performed recently.
The engineer’s mistake was necessary for the incident. It was not sufficient. Every one of those other conditions was equally necessary, and each represents an improvement opportunity that a single-cause conclusion discards.
This is the central limitation of root cause thinking. Complex systems maintain multiple layers of defence, and incidents occur when several fail together. Naming one as the root implies the others were fine, which is precisely the wrong lesson — the others were the reason a single mistake became an outage.
The productive reframing is to ask what conditions made this possible and which defences did not hold. That question has many answers, and each answer is actionable.
Why Human Error Is Never the Answer
“Human error” as a conclusion is always available and never useful, because it terminates the investigation at the point where it should begin.
When someone makes a mistake, the questions worth asking are about the system that surrounded them. What information did they have available? What did the interface suggest? Was there time pressure? Was this action reversible? Did anything validate the input? Would a different competent person have done the same thing?
That last question is the most diagnostic. If a reasonable person with the same information would plausibly have made the same choice, the problem is the situation rather than the person. And in most incidents, the honest answer is yes — which is why blaming individuals produces no improvement.
There is a well-established distinction here between systems where mistakes are cheap and systems where they are catastrophic. A deployment tool that applies changes to all production instances immediately, with no validation and a manual rollback, converts an ordinary typing error into an outage. The same error in a system with progressive rollout, schema validation, and automatic rollback produces a failed canary that nobody outside the team notices.
The engineer is identical in both cases. The system determines the outcome.
Blameless Means Something Specific
“Blameless postmortem” is widely adopted as a phrase and frequently misunderstood in both directions.
It does not mean no accountability. Teams remain responsible for their systems, and the organisation is entitled to expect improvement.
It does mean that the investigation assumes everyone acted reasonably given what they knew at the time, and that the purpose is understanding rather than attribution.
The practical justification is straightforward: blame suppresses information. An engineer who expects consequences for reporting a mistake reports less, later, and with less detail. Since the person closest to the incident holds the most valuable information about it, anything that reduces their willingness to share it makes the organisation less safe.
The behaviours that signal genuine blamelessness: names are absent from the written record, replaced by roles. The engineer involved participates in the investigation rather than being subject to it. Discussion focuses on why an action seemed correct at the time rather than on why it was wrong in hindsight. And the output is a list of system changes rather than a list of people who should be more careful.
Where organisations get this wrong subtly: a postmortem that is formally blameless while everyone in the room knows who did it, and the individual is quietly marked. That produces the appearance of psychological safety without the substance, and engineers detect it quickly.
What a Useful Postmortem Contains
The structure that produces learning rather than documentation:
A timeline with sources. What happened when, and how you know — log entries, alert timestamps, chat messages. Precision matters because the gaps in the timeline are frequently where the learning is.
Impact, quantified. Duration, affected users, failed requests, revenue effect where measurable. This calibrates how much investment the improvements justify.
What people believed at each point. Not what was true — what the responders thought was happening. Divergence between belief and reality is where observability gaps become visible, and it is the section most postmortems omit.
Contributing conditions. Plural. Everything that had to be true for this to happen, including the defences that did not hold.
What went well. Genuinely. Which detection worked, which mitigation was effective, what would have been worse without existing safeguards. This identifies what to preserve and it is not merely morale management.
Where you got lucky. The parts that could easily have been worse. An incident that occurred during business hours when the team was available would have been substantially worse at 3am, and that is a finding.
Action items with owners and dates. Discussed in the next section.
The “what people believed” section deserves emphasis because it is where the most transferable learning lives. An incident where responders spent forty minutes investigating the wrong subsystem has identified an observability problem more precisely than any dashboard review would.
Counterfactuals Are a Trap
Postmortems fill with statements about what should have happened, and these feel like analysis while producing nothing.
“The engineer should have noticed the incorrect value.” “The alert should have fired sooner.” “Someone should have checked the runbook.”
Each of these describes a world that did not occur. None explains why the actual world unfolded as it did, and none produces a change.
The corresponding productive formulations:
| Counterfactual | Investigative question |
|---|---|
| “Should have noticed the value” | What did the interface show? Was the error detectable? |
| “Alert should have fired sooner” | What was the detection threshold and why? |
| “Should have followed the runbook” | Did it exist, was it findable, was it correct? |
| “Should have tested this” | What would that test have required? Why was it absent? |
| “Should have rolled back faster” | What did rollback require? Had anyone done it recently? |
The pattern is that counterfactuals assign fault to a decision, while investigative questions examine the conditions that shaped it. Only the second kind produces an action item.
A related discipline: resist hindsight bias in reading the timeline. Signals that appear obviously significant in retrospect were one of many signals at the time, most of which were noise. The question is not why nobody noticed the important one — it is what made it indistinguishable from the others.
Action Items That Actually Happen
Most postmortem action items are never completed. The reasons are consistent and addressable.
Too large. “Redesign the deployment system” will not happen. “Add schema validation to the configuration parser” will.
No owner. Assigned to a team rather than a person means assigned to nobody.
No date. Items without deadlines drift indefinitely.
Not in the normal backlog. Items tracked in the postmortem document rather than in the work tracking system are invisible during planning.
No prioritisation basis. Everything marked important means nothing is.
What improves completion rates: fewer items, each small enough to complete in days. A named individual owner. A date. Entry into the same backlog as feature work, prioritised against it explicitly. And a review of outstanding items at the next incident, because the strongest argument for completing them is an incident that a previous action item would have prevented.
A useful categorisation for prioritisation: items that would have prevented the incident, items that would have detected it faster, and items that would have mitigated it faster. Detection and mitigation items are frequently cheaper than prevention and reduce impact across a whole class of future incidents rather than one specific cause.
Incident Roles Under Pressure
Incidents degrade when everyone investigates simultaneously and nobody coordinates. Defined roles prevent that, and the roles need to be lightweight enough to actually be used.
Incident commander. Coordinates, decides, and delegates. Explicitly does not debug — the moment the commander starts investigating, coordination stops. This is the role teams most often skip and most need.
Communications lead. Handles stakeholder updates, status page, and customer communication. Without this role, the commander spends the incident answering questions instead of managing it.
Operations lead. Performs the technical investigation and remediation.
Scribe. Records the timeline as it happens. Reconstructing a timeline afterwards from memory and scattered logs is substantially harder and less accurate.
For small incidents one person holds several roles, which is fine. The value appears in larger incidents, where the separation between deciding and debugging is what keeps the response coherent.
Two practices that matter as much as the roles: an explicit declaration that an incident has started, with a named commander, so everyone knows who is coordinating. And explicit handover when someone is fatigued, because incident response degrades sharply after a few hours and people do not notice their own degradation.
On-Call That Does Not Burn People Out
On-call sustainability determines whether your reliability practice survives, because the alternative is attrition among exactly the people who understand your systems.
Alert volume is the primary variable. More than a few pages per shift is unsustainable. If volume is high, the correct response is reducing alerts rather than expanding the rotation.
Every page must be actionable. A page requiring no action should not be a page. This is the single most effective intervention available, and it requires ruthlessly deleting alerts that fire without consequence.
Runbooks for every alert. The person paged at 3am should not be deriving the diagnosis from scratch. A runbook linked from the alert, kept current, is what makes an unfamiliar alert survivable.
Rotation size of at least six. Fewer means on-call too frequently to recover between shifts.
Compensation or time off. Being available outside working hours is work. Treating it as an unpaid expectation produces resentment and eventual departure.
Explicit permission to escalate. Nobody should feel obliged to solve everything alone. Escalation should be normal rather than an admission of inadequacy.
Follow the sun where possible. Nobody being woken is better than fair distribution of being woken.
The metric worth tracking is pages per shift outside working hours, trended over time. Rising, it predicts attrition. It is also a reasonable proxy for system health, since a system that pages constantly is a system with unaddressed problems.
Common Pitfalls
Concluding with one root cause. Stops the investigation where it should broaden.
Human error as an explanation. Terminates analysis at the point it should start.
Counterfactual statements. Describe a world that did not happen and produce no change.
Action items too large to complete. Guarantees they will not be.
No owner or date. Same outcome.
Postmortems only for major incidents. Near-misses are the cheapest learning available.
Formally blameless while informally attributing. Engineers detect this and share less.
Conclusion
Root cause analysis asks a question complex systems do not answer. Incidents happen when several defences fail together, and naming one as the root implies the others were adequate — when the others are precisely what allowed a single mistake to become an outage.
Ask instead what conditions made this possible and which defences did not hold. That question produces many findings, each actionable, and it directs attention to the layers that would have contained the failure rather than to the last person who touched it.
Then keep the improvements small enough to complete, owned by a person, dated, and in the same backlog as everything else. Prioritise detection and mitigation alongside prevention, because faster detection reduces impact across every future incident rather than preventing one specific recurrence.
And protect the conditions that make honest reporting possible. The person closest to the incident holds the most valuable information, and anything that makes them reluctant to share it makes the whole system less safe.
Frequently Asked Questions
Should every incident get a postmortem? Every incident with user impact, and near-misses where a small difference would have caused impact. Near-misses are the cheapest learning available because nothing broke.
How quickly should a postmortem be written? Within a few days, while memory is accurate and attention is present. Longer delays produce reconstruction rather than recollection.
Who should write it? Someone involved in the response, reviewed by others who were present. Written by someone uninvolved, it becomes documentation rather than analysis.
How is genuine blamelessness maintained when someone made a clear mistake? By asking what made the mistake possible and what made it consequential. A competent person making a reasonable error in a system that amplified it is a system finding, not a personal one.
Should postmortems be visible across the organisation? Internally, yes — the learning transfers and visibility demonstrates that incidents are treated seriously. External publication is a separate decision with different considerations.
How many action items should a postmortem produce? Three to five completable items beats fifteen aspirational ones. Completion rate matters more than coverage.
What if the same incident recurs? That is a strong signal the previous action items were not completed or not sufficient. Recurrence should trigger examining the earlier postmortem’s follow-through rather than only the new incident.
