Skip to content

Guide

A blameless postmortem guide

How to write an incident review that explains what happened, why it made sense at the time, and which changes will help.

SRE Academy6 min readIncident response

A postmortem is the written record of an incident: what users hit, what people knew at the time, which conditions had to line up, and which changes you’re actually going to make. Write it for the next person who wasn’t in the room. If the only result is a meeting people already attended, you can skip the document. You shouldn’t skip the document.

What “blameless” means

Blameless means you don’t stop at “someone made a mistake.” The people in the incident were working with the goals, the tools, and the information they had. The review asks how the system made that action a reasonable one. A name in the timeline is a fact about who did the work. It isn’t a verdict.

Blameless doesn’t mean nobody owns anything afterward. Action items have owners. Repeating the same failure without changing the system is a management problem. The review itself isn’t where you score the person who was on call.

If the write-up can be used in a performance review, people will edit the timeline. Say, out loud and in the doc’s header, that this document isn’t an input to compensation or promotion. Then don’t use it that way.

Hindsight is a distorting lens

After the outcome is known, the signals look obvious. They weren’t, not to the person who had to choose among three plausible stories at 2 a.m. Write what was visible then. Include the signals that pointed somewhere else, and the ones nobody saw because they weren’t on a dashboard anyone had open.

“They should have been more careful” is a feeling, not a finding. Care doesn’t scale, and it doesn’t survive the next incident. A finding names a condition you can change: a default, a guard, a missing signal, a procedure nobody had practiced, a blast radius that was larger than anyone had said.

There usually isn’t one root cause

A production incident in a system with real dependencies is a combination. A bad change, a slow detector, a runbook that skipped a step, an assumption two teams didn’t share. Any one of those might have been survivable. The outcome needed several.

Call them contributing factors. Say which ones were necessary for this outcome, if you can tell. “Root cause: human error” ends the learning, because the implied fix is “don’t do that,” and the next person will be just as human.

“Five whys” can be a useful prompt and a harmful ritual. It’s useful when it pushes you past the first symptom. It’s harmful when the format forces a single chain and a single villain at the bottom. Use it as one tool. Stop when you reach a change you can make. Allow more than one chain.

The short essay How Complex Systems Fail is the cleanest public statement of this. It’s a few pages. It’s worth reading before you facilitate your next review.

A structure a tired person can finish

Write the doc while the details are still available, ideally within a few days, and timebox it. A perfect essay next month loses to an honest document this week. Use one timezone in the timeline and say which one.

Summary. A few sentences a busy reader can use. What broke, who was affected, how long the user impact lasted, and what restored service. If you don’t have a measured duration, say you don’t. Don’t invent a precise number to make the summary look finished.

Impact. Who couldn’t do what, starting when and ending when. Separate “we were paged” from “users were affected.” They’re often different clocks. Note anything you got lucky about: the incident hit a low-traffic hour, a customer didn’t notice, a second failure didn’t quite trigger.

Timeline. Detection, mitigation, and the end of user impact, each marked. Include the change that contributed, not only the response. A timeline that starts at the page and ignores the deploy from an hour earlier is missing the incident.

What went well. Detection that was fast. A rollback that had been rehearsed. A status update someone actually sent. This section is easy to skip and useful later, because you’ll want to keep those things when you change the system.

What went poorly. Confusion about who was deciding. A dashboard that lied, or one nobody could find. Two people making conflicting changes. Write the condition, not a character sketch.

Where you got lucky. This is the section teams drop, and it’s often the one that should scare you. The migration finished before the disk filled. The bad config was only on one region. Luck isn’t a control. If you needed it, say so.

Contributing factors. The combination, in plain language. Link to logs, changes, and graphs that support each one. A factor you can’t point at is a hypothesis. Label it as one.

Action items. Each one has an owner, a change to the system, and a way to know it’s done. Separate “make the next one shorter” from “make this class of failure harder.” A ticket titled “be more careful when deploying” isn’t an action item.

Action items that change the system

Prefer changes that don’t depend on someone remembering:

  • A guard that rejects the dangerous config.
  • A default that’s safe if a field is left blank.
  • A smaller blast radius: one region, a small percentage of traffic, a flag you can turn off without a deploy.
  • A signal that shows the failure mode you actually hit, not a cousin of it.
  • A procedure that’s short enough to follow, and that someone has followed on purpose before the night you need it.

“Add monitoring” is too vague to finish. “Page when checkout error rate would exhaust the 30-day error budget inside six hours, measured at the edge” can be done or not done. You can tell which.

Not every action is a code change. Sometimes the missing piece is a decision that was never written down: who is allowed to roll forward during an incident, or which customer communication the incident commander can send without another meeting. Write that down too. It’s still a change to the system. The system includes the team.

The meeting

The facilitator shouldn’t also be the person reconstructing their own hardest hour, if you can avoid it. A peer from another team, or the incident commander if they weren’t deep in the debugging, is a better chair. The people who were in the incident are there to supply what they saw, not to defend it.

Read the timeline before you debate causes. Most arguments in these meetings are two people holding different clocks. A shared timeline ends that faster than a debate does.

Keep the meeting short enough that it happens. Ninety minutes is a reasonable ceiling for a serious incident. If you need more, you probably need a clearer doc, not a longer meeting. End with the action list visible, owners named, and a date when someone will check that the items moved.

Share the document. A private postmortem teaches one team once. A shared one is how the next team avoids building the same trap. Redact secrets and customer data. Don’t redact the mechanism.

What to leave out

Leave out blame, sarcasm, and speculation you haven’t labeled as speculation. Leave out a single “root cause” box if filling it in would flatten a real combination into a slogan. Leave the document out of performance management.

If legal, security, or a customer commitment constrains what you can publish, say what the constraint is and share as much of the mechanism as you’re allowed to. An empty doc helps no one. A doc that hides the only useful paragraph doesn’t either.

When you’re ready to write one, start from the postmortem template. The prompts are the questions. They aren’t sample answers.

Further reading

Written by SRE Academy. Published .

Templates for this topic

  • Postmortem template

    A blank incident write-up. Impact, a timeline, contributing factors, and action items with owners.

  • Incident commander checklist

    A one-page checklist for the person coordinating a production incident. Declare, assign, mitigate, hand off, close.

More guides