Skip to content

Guide

What an incident commander does

What the incident commander owns during a production incident, from roles and mitigation to a clean handoff.

SRE Academy6 min readIncident response

On a small incident, one person can debug and still tell everyone else what’s going on. That stops working once several people are responding, users are affected, or the channel is noisy enough that two fixes get attempted at once. That’s when you want an incident commander. The commander coordinates. Someone else holds the keyboard.

This guide is the role, not a severity chart. Severity labels only work if you defined them before the incident. The commander’s job is to use the scale you have, and to change the level when the facts change.

The job

The incident commander keeps a current picture and makes the tradeoffs that shouldn’t wait for a complete diagnosis.

In practice that means:

  1. Say that this is an incident, in one channel, and name the severity if your team has a scale.
  2. Make sure each role you need is filled, or say out loud that it isn’t and you’re accepting that.
  3. Keep the picture current: who is affected, what you know, what you’re doing, and what you’ve decided not to do.
  4. Keep the work in that one channel. A side conversation gets summarized back into it, or it didn’t happen.
  5. Choose mitigation over a full explanation when users are still affected. A rollback, a failover, a flag, or shedding load can be the right call before anyone can draw the mechanism.
  6. Set the pace. Ask for status on a timer. Stop two people from restarting two different things. Ask for help before the current group is stuck.
  7. Hand off cleanly if the incident outlasts the people. A handoff includes impact, what’s in flight, and what you’ve ruled out.
  8. Declare the incident over when user impact has stopped, or when you’ve deliberately accepted what’s left. Put a review on the calendar before everyone disappears.

The commander can be wrong. Say the decision in a form other people can object to: “I’m choosing the rollback because charges are still failing. Say so now if that makes this worse.” Silence isn’t consensus. Waiting for consensus during an outage is also a decision, and it’s often a slow one.

Roles worth naming

You don’t need an org chart. You need to know who is allowed to decide, who is changing the system, who is writing it down, and who is telling people outside the channel.

Incident commander. Coordination and decisions. Not also the only person typing commands, once the incident is big enough to need a commander at all.

Operations. The technical work. This person, or this small group, is who the commander asks for options and then points at one of them. They aren’t a committee.

Communications. Status for the people who need it and aren’t in the debugging channel: support, leadership, customers if you owe them an update. On a small incident this can be the commander. On a large one it shouldn’t be, because a customer update and a mitigation decision will want to happen at the same minute.

Scribe. The timeline, with timestamps. Memory is a bad log, and the postmortem is much cheaper to write if this role existed. It’s a real job. Don’t assign it to the person who is also rolling back the deploy.

Collapse roles when the incident is small. Split them when it isn’t. The failure mode to avoid is one person who is debugging, deciding, and composing the customer note, while the channel goes quiet.

The first minutes

You don’t need a speech. You need a few sentences in the channel:

  • This is an incident. Here is the severity, and here is why.
  • Here is the user impact, as far as we know. Here is what we don’t know yet.
  • I am the incident commander. Named person is on operations. Named person is writing the timeline.
  • The next check-in is at a specific time, not “soon.”

Then stop people from starting five fixes. Pick the mitigation with the best chance of restoring users, given what you know, and say which alternatives you’re not doing. “We’re rolling back. We’re not also restarting the fleet.” Two simultaneous changes make the next data point useless, because you won’t know which one mattered.

If you don’t know whether a rollback is safe, that uncertainty is a fact to say, not a reason to freeze. Ask the person who shipped it, or read the change, on a short clock. Then decide.

Mitigation before the full story

A complete explanation is a gift you give the postmortem. Users need the service back. Those are different tasks, and the commander protects the order.

Good mitigations are reversible and small enough to reason about: the previous build, a feature flag, traffic shifted off a bad region, a dependency you can shed. A mitigation you can’t undo is just another incident you started on purpose. Prefer the one you can reverse, unless the reversible options are gone.

Don’t announce a cause you can’t support. “We’re seeing failed charges after the 13:40 release, and we’re rolling that release back” is a status. “The database is corrupted” is a conclusion. If you aren’t there yet, don’t say it to customers or to the channel. People will build the next hour of work on whatever sentence you used.

Keep one channel honest

Pick the channel at the start and put the link in the page, the ticket, and the status mail. Debugging in a side thread means the scribe can’t see it and the next commander can’t inherit it. If a useful conversation happens somewhere else, one of those people pastes the conclusion back.

Status updates are for humans who aren’t watching every line. A steady cadence beats a perfect essay. Say what changed since the last update, what users still can’t do, and when you’ll update again. If nothing changed, say that. Silence reads as confusion.

The communications person shouldn’t invent optimism. “We’re working on it” is weaker than “charges are still failing, rollback is in progress, next update at 14:40 UTC.”

Handing off

Incidents that cross a shift change are where details die. The handoff is a short, spoken exchange, written into the channel as it happens. Cover:

  • Current user impact, and when it started.
  • What you’re doing right now, and who is doing it.
  • The leading idea, and what you’ve already ruled out.
  • Changes that are in flight and haven’t finished.
  • The next decision, and the time you’ll make it if nothing new shows up.
  • Who has which role after this handoff. The old commander is done deciding when they say they are.

Don’t hand off by disappearing. Don’t accept a handoff you can’t restate. If you can’t explain the impact and the action in flight, you don’t have the incident yet.

Ending it

User impact stopping is the usual end. Sometimes a residual risk is acceptable and you choose to stop anyway: a workaround is in place, the remaining errors are within the error budget, or further changes at this hour are more dangerous than the leftover symptom. Say which of those it is. “I think it’s fine” isn’t a close.

Then:

  • Note the end time in the timeline.
  • Thank people in one line, without turning it into a speech.
  • Name who will draft the postmortem, and when the review is. The blameless postmortem guide is the shape of that doc. The checklist is the short version of this role for the next time.

If the same incident reopens an hour later, it’s the same incident. Reopen the channel and the timeline. A fresh document for the sequel is how you lose the first half of the story.

What the commander doesn’t do

Don’t vanish into a terminal while nobody is updating the channel. Don’t let two mitigations race. Don’t grade people in the channel. Don’t skip the handoff because it feels almost over. Almost over is when incidents get longer.

And don’t treat this role as a promotion. The best commander in the room is the person who can keep the picture straight and make a reversible call. It isn’t automatically the most senior engineer, and it isn’t automatically the person who knows the failing service best. That person is often more useful on operations, where the knowledge changes the fix.

Further reading

  • Managing Incidents, in the Site Reliability Engineering book, is the public description this role grew out of.
  • Being On-Call covers the rotation around the incident. A guide to that rotation isn’t on this site yet.
Written by SRE Academy. Published .

Templates for this topic

  • Postmortem template

    A blank incident write-up. Impact, a timeline, contributing factors, and action items with owners.

  • Incident commander checklist

    A one-page checklist for the person coordinating a production incident. Declare, assign, mitigate, hand off, close.

More guides