Incident response and debugging

How to set severity: impact, not difficulty

Severity is a statement about the blast radius, not about how hard the bug is to fix. The scale most teams use runs SEV-1 (critical, many customers cannot use the product, executives and public status get involved) down through SEV-3 (a minor or partial problem service owners handle in hours) to cosmetic SEV-5s that become tickets. PagerDuty and Atlassian both anchor the top of the scale on customer impact and scale of users affected, and Atlassian even puts response targets on it: for its highest levels, start investigating within 15 minutes of a page and publish an initial status update within an hour.

A common tiebreaker teams add is whether a workaround exists. Checkout completely down with no way through is a SEV-1. Search is broken but customers can still browse by category and buy is a SEV-2, because the core job still gets done another way.

Wrong "This bug is really gnarly and will take days, so it's a SEV-1." Difficulty is not severity. A hard-to-fix typo that nobody sees is low severity; a one-line config change that takes down payments is a SEV-1 while it is live. Set severity from who is affected and how badly, then let that drive who you wake and how loudly.

Mitigate first, diagnose second

The single most common way engineers mishandle an incident is by treating it as a debugging puzzle. During an outage your first job is to stop customer impact, and root cause can wait. Google's SRE guidance is blunt about the order: "Stop the bleeding, restore service, and preserve the evidence for root-causing."

When a deploy went out shortly before the symptoms started, that deploy is your prime suspect, and rolling it back is usually the fastest route to a known-good state. Fixing forward (shipping a new patch on top) is the alternative, and it is the right call when a rollback would be unsafe: most often when the deploy also changed the database in a way the old code cannot read.

Wrong "I won't roll back because we just shipped three features in that release and we'd lose them." A rollback is reversible. You revert now to stop the impact, then re-ship those features once you have the real fix. Refusing to roll back to protect features keeps customers broken to save yourself a redeploy.

What changed? Start from the deploy log

Before you read a single trace, ask what changed. Most incidents trace back to a recent deploy, config flip, feature-flag toggle, or an infrastructure change, so the deploy and change logs are the first thing to open. Change-induced problems are the everyday case; a system that was healthy an hour ago and is broken now almost always broke because something moved.

Then localize with your telemetry. The mechanics of that (scope with metrics, follow one slow or errored trace, confirm with logs joined by trace_id) live in the observability module, which owns the signal stack. Here the discipline that matters under fire is narrower: form one hypothesis, change one thing at a time, and check whether the metric that defines the incident actually recovered before you move on.

Wrong "Let me try three fixes at once to resolve it faster." Batched changes during an incident mean you cannot tell which one helped or hurt, and you can turn one outage into two. One change, then read the graph, then the next.

Who runs the incident

Past a certain size an incident needs someone coordinating who is not also head-down in the logs. That person is the Incident Commander: they hold the current picture, decide, and delegate the actual fixing to a resolver who knows the system. The IC does not need to be the most senior or the most technical person in the room; their authority comes from the role. The failure mode to avoid is one person trying to command and debug at the same time, which is exactly when the coordination falls apart.

Keep the number of people involved small. Paging everyone feels like urgency but mostly wakes engineers who then sit idle, and it makes the call harder to run. Bring in the specific person the fix needs.

On call, acknowledge fast and escalate without shame

On call, the contract is simple: when a page fires, acknowledge it inside the escalation window (PagerDuty suggests around five minutes) so it does not bounce to your backup, then start working or pull in help. If you cannot make progress, escalate. That is the system working, not a personal failure. PagerDuty's own guidance is "Never hesitate to escalate" and there is "no expectation to fix all issues by yourself."

Wrong "Escalating makes me look like I can't handle on call." The hero who sits on a worsening incident for an hour rather than waking the person who owns the system does more damage than the one who escalates in ten minutes. Escalation paths exist to be used; a page you cannot action within your window belongs with someone who can.

Command structure and why unmanaged incidents fail

Google's SRE book names the anatomy of a badly-run incident: responders get so absorbed in the technical problem that nobody handles mitigation, people cannot tell what their colleagues are doing, and engineers freelance changes without telling anyone. The fix is a small role structure, what the book calls recursive separation of responsibilities.

The Incident Commander holds the high-level state and assigns everything they do not personally keep; they coordinate and decide, and they stay off the keyboard. PagerDuty is explicit that the IC's decisions are final and that they are the highest-ranking person on the call regardless of day-to-day rank, that deep technical knowledge is not a prerequisite, and that "You should not be performing any actions or remediations, checking graphs, or investigating logs." A resolver or subject-matter expert does the actual fixing. A communications lead owns stakeholder updates. On longer incidents a scribe keeps the timeline and a planning role tracks follow-ups and handoffs.

Three mechanics make this work under load. There is a recognized command post (one channel or bridge everyone joins). There is a living incident document the IC keeps current, which the SRE book calls the commander's most important responsibility. And status flows up in a structured shape: PagerDuty's resolvers answer a CAN report on request, meaning Condition (current state of the service), Actions (what is being done or needs doing), and Needs (what support the resolver requires). Handoffs are explicit, down to confirming "you are now the incident commander, okay?", so command is never ambiguous.

Wrong "The person who knows the system best should run the incident." The deepest expert is your best resolver, and putting them in command wastes that: they end up head-down in logs while coordination collapses, which is the exact tunnel-vision failure the role structure exists to prevent. Keep your expert fixing and give command to someone whose whole job is the bigger picture.

Rollback vs fix-forward: the cases where rollback bites

Rollback is the default mitigation for a change-induced incident because it is fast and returns you to a state you have seen work. The judgment senior interviewers probe is when that default is unsafe.

The classic trap is a deploy that also migrated the schema. If the release added a NOT NULL column, backfilled it, and new writes depend on it, reverting only the binary leaves the old code facing a schema it was never written for, and the rollback triggers a second, different outage. The same coupling appears when the new code has already written data in a format the old code cannot read. In those cases fixing forward is safer: a feature flag that disables the new path buys you the fastest mitigation without touching the deploy, and a small targeted patch can follow.

The durable answer is to make rollback safe by construction, using expand/contract (also called parallel change) for anything that touches storage:

1. expand   add the column nullable; deploy code that writes it
2. migrate  backfill existing rows in the background
3. contract enforce NOT NULL in a LATER release, once all code writes it

Every intermediate state runs under both the old and the new code, so a rollback at any step lands somewhere both versions understand. Schema changes shipped this way keep the rollback button honest, which is what lets you reach for it during the next incident without a second thought.

Blameless postmortems and why blame corrupts your data

A postmortem is blameless when it assumes, in the SRE book's words, that "everyone involved in an incident had good intentions and did the right thing with the information they had," and looks for contributing causes in the system rather than culprits in the org chart. This is not politeness. Blame changes behavior at the source: once people expect to be punished, they stop reporting near-misses and shade their accounts, and "an atmosphere of blame risks creating a culture in which incidents and issues are swept under the rug." You lose the very signal the postmortem exists to collect.

Trigger a postmortem on impact, not on drama. Common thresholds: user-visible downtime or degradation past a set bar, data loss of any kind, an on-call engineer having to intervene, or a monitoring miss, plus any stakeholder can request one. The write-up captures impact, the timeline, what mitigated it, the contributing causes, and follow-up actions. Two practices decide whether it was worth writing: action items are filed as tracked, owned work with dates (a postmortem whose fixes never ship is theater), and no postmortem is left unreviewed, because the review is where the lessons actually land and where the culture is reinforced.

Wrong "The root cause was that an engineer pushed a bad config." That is a name, not a cause. Ask why a bad config could reach production at all: no validation in the pipeline, no canary or staged rollout, no automated rollback on the error-rate spike. The systemic answers are the ones that stop a class of incidents; the human-error framing stops exactly one person, once, and quietly teaches the rest to say less.

On-call load is a reliability control

Alerting quality is part of incident response because a fatigued on-call responds worse. Google caps operational work: SREs spend at least 50% of their time on engineering, on-call is bounded, and a single on-call should handle at most two incidents per 12-hour shift, on the reasoning that properly handling one incident (mitigation, root-cause, and postmortem) runs about six hours. Blow past that ceiling and you stop staffing for reliability and start burning people, who then miss the serious page buried in the noise.

That sets the bar for what may page: every paging alert should be actionable and urgent, related alerts should collapse toward a 1:1 alert-to-incident ratio, and anything that is neither should be a ticket or nothing. The construction of good alerts, multi-window burn-rate alerting on an error budget so a page fires on symptoms rather than a CPU gauge crossing a round number, belongs to the observability module. The incident-response point is downstream: recurring false pages are a defect you fix like any other, because each one erodes the attention the next real SEV-1 depends on.