Runbook Authoring
Produces the procedure someone follows under pressure, at night, with no context and no author to ask.
Deliverable
One Markdown document, runbook.md, in the structure set out under Output below. One runbook covers one procedure; a second procedure gets a second document.
Required inputs
- The procedure — the operation to be written down, and the condition that starts it.
- The system it runs against — access to it, or to someone who will execute the steps while you record them.
- Who will run it — the role, and what that role is permitted to do without asking.
If any of the three is absent, stop and report which one. A runbook written without knowing who runs it either assumes access they do not have or withholds steps they need.
Optional inputs
- Previous incidents where this procedure was used, and what went wrong
- The monitoring, alerts or symptoms that indicate this runbook applies
- The change and approval process this operation must respect
- A non-production environment the steps can be rehearsed in
- The escalation structure, by role
- The audit or compliance record this operation must leave behind
Where an optional input is absent, the runbook says so at the point it matters: an unrehearsed step is marked untested, an unknown escalation route is written unknown in section 7. It is never filled in with what is likely.
Execution
1 — Fix the boundary. When this runbook applies and — just as important — when it does not, naming the nearest procedure for the cases it does not cover. An operator confidently following the wrong runbook does more damage than one who stops.
2 — List the preconditions and the access. What must be true before the first step, and what permissions, credentials or approvals the operator needs in hand already. Anything that would have to be requested mid-procedure is a precondition, not a step.
3 — Write the steps, each with its verification. One action per step, in order, with the command or control named exactly and the variable parts written as <name>. Under each one, how the operator confirms it worked. A step with no verification is a step whose failure is discovered later, by something else.
4 — Write the decision points as branches. Where the procedure forks, state the observable condition and what to do on each side. No step may say as appropriate: under pressure that phrase means the operator is guessing, and the runbook has handed the decision back to the person least able to make it.
5 — Write the rollback. For every step that changes state, how to undo it and whether it can be undone at all. A step that cannot be reversed is marked IRREVERSIBLE and carries its own confirmation before it runs.
6 — Handle failure in place. Under each step, what to do when it fails, rather than one section at the end. Then the escalation: who to involve, by role and not by name, at what point, and what to tell them.
7 — Rehearse, then mark what was not rehearsed. Execute the procedure where it is safe to, correct what the run reveals, and mark untested every step you could not execute. Record what must be captured afterwards: times, decisions taken, and the state the system was left in.
Output
runbook.md, in this order:
- 1. Scope — when this applies, when it does not, and where to go instead
- 2. Preconditions — what must be true, and the access the operator needs
- 3. Roles — who runs this, and who must be told that it is running
- 4. Steps — numbered, one action each, with the verification and the failure response beneath
- 5. Decision points — the observable condition, and the branch taken in each case
- 6. Rollback — per state-changing step: how to undo it, or
IRREVERSIBLE
- 7. Escalation — the trigger, the role to contact, and what to hand them
- 8. After the run — what to record, where, and who reviews it
- 9. Rehearsal status — which steps were executed, which are
untested, and when this was last run
Validation
The runbook is ready when all of these hold:
- Every step in section 4 states one action and how to verify it
- No step contains
as appropriate, if needed or any other instruction that returns the decision to the operator
- Every state-changing step appears in section 6, with its rollback or as
IRREVERSIBLE
- Every branch in section 5 names an observable condition, not a judgement
- Section 7 names roles, never individuals
- Every step not executed during rehearsal is marked
untested in section 9
Fail the run if a step changes state with no entry in section 6, or if any step was written without being executed and without being marked untested.
Failure handling
- No access to the system — write the procedure from the supplied material, mark every step
untested, and state at the top that this runbook has never been executed. Say plainly that it must be rehearsed before it is relied on.
- A step cannot be rehearsed safely — mark it
untested with the reason, and write the verification the operator should expect from the step's definition rather than from a run.
- The procedure has no rollback — say so in section 6 rather than leaving it empty, mark the step
IRREVERSIBLE, and add the confirmation the operator must make before it.
- No escalation structure supplied — write
unknown in section 7 and list it as blocking. A runbook that ends on a failed step with no route onward leaves the operator stranded.
- Conflicting accounts of the procedure — record each version with its source, mark the point
disputed, and do not merge them into one sequence. A merged procedure nobody has run is worse than two that someone has.