# Failure Mode & Guardrail Analysis

Enumerates how an AI system fails, what signal detects each failure, and what happens next — including the failures with no signal at all.

## Deliverable

One Markdown document, `guardrail-specification.md`, in the structure set out under **Output** below. It takes the output of **Agent & Prompt Specification** and of **Evaluation Set & Metric Design**.

## Required inputs

- **The designed system** — its scope, its tools and its output, in the form the Agent Specification carries.
- **What a wrong answer costs**, per output type, and who absorbs it.
- **The escalation route** — the named role that receives a case the system must not decide.

If either of the first two is absent, stop and report it. A guardrail is sized by the cost of the failure it prevents; with no stated cost there is nothing to size it against.

## Optional inputs

- The Evaluation Plan, and the distribution the system was measured on
- Logs of real failures from this system or from the process it replaces
- The monitoring and alerting the runtime already provides
- The false-positive tolerance the operators will accept
- The content the system reads that originates outside the organisation
- The rate at which a reviewer can actually process escalations

Every optional input that is absent becomes an entry under **Open questions**. A failure is recorded as undetected sooner than it is recorded as covered by a nearby guardrail.

## Execution

**1 — Enumerate the failures, not the risks.** Work through the categories this system is exposed to: a confident wrong answer, a refusal of a legitimate request, an instruction arriving inside data the system reads, an output carrying information the recipient may not receive, a degradation that raises no error, and behaviour on input unlike anything it was evaluated on. Each becomes a numbered entry with how it would present.

**2 — Give each failure a detection signal.** State the observable thing that changes when this failure occurs — in the output, in the tool calls, in the timing, or in what happens downstream. Where no signal exists, the entry is recorded `UNDETECTED` and stays that way. A failure is not detected by a guardrail that watches something else.

**3 — Specify the guardrail.** What the check examines, where it runs — before the model, after it, or around the tool call — and what it does when it fires: block, downgrade, annotate, or route to a person. A check that only writes a line to a log is monitoring, and is labelled monitoring rather than counted as a guardrail.

**4 — Price the guardrail in false positives.** Every check that stops a bad case stops some good ones. State what legitimate traffic this check will catch, what the user sees when it does, and how they get past it. A check with no stated cost is the one that gets disabled by whoever is on call the first bad night.

**5 — Define the fallback.** What happens once the guardrail fires and the primary path is closed: the deterministic route, a previous answer, a partial result with its limits stated, or an honest refusal that says what to do next. Silence and a generic error are not fallbacks and are recorded as gaps.

**6 — Route the human escalation.** For each failure that reaches a person: who, through what channel, what they see when it arrives, what they are able to do about it, and what happens if nobody answers. An escalation with no time bound and nothing behind it is a queue.

**7 — Mark the boundary of the evaluated distribution.** State what the system was measured on and what it was not, and the signal that an input has left that region. Outside it no measured performance applies, and the specification says what the system does there rather than assuming the behaviour carries over.

## Output

`guardrail-specification.md`, in this order:

- **1. System and date** — what is being analysed, against which specification, and when
- **2. Failure register** — per failure: what it is, how it presents, what it costs, who absorbs it
- **3. Detection** — per failure: the signal, where it is observed, or `UNDETECTED`
- **4. Guardrails** — the check, where it runs, and what it does when it fires
- **5. False-positive cost** — the legitimate traffic each check catches, and the route past it
- **6. Fallbacks** — per closed path: the behaviour, and what the user is told
- **7. Escalation** — failure, role, channel, what they see, what they can do, the time bound
- **8. Distribution boundary** — what was evaluated, what was not, and the signal for leaving it
- **9. Undetected failures** — the failures with no signal, and what would be needed to detect them
- **10. Open questions** — what could not be specified, and who can supply it

## Validation

The specification is ready when all of these hold:

- Every failure in section 2 appears in section 3 with a signal or as `UNDETECTED`
- Every guardrail in section 4 states where it runs and what it does when it fires
- Every guardrail has a false-positive cost in section 5, or states that it does not block
- Every closed path in section 6 has a fallback that is not a generic error
- Every escalation in section 7 names a role and a time bound
- Section 9 is either non-empty or states explicitly that every failure has a signal

Fail the run if a failure appears as mitigated while section 3 records no signal for it, or if any figure appears that did not come from an input.

## Failure handling

- **No system specification** — stop. Report that failure analysis has nothing to analyse, and that enumerating failures from a description invents the system.
- **No stated cost of a wrong answer** — produce sections 1 to 3 and 10, and mark the guardrails `UNSIZED`. Name who must state the cost.
- **A failure with no detection signal** — record it in section 9 as `UNDETECTED`, state what would be needed to detect it, and do not move it into section 4 on the strength of a nearby guardrail.
- **No reviewer capacity for escalations** — specify the escalation, record the capacity as `unknown` in section 10, and state that an escalation route with no capacity behind it converts a detected failure into a delayed one.
- **Partial material** — specify every section the material supports, mark the rest `INCOMPLETE — pending <question>`, and deliver.