# Regression Suite Audit Reports what an existing automated suite is actually worth — which tests protect something, which train people to ignore failures, and which should be deleted. ## Deliverable One Markdown document, `regression-suite-audit.md`, in the structure set out under **Output** below. It reads the suite against the levels set by **Test Strategy Definition**, and its findings feed the next revision of that strategy. ## Required inputs - **The automated suite** — the test code, or a listing detailed enough to show what each test asserts and at which level it runs. - **What the suite is meant to protect** — the behaviour whose breakage it exists to catch. - **Who owns the suite**, at least by role — the person who can act on a recommendation to delete a test. If any of the three is absent, stop and report it. Run history is not required, but its absence changes what this audit is allowed to conclude. ## Optional inputs - Run history: pass, fail and skip per test, over a period the requester states - The defects that reached production since the suite has existed - Measured runtime, per test and for the suite as a whole - The maintenance record: which tests were edited, and whether the edit followed a behaviour change or a refactor - Ownership per test or per area - The test strategy the suite was built against Without run history the audit reports structural findings only — what a test asserts, what it duplicates, what it is coupled to — and says on its first line that no behavioural judgement was possible. No test is ever called flaky, or called a test that never fails, without the history that shows it. ## Execution **1 — Inventory the suite.** One line per test: name, level, what it asserts, the area it covers, and its owner where one is known. A test whose assertion cannot be stated in a line is recorded `assertion unclear` and carried forward — it is the first candidate for rewriting, because nobody can act on its failure either. **2 — Classify each test against its run history.** Never failed, failed on real regressions, fails intermittently, fails on changes unrelated to what it asserts, skipped or disabled. Every classification cites the history behind it. A test with no history is recorded `no history` and receives no behavioural classification at all. **3 — Separate behaviour from implementation.** For each test, decide whether it asserts something a user or a calling system depends on, or something internal: call order, private structure, exact markup, the shape of an intermediate value. Record each implementation assertion with the refactor that would break it while the behaviour stayed correct. **4 — Price the intermittents.** List every intermittently failing test with how the team currently responds to it: rerun, ignore, investigate. Where the response is rerun or ignore, say plainly that the suite has stopped being a signal in that area, and that a real failure there would be treated the same way. **5 — Price the suite.** Runtime per test and in total, from the supplied measurements only; the maintenance the record shows; and where in the pipeline the cost lands. A runtime nobody measured is `unknown`, and a total built from unknowns is reported as unknown rather than summed. **6 — Check what escaped.** For each production defect supplied, decide whether a test could have caught it, at which level, and what gap let it through. Gaps something has actually broken through are the gaps that matter; gaps in areas nothing has ever broken are listed separately and below them. **7 — Recommend per test.** `keep`, `fix`, `rewrite` or `delete`, each with the evidence behind it and the cost of acting on it. A `delete` names what stops being protected and who must accept that. A recommendation with no evidence is an opinion about someone else's code. ## Output `regression-suite-audit.md`, in this order: - **1. Input and date** — which suite was audited, what history was supplied and for what period, who audited it and when - **2. Register** — one line per test: name, level, what it asserts, classification, recommendation, and the evidence cited - **3. Tests that never fail** — per test: what it asserts, and whether it guards something stable or asserts nothing - **4. Intermittent tests** — per test: the pattern the history shows, and how the team currently responds - **5. Implementation-coupled tests** — per test: the internal detail asserted, and the refactor that would break it - **6. Cost** — measured runtime per test and in total, maintenance observed, and where the cost lands; `unknown` where nothing was measured - **7. Escaped defects** — per defect: what the suite missed, at which level it could have been caught, and the gap responsible - **8. Coverage gaps** — gaps something has broken through, then gaps nothing has broken, kept apart - **9. Recommendations** — per test: `keep`, `fix`, `rewrite` or `delete`, the evidence, the cost of acting, and what a `delete` stops protecting - **10. Missing information** — what could not be judged, and who can supply what is needed ## Validation The audit is ready when all of these hold: - Every test in the supplied suite appears exactly once in section 2 - Every classification in section 2 cites the run history it rests on, or reads `no history` - Every recommendation names its evidence, and every `delete` names what stops being protected - No test is described as intermittent or as never failing without the history that shows it - Section 6 carries measured figures only; anything unmeasured reads `unknown` - Section 1 states whether run history was supplied, and what the audit therefore could not conclude Fail the run if a judgement appears without the evidence behind it, or if a runtime figure appears that nobody measured. ## Failure handling - **No run history** — produce sections 1, 2, 5, 9 and 10 from the code alone, mark sections 3, 4 and 6 `no history supplied`, and state on the first line that the audit is structural and no test was judged on how it behaves. - **No access to the test code** — audit from the listing supplied, mark every finding `reported`, and state that nothing was read. A suite judged from its names is judged on what someone hoped each test did. - **The suite and the history disagree** — where the history names tests that no longer exist, or tests carry no history at all, record both, name both sources, and raise it in section 10. Do not reconcile them by dropping one side. - **A test nobody owns** — record `owner unknown` in section 2 and list it in section 10. An unowned test that fails is the one that gets rerun until it passes. - **Partial material** — audit what was supplied, mark the rest `INCOMPLETE — pending `, and name the part of the suite that was not examined. An audit that covers half a suite and says so is usable; one that implies it covered all of it is not.