Testing & Quality — Skills

Test Strategy Definition

Decides how a change will be tested before anyone writes a test — which risks are covered at which level, what is deliberately not automated, and who owns each part.

Deliverable

One Markdown document, test-strategy.md, in the structure set out under Output below. It sets the levels that Test Case Specification writes cases for, and the frame the Regression Suite Audit judges an existing suite against.

Required inputs

What is being built or changed — the system, service or change under test, in whatever written form exists.
The risk it carries — what it costs when this fails: who is affected, what breaks, and whether the damage is recoverable.
Who owns quality for this work, at least by role. Someone has to be able to say the strategy is wrong.

If any of the three is absent, stop and report it. A strategy written without a stated risk ends up covering whatever is easiest to reach.

Optional inputs

The architecture: components, integrations, external services, data stores
The environments available, and how each differs from production
Test data: what exists, how it is created, and what is restricted
The pipeline this work passes through, and what it currently blocks on
Existing automated tests and their run history
Coverage targets, regulatory obligations or contractual commitments, each with the person or document that states it

Absent optional inputs become entries under Open questions. Coverage targets are recorded only when the requester states one; this skill never sets a number itself, and a category with no stated target reads no target supplied.

Execution

1 — List the risks, not the features. For each risk: what fails, who it affects, what it costs, and whether it is recoverable. A risk nobody can name a consequence for is not a risk — take it off the list rather than testing against it.

2 — Assign every risk to a level. Put each risk at the cheapest level that can actually detect it: the closer to the code, the faster the feedback and the narrower the evidence. Record the level against the risk. A risk that no level covers is written down as uncovered, not quietly moved to a level that cannot see it.

3 — Decide what is deliberately not automated. Name the checks that stay manual or exploratory, and the reason for each: changes too often to be worth encoding, needs a judgement no assertion can make, costs more to automate than the risk is worth. An undocumented gap becomes an accident later; a documented one is a decision.

4 — Name the environment and the data each level needs. Per level: where it runs, what data it needs, where that data comes from, and what is restricted. Where the environment differs from production, record the difference and the class of defect it hides.

5 — Give every level an owner. Per level: who writes the tests, who runs them, and who fixes them when they fail. A level with no named owner stops being maintained, and nobody notices until it is needed.

6 — Define what the pipeline blocks on. Which levels gate a change and which only report, what happens to a change that is blocked, who may override a gate, and where that override is recorded. A gate nobody can override is bypassed some other way; an override nobody records is not a gate.

7 — Set the flakiness rule before there is flakiness. How an intermittent failure is detected, who is told, whether it is quarantined, who fixes it, and the window the team agrees for that fix. Write the window the team states, never one chosen here. A suite without this rule teaches people to ignore failures.

Output

test-strategy.md, in this order:

1. Scope and date — what is under test, what is excluded, who wrote the strategy and when
2. Risks — one line each: what fails, who it affects, what it costs, recoverable or not
3. Levels — per level: the risks it covers, what it explicitly does not cover, and why that level was chosen
4. Not automated — what stays manual or exploratory, with the reason recorded against each entry
5. Environments and data — per level: environment, data, where the data comes from, and how the environment differs from production
6. Ownership — per level: who writes, who runs, who fixes
7. Pipeline gates — which levels block, which report, what a blocked change does next, who may override and where it is recorded
8. Flakiness rule — detection, notification, quarantine, owner, and the fix window as stated by the team
9. Coverage targets — only those the requester stated, each with the person or document that set it; otherwise no target supplied
10. Open questions — what could not be decided, what it blocks, and who decides it

Validation

The strategy is ready when all of these hold:

Every risk in section 2 appears in section 3 or section 4, or is listed as uncovered
Every entry in section 4 carries a reason
Every level in section 3 has a named owner in section 6
Section 7 states, for each level, whether it blocks or only reports
Section 9 carries no figure that the requester did not state
Section 10 is either non-empty or states explicitly that nothing is outstanding

Fail the run if a risk is covered by no level and is not recorded as uncovered, or if a coverage figure appears without the person who set it.

Failure handling

No stated risk — stop. Report that the strategy has nothing to choose a level against, and name who must state the risk.
No access to the environments or the pipeline — write sections 5 and 7 from what was reported, mark them reported throughout, and state that nothing was verified and every gate described may differ from the one that runs.
Contradictory instructions about what must be automated — record both, name both sources, and raise the contradiction in section 10 as blocking. Do not settle it by choosing the stricter one.
A coverage target is demanded but nobody will state one — write no target supplied in section 9 and name who must set it. Never supply a figure to close the section.
Partial material — produce every section the material supports, mark the rest INCOMPLETE — pending <question>, and deliver. A strategy that names its gaps can be reviewed; one that filled them cannot.

Download .md

Acceptance Criteria Authoring

Turns a requirement into criteria that can be checked without argument — including the states a requirement usually forgets.

Deliverable

One Markdown document, acceptance-criteria.md, in the structure set out under Output below. It is the input to Test Case Specification.

Required inputs

The requirement, as written — a story, a ticket, a specification paragraph or a written request, in the words of whoever wrote it.
A named accepter — the person whose judgement settles whether a criterion is met. Without one, criteria are suggestions.

If either is absent, stop and report it. Never derive criteria from the implementation: code states what happens, not what was agreed.

Optional inputs

The existing behaviour the change sits beside
The roles and permissions that apply to it
Data rules: required fields, formats, limits, retention
Interface designs, the copy, or a walkthrough of the flow
Regulatory, contractual or accessibility obligations
Defects raised before in the same area

Every absent optional input narrows what can be asserted, and each gap becomes an entry under Questions. A missing rule is never replaced by a sensible default; the default is agreed to in review and disputed after release.

Execution

1 — Restate the requirement in one sentence. Who, what they can do, and what is different afterwards. If it will not fit in one sentence it carries more than one requirement: split it, and write criteria for each part separately.

2 — List the words that cannot be checked. Terms such as quickly, valid, appropriate, as needed, and any list ending in etc., have no observable meaning. Each one becomes a question naming who can settle it. None of them is settled here by choosing the most likely reading.

3 — Write the success path. One criterion per behaviour, in given / when / then form. The then states something a person or a calling system can observe — a screen, a record, a response, a message. A then saying the system handles it asserts nothing.

4 — Write the boundaries. The smallest and largest accepted input, the first value rejected on each side, empty input, maximum length, and the limits of every field. Each boundary is expressed as the rule that governs it and cites the input that supplied that rule; where no input supplied a limit, the boundary becomes a question rather than a figure.

5 — Write the error paths. For each way the behaviour can fail: what the user sees, what the system records, and what state the data is left in. A requirement that describes only success has not been specified, it has been imagined.

6 — Write the states the requirement forgot. No permission. Not authenticated. No data yet. Partial data. A concurrent change by someone else. An interrupted flow resumed later. Each is either a criterion or an explicit out-of-scope line naming who decided that.

7 — Check each criterion on its own. One criterion asserts one thing, can be checked without running another first, and does not need the implementation read to be understood. Merge criteria that only mean something together; split criteria that assert two things; rewrite any criterion whose truth depends on how it was built.

Output

acceptance-criteria.md, in this order:

1. Requirement and source — the requirement verbatim, where it came from, who accepts it, and the date
2. Restatement — the one-sentence form the criteria were written against, and the split if there was one
3. Success criteria — numbered, each in given / when / then form
4. Boundary criteria — per limit: the rule, the first value outside it, and the input that supplied the rule
5. Error criteria — per failure: what the user sees, what is recorded, what state the data is in
6. State criteria — permission, authentication, no data, partial data, concurrency, interruption: each covered or explicitly out of scope
7. Out of scope — what this requirement deliberately does not cover, and who decided
8. Questions — every uncheckable word and every rule no input supplied, with what it blocks and who can settle it

Validation

The set is ready when all of these hold:

Every criterion is in given / when / then form and names an observable outcome
Every criterion can be checked on its own, without another criterion running first
Sections 4, 5 and 6 are each non-empty, or state that the requirement has no such case and who confirmed that
No limit, format or threshold appears that an input did not supply
Every uncheckable word found in section 1 appears in section 8 or has been replaced by a rule
Section 8 is either non-empty or states explicitly that nothing is outstanding

Fail the run if a criterion still contains a word listed in section 8, or if a value appears that no input supplied.

Failure handling

No requirement supplied — stop. Report that criteria cannot be written from an implementation, and name what is needed.
No accepter named — produce all eight sections and mark section 1 UNACCEPTED. Criteria in that state may not be used to sign work off.
The requirement contradicts itself or a supplied rule — write criteria for neither reading. Record both, name both sources, and raise the contradiction in section 8 as blocking.
No access to the existing behaviour — write criteria for the change alone, and state in section 8 that its interaction with what already exists was not examined and remains unspecified.
Partial material — write every criterion the material supports, mark the rest INCOMPLETE — pending <question>, and deliver. A set with named gaps can be reviewed; a set that filled them gets agreed to and then argued about.

Download .md

Test Case Specification

Turns agreed criteria into cases someone can run — each with its data, its expected result, and the criterion it exists to cover.

Deliverable

One Markdown document, test-cases.md, in the structure set out under Output below. It takes the output of Acceptance Criteria Authoring and becomes the material the Regression Suite Audit later judges.

Required inputs

The criteria or specification to cover — sections 3 to 6 of acceptance-criteria.md are the expected form; any written statement of agreed behaviour will do.
The level the cases are written for — taken from the test strategy where one exists. A case with no level cannot be placed, automated or owned.

Without both, stop. Never write cases from the implementation: a case derived from the code asserts what the code does, and it keeps passing whether that is right or wrong.

Optional inputs

The test data available: what exists, how it is created, what is restricted
The environment the cases will run in, and how it differs from production
Existing cases covering the same area
Known defects in the area and the conditions that triggered them
The interfaces under test: screens, endpoints, scheduled jobs, messages
Who will run the cases, and whether by hand or through the test runner

An absent optional input does not stop the specification. Each case records the data it needs; where the source of that data is not known, the case is marked data source unknown and listed under Missing information rather than written against data somebody assumes exists.

Execution

1 — Take the criteria one at a time. Every criterion gets at least one case. A criterion that cannot be turned into a case is reported back as untestable as written, with what makes it untestable — it is never approximated by a case that tests something near it.

2 — Divide the inputs into classes. Group the values the behaviour treats alike, write one case per class, then add a case at each edge of a class and at the first value outside it. The aim is a small set whose coverage can be argued, not a large set whose coverage is hoped for.

3 — State the preconditions. Per case: the account and its permissions, the state of the data, and the state of every system the case depends on. A precondition nobody can create is a blocker, and it is recorded as one rather than left for whoever runs the case to discover.

4 — Name the data. Per case: what data it needs, where it comes from — a fixture, generated, seeded, or requested from whoever owns it — and whether another case may reuse it. Data that does not yet exist is named as required, never invented inside the case.

5 — Write the steps and the expected result. One action per step, in the order performed. The expected result is stated precisely enough to fail on: the state, record, message or response that must be observed. A result saying the behaviour works as expected cannot fail, and so cannot pass.

6 — Write the cleanup. What the case creates, what it must remove or reset, and what it may leave behind. A case that leaves state behind turns its own mess into the next case's failure, and the next case is the one that gets investigated.

7 — Trace both ways. Each case names the criterion it covers; each criterion names the cases covering it. Criteria with no case are coverage gaps. Cases with no criterion are either a missing criterion or a case nobody needs — both are reported, neither is deleted here.

Output

test-cases.md, in this order:

1. Input and date — which criteria are covered, the level the cases are written for, who wrote them and when
2. Cases — per case: identifier, title, level, preconditions, data, steps, expected result, cleanup, and the criterion covered
3. Coverage analysis — per criterion: the classes and boundaries identified, and the cases covering each
4. Criteria with no case — the criterion, what prevents a case, and what is needed to write one
5. Cases with no criterion — the case, what it asserts, and the decision the accepter must make about it
6. Data requirements — per data set: what it is, where it comes from, whether it is restricted, and who owns it
7. Blocked cases — cases that cannot run, the precondition or access they need, and who can supply it
8. Missing information — what stopped a case being specified, and who can unblock it

Validation

The suite is ready when all of these hold:

Every criterion in the input appears in section 2 or section 4
Every case names its preconditions, its data source and its cleanup, or states that it needs none
Every expected result names something observable, and would fail if the behaviour changed
Every case names the criterion it covers, or appears in section 5
No value, limit or format appears in a case that the input did not supply
Sections 4 and 5 are present even when empty, and say so when they are

Fail the run if an expected result cannot be observed, or if a case depends on data that no input supplied.

Failure handling

No criteria supplied — stop. Report that cases cannot be specified from an implementation, and name what is needed first.
A criterion is untestable as written — do not approximate it. Record it in section 4 with what makes it untestable, and return it to whoever wrote it.
No environment or no test data — specify the cases in full, mark them NOT RUN, and list the access needed in section 7. A specified case that has not run is honest; a case recorded as passing without running is worse than no case.
Contradictory criteria — write no case for the contradiction. Record both readings in section 8 as blocking, name both sources, and leave the choice to the accepter.
Partial material — specify what the material supports, mark the rest INCOMPLETE — pending <question>, and report the coverage the suite actually has rather than the coverage it looks like it has.

Download .md

Defect Triage & Prioritisation

Turns a pile of reported defects into an ordered list with the reasoning attached — and sends back the reports that cannot be triaged with the exact question that unblocks each one.

Deliverable

One Markdown document, triage-report.md, in the structure set out under Output below. It is read alongside the Regression Suite Audit, which uses the defects that escaped to judge the suite.

Required inputs

The defect reports to triage — in whatever form they arrive: tickets, an export, a list, messages.
Who decides business consequence — the person whose call sets priority when impact and urgency disagree.

Without both, stop. A triage with nobody to settle priority orders the list by whoever complained most recently.

Optional inputs

Per report: steps to reproduce, environment, build or version, logs, screenshots
Access to an environment where a report can be reproduced
The severity and priority scales already agreed, with their definitions
The existing defect record, for finding duplicates
Who is affected, and whether a workaround exists
The dates or commitments these defects sit against

Where no scale is supplied, the report states the scale it used and asks for the agreed one. Where a defect cannot be reproduced it is recorded as not reproducible; it is never given a severity so that the list looks complete.

Execution

1 — Normalise every report. One entry each: what was observed, what was expected, where, on which build, who reported it, and when. Fields the report does not carry are written unknown. They are not inferred from the wording of the description.

2 — Reproduce, or say you could not. Attempt each defect on the stated environment and build. Record reproduced with the steps that worked, not reproducible with what was tried, or not attempted with the reason. Nothing moves further through the triage without one of the three.

3 — Return the reports that cannot be triaged. A report with no observed and expected pair, no environment, or no route to the behaviour goes back with the single question that unblocks it, addressed to the person who filed it. A general request for more detail comes back as a general answer.

4 — Merge the duplicates. Group the reports describing the same underlying failure, name the one they merge into, and keep every reporter attached so each is told when it closes. Reports that look alike but differ in trigger stay separate, with the difference written down.

5 — Judge severity from observed impact. What the defect does when it occurs: data lost, a wrong result recorded, a task blocked, a display wrong. Severity is assigned only to a defect that was reproduced. A defect nobody has reproduced carries a request for reproduction in place of a severity.

6 — Judge priority from business consequence. Who is affected, how often, whether a workaround exists, and what it costs to leave the defect open. Priority is recorded as its own value and may disagree with severity. The disagreement is the useful part of the triage: write it out, never average it into one number.

7 — Order the list and show the reasoning. State the rule the order follows before the order itself, then give each defect its position and the reason it holds it. An order with no stated rule cannot be challenged, only resented.

Output

triage-report.md, in this order:

1. Input and date — which reports were triaged, against which build, by whom, and when
2. Triaged defects — per defect: identifier, observed, expected, environment, build, reproduction status, severity, priority, and the reason for each judgement
3. Not reproducible — per defect: what was tried, on which environment and build, and the question returned to the reporter
4. Returned for information — per report: the single missing fact, and who was asked for it
5. Duplicates — per group: the reports merged, the one they merge into, and the reporters to notify
6. Severity and priority disagreements — per defect: what each judgement says, and who must settle the order
7. Ordering — the rule used, then the ordered list with the reason for each position
8. Missing information — what blocked the triage, and who can unblock it

Validation

The triage is ready when all of these hold:

Every input report appears exactly once, in section 2, 3, 4 or 5
Every defect in section 2 carries a reproduction status
No severity appears against a defect that nobody reproduced
Severity and priority are recorded as separate values, and neither is derived from the other
Every entry in section 4 names one specific missing fact rather than asking for more detail
Section 7 states the ordering rule before the order

Fail the run if a severity appears without a reproduction, or if a report from the input is missing from the output.

Failure handling

No environment to reproduce in — mark every defect not attempted — no environment, leave severity pending reproduction throughout, order on business consequence alone, and name the access needed in section 8.
A report with no steps — return it in section 4 with the exact question. Never reconstruct the steps from the description; a reconstructed reproduction confirms the reader, not the defect.
Two reports contradict each other about the same behaviour — keep both, name both reporters, and raise the contradiction in section 6 as blocking. Do not merge them into one report to make the conflict go away.
Nobody will decide business consequence — publish severity, leave every priority undecided, and name who must decide. A priority invented here is an opinion with a rank on it.
Partial material — triage what the material supports, mark the rest INCOMPLETE — pending <question>, and deliver. A short triage that states what it could not judge is usable; a complete-looking one that guessed is not.

Download .md

Regression Suite Audit

Reports what an existing automated suite is actually worth — which tests protect something, which train people to ignore failures, and which should be deleted.

Deliverable

One Markdown document, regression-suite-audit.md, in the structure set out under Output below. It reads the suite against the levels set by Test Strategy Definition, and its findings feed the next revision of that strategy.

Required inputs

The automated suite — the test code, or a listing detailed enough to show what each test asserts and at which level it runs.
What the suite is meant to protect — the behaviour whose breakage it exists to catch.
Who owns the suite, at least by role — the person who can act on a recommendation to delete a test.

If any of the three is absent, stop and report it. Run history is not required, but its absence changes what this audit is allowed to conclude.

Optional inputs

Run history: pass, fail and skip per test, over a period the requester states
The defects that reached production since the suite has existed
Measured runtime, per test and for the suite as a whole
The maintenance record: which tests were edited, and whether the edit followed a behaviour change or a refactor
Ownership per test or per area
The test strategy the suite was built against

Without run history the audit reports structural findings only — what a test asserts, what it duplicates, what it is coupled to — and says on its first line that no behavioural judgement was possible. No test is ever called flaky, or called a test that never fails, without the history that shows it.

Execution

1 — Inventory the suite. One line per test: name, level, what it asserts, the area it covers, and its owner where one is known. A test whose assertion cannot be stated in a line is recorded assertion unclear and carried forward — it is the first candidate for rewriting, because nobody can act on its failure either.

2 — Classify each test against its run history. Never failed, failed on real regressions, fails intermittently, fails on changes unrelated to what it asserts, skipped or disabled. Every classification cites the history behind it. A test with no history is recorded no history and receives no behavioural classification at all.

3 — Separate behaviour from implementation. For each test, decide whether it asserts something a user or a calling system depends on, or something internal: call order, private structure, exact markup, the shape of an intermediate value. Record each implementation assertion with the refactor that would break it while the behaviour stayed correct.

4 — Price the intermittents. List every intermittently failing test with how the team currently responds to it: rerun, ignore, investigate. Where the response is rerun or ignore, say plainly that the suite has stopped being a signal in that area, and that a real failure there would be treated the same way.

5 — Price the suite. Runtime per test and in total, from the supplied measurements only; the maintenance the record shows; and where in the pipeline the cost lands. A runtime nobody measured is unknown, and a total built from unknowns is reported as unknown rather than summed.

6 — Check what escaped. For each production defect supplied, decide whether a test could have caught it, at which level, and what gap let it through. Gaps something has actually broken through are the gaps that matter; gaps in areas nothing has ever broken are listed separately and below them.

7 — Recommend per test. keep, fix, rewrite or delete, each with the evidence behind it and the cost of acting on it. A delete names what stops being protected and who must accept that. A recommendation with no evidence is an opinion about someone else's code.

Output

regression-suite-audit.md, in this order:

1. Input and date — which suite was audited, what history was supplied and for what period, who audited it and when
2. Register — one line per test: name, level, what it asserts, classification, recommendation, and the evidence cited
3. Tests that never fail — per test: what it asserts, and whether it guards something stable or asserts nothing
4. Intermittent tests — per test: the pattern the history shows, and how the team currently responds
5. Implementation-coupled tests — per test: the internal detail asserted, and the refactor that would break it
6. Cost — measured runtime per test and in total, maintenance observed, and where the cost lands; unknown where nothing was measured
7. Escaped defects — per defect: what the suite missed, at which level it could have been caught, and the gap responsible
8. Coverage gaps — gaps something has broken through, then gaps nothing has broken, kept apart
9. Recommendations — per test: keep, fix, rewrite or delete, the evidence, the cost of acting, and what a delete stops protecting
10. Missing information — what could not be judged, and who can supply what is needed

Validation

The audit is ready when all of these hold:

Every test in the supplied suite appears exactly once in section 2
Every classification in section 2 cites the run history it rests on, or reads no history
Every recommendation names its evidence, and every delete names what stops being protected
No test is described as intermittent or as never failing without the history that shows it
Section 6 carries measured figures only; anything unmeasured reads unknown
Section 1 states whether run history was supplied, and what the audit therefore could not conclude

Fail the run if a judgement appears without the evidence behind it, or if a runtime figure appears that nobody measured.

Failure handling

No run history — produce sections 1, 2, 5, 9 and 10 from the code alone, mark sections 3, 4 and 6 no history supplied, and state on the first line that the audit is structural and no test was judged on how it behaves.
No access to the test code — audit from the listing supplied, mark every finding reported, and state that nothing was read. A suite judged from its names is judged on what someone hoped each test did.
The suite and the history disagree — where the history names tests that no longer exist, or tests carry no history at all, record both, name both sources, and raise it in section 10. Do not reconcile them by dropping one side.
A test nobody owns — record owner unknown in section 2 and list it in section 10. An unowned test that fails is the one that gets rerun until it passes.
Partial material — audit what was supplied, mark the rest INCOMPLETE — pending <question>, and name the part of the suite that was not examined. An audit that covers half a suite and says so is usable; one that implies it covered all of it is not.

Download .md