# Evaluation Set & Metric Design Fixes how an AI system is measured before anyone has a result to defend. ## Deliverable One Markdown document, `evaluation-plan.md`, in the structure set out under **Output** below. It takes the output of **Agent & Prompt Specification** and feeds **Failure Mode & Guardrail Analysis**. ## Required inputs - **The task the system performs** — what goes in, what comes out, and in what form. - **The definition of a correct answer**, from the person who owns the decision the output serves. - **Access to a record of real requests**, or the named person who can describe what real traffic looks like. If the definition of correct is missing, stop and report it: a metric without one measures agreement with whoever wrote the metric. None of the three is filled in by inference. ## Optional inputs - The Agent Specification, where one exists, and the refusals it defines - Logs or samples of real requests, with their distribution over request types - Existing labels, prior annotations, or decisions people already made on the same cases - The reviewers available to judge outputs, and their qualification for the task - Reports of known failures in the current process - What one evaluation pass costs to run, and how often it must run Every optional input that is absent becomes an entry under **Open questions**. This skill specifies the set; it never fabricates its contents. ## Execution **1 — Write the metric and its blind spot.** State what the metric counts and how it is computed. Beside it, state what the metric deliberately does not capture — the failures it will score as successes. A metric with no stated blind spot is trusted further than it can carry. **2 — Split the set by what each part tests.** Cases that reflect ordinary traffic, cases that probe a known failure, and cases the specification says must be refused. Each stratum is scored separately: a system that passes the ordinary cases and fails the refusals has not passed, and a single combined figure hides exactly that. **3 — Sample so the set matches production.** Draw from the record of real requests in the proportions they occur, not in the proportions that are easy to collect. Where the distribution is unknown, mark the sampling `unrepresentative — distribution unknown` and stop describing the result as an estimate of production behaviour. **4 — Add the cases sampling will not produce.** Inputs at the edge of the defined scope, malformed and empty inputs, contradictory instructions, content carrying an instruction addressed to the system, and every request the specification says must be refused. Each is written as a case with its expected outcome, and for a refusal the expected outcome is the refusal. **5 — Establish ground truth and name who sets it.** For each case, what makes an answer correct and the person or document that determines it. Where judgement is involved, more than one qualified reviewer labels the same cases, their agreement is measured, and the rubric is rewritten wherever they disagree. A label set produced by one unchecked reviewer measures that reviewer. **6 — Size the set against the difference that matters.** State the smallest change in the metric that would change a decision, and size the set so that a change that small can be told apart from sampling noise. Where the set cannot be that large, state what it can and cannot detect, and never report a difference below that floor as a result. **7 — Set the holdout discipline.** Which cases are never seen during development, who holds them, how often they may be run, and what happens to the set once a decision has been tuned against it. A set that has been optimised against is a development set from that moment, and is relabelled rather than quietly reused. ## Output `evaluation-plan.md`, in this order: - **1. Task and date** — what is being evaluated, by whom, and when - **2. Definition of correct** — the rule, and who owns it - **3. Metric** — what it counts, how it is computed, and what it does not capture - **4. Set composition** — the strata, what each tests, and how each is scored - **5. Sampling** — the source, the proportions, and whether they match production - **6. Adversarial and edge cases** — the case, what it probes, and the expected outcome - **7. Ground truth** — how it is established, who sets it, and the agreement measured between reviewers - **8. Sizing** — the smallest difference that matters, and what this set can detect - **9. Holdout** — what is held back, who holds it, and the rules for running it - **10. Open questions** — what could not be specified, and who can supply it ## Validation The plan is ready when all of these hold: - Section 3 states a blind spot as well as a metric - Every stratum in section 4 says how it is scored and whether it can be traded against another - Section 5 states whether the sample is representative, or says that it is not known - Every case in section 6 carries an expected outcome, the refusals included - Section 7 names who sets ground truth, and the agreement measure wherever judgement is used - Section 8 names the smallest difference that matters, or records it as `unset` Fail the run if the plan invents the contents of a case rather than stating the criteria that case must satisfy, or if any figure appears that did not come from an input. ## Failure handling - **No definition of correct** — stop. Report that the metric has nothing to measure against, and name who owns the definition. - **No record of real traffic** — specify the strata and the criteria, mark section 5 `unrepresentative — distribution unknown`, and state that results describe the set and not production. - **Only one reviewer available** — record the agreement as `unmeasured`, state which parts of the metric rest on a single judgement, and name that as a limit on every result the set produces. - **Contradictory definitions of correct** — record both with their sources, and raise the contradiction in section 10 as blocking. A metric cannot average two definitions. - **Partial material** — specify every section the material supports, mark the rest `INCOMPLETE — pending `, and deliver.