# Model & Provider Selection Chooses between candidates against written requirements, with evidence for every requirement — and not one figure taken from memory. ## Deliverable One Markdown document, `model-selection-report.md`, in the structure set out under **Output** below. It takes the system's requirements and the output of **Evaluation Set & Metric Design**. ## Required inputs - **The requirements the system must meet**, as stated by the requester: latency, cost ceiling, context, modality, data residency, privacy terms and availability commitments, each with the value that makes it pass. - **The candidates to evaluate**, and the documentation or contract terms the requester holds for each. - **The evaluation set or task sample** the candidates will be measured on. Without stated requirements, stop. Nothing is the best model; a model is only adequate against a requirement someone wrote down. If candidates are missing, report that and never supply them from memory. ## Optional inputs - The Evaluation Plan and the metric it defines - Measurements the requester has already taken, with the conditions they were taken under - Existing contracts, procurement constraints or approved supplier lists - The volume and shape of the expected traffic - The switching cost the architecture already carries — where the provider appears in the code - Deployment constraints: region, network, self-hosting requirements Where a measurement is absent, the report states what to measure and how, and leaves the cell `unknown`. It is never filled from recollection. ## Execution **1 — Turn requirements into pass conditions.** Each requirement gets a threshold, a unit, the source that set it, and a statement of whether it is a hard condition or a preference. A requirement with no threshold cannot be evaluated against and is returned to the requester. **2 — Fix the candidate list.** Record each candidate the requester named, with the version or identifier they intend to use and the document its terms come from. Do not add candidates that were not supplied, and do not discard one without recording why. **3 — State what must be measured.** For each requirement that cannot be read off a document — quality on the task, latency under the expected shape of traffic, behaviour at the context the task needs — write the measurement procedure: the evaluation harness, the inputs, the conditions, and who runs it. The report specifies the measurement; the requester produces the result. **4 — Read the contractual requirements off the documents.** Residency, retention, training on submitted data, availability commitments, certification, and notice of change. Each with the document and the clause it came from. Where the requester holds no document for a term, the cell reads `unknown — no document supplied`, whatever the provider's public reputation. **5 — Fill the matrix and mark the holes.** Requirement against candidate, with the evidence for each cell: a measurement the requester supplied, a clause in a document, or `unknown`. An `unknown` in a hard requirement stops that candidate being recommended, and the report says so rather than scoring around it. **6 — Recommend against the conditions.** Name the candidate that meets the hard requirements on the evidence present, and state which requirements it meets on measurement, which on document, and which remain unknown. Where no candidate meets them all, say so, and name the requirement that has to move. **7 — Write the revisit conditions and the exit.** The changes that would reopen the decision — a requirement moving, a term changing, a measurement failing to reproduce — and the cost of switching: what is coupled to this provider, what would have to be re-measured, and who owns the migration. ## Output `model-selection-report.md`, in this order: - **1. Decision and date** — what is being selected, for which system, by whom, and when - **2. Requirements** — requirement, threshold, unit, source, hard or preference - **3. Candidates** — each as named by the requester, with version and the documents held - **4. Measurement procedures** — what must be measured, how, under what conditions, by whom - **5. Evidence** — per requirement and candidate: the value, its source, or `unknown` - **6. Gaps** — the cells with no evidence, and what would fill each - **7. Recommendation** — the candidate, the requirements it meets, and on what evidence - **8. Revisit conditions** — what would reopen this decision - **9. Switching cost** — what is coupled to the choice, what must be re-measured, who owns the move - **10. Open questions** — what blocked the comparison, and who can unblock it ## Validation The report is ready when all of these hold: - Every requirement in section 2 has a threshold and a source, or is returned as unevaluable - Every candidate in section 3 was named by the requester - Every cell in section 5 carries a measurement, a document reference, or `unknown` - No performance, price or capability figure appears without the measurement or document it came from - Section 7 states which requirements rest on evidence and which on `unknown` - Section 9 is non-empty whenever a recommendation is made Fail the run if any figure in the report came from recollection rather than a supplied measurement or document, or if a candidate is recommended while a hard requirement is `unknown`. ## Failure handling - **No stated requirements** — stop. Report that there is nothing to select against, and that a comparison with no thresholds ranks candidates by whichever attribute was easiest to find. - **No measurements supplied** — deliver sections 1 to 4, 6 and 10: the requirements, the candidates, the procedures to run, and the holes. Deliver no recommendation. - **No documents for the contractual terms** — mark every such cell `unknown — no document supplied` and list the documents to obtain. A provider's public statements are not evidence of a term. - **A requirement no candidate meets** — report it plainly, name the requirement, and return it to whoever set it. Do not recommend the closest candidate as though the requirement had been met. - **Partial material** — evaluate what the evidence supports, mark the rest `INCOMPLETE — pending `, and deliver.