AI & Machine Learning — Skills

AI Use-Case Feasibility Assessment

Decides whether a use case needs a model at all, and whether the data and the error tolerance exist to build one.

Deliverable

One Markdown document, feasibility-assessment.md, in the structure set out under Output below. It is the input to Agent & Prompt Specification.

Required inputs

The use case, in the requester's own words — what the system should produce, for whom, and what decision or action follows from its output.
The data that exists today for this task — where it lives, who may use it, and under what terms.
The acceptable error — what a wrong answer costs and who absorbs it.

If any of the three is absent, report it as missing and never fill it in. A use case with no defined acceptable-error threshold is reported as not yet assessable: without one there is nothing to judge a result against.

Optional inputs

The non-AI process that handles this task now, and what it costs to run
How often the task occurs and how quickly an answer is needed
Labels, annotations or historical outcomes attached to the existing data
Regulatory, contractual or residency terms that bind the data
Who reviews the output before it acts, and what they see when they review it
Earlier attempts at the same task, and why they stopped

Every optional input that is absent becomes an entry under Open questions. It never becomes an assumption, and it never becomes a figure.

Execution

1 — Restate the task as a decision. Write down what the system is asked to produce, who consumes it, and what happens next because of it. A use case that cannot name the action its output changes is not yet a use case — record that and continue.

2 — Test whether a model is needed. Describe the deterministic route — a rule, a lookup, a form, a person following a procedure — that would produce the same output. Where that route exists and meets the acceptable error, it is the recommendation. A model is justified only where the deterministic route fails, and this step names where it fails.

3 — Establish the baseline. Record how the task is performed today and what that achieves, using measurements the requester supplies. Where nothing has been measured, write unknown and name what must be measured before any comparison can exist. A system with no baseline cannot be shown to be an improvement on anything.

4 — Audit the data. For each source: what it holds, whether it is labelled for this task, who may use it for this purpose and under what terms, and whether it represents the population the system will meet in production. Tag each answer verified where a document or a sample was inspected, reported where a person stated it, and unknown otherwise.

5 — Define the acceptable error. Both directions separately — what a wrong answer in each direction costs, who absorbs it, and whether it can be reversed. If the requester cannot state this, the assessment ends here with the verdict not yet assessable, because every later judgement depends on it.

6 — Bound the irreducible ambiguity. Have the same cases answered independently by more than one qualified person and record where they disagree. Where competent people disagree, no system agrees with all of them, and the target sits below that ceiling. Where this has not been done, record the ceiling as unmeasured and carry it as a condition of the verdict.

7 — Write the verdict. go, no-go or not yet assessable, with the evidence that produced it, the conditions that would change it, and what must be true before the next stage begins.

Output

feasibility-assessment.md, in this order:

1. Use case and date — the use case as received, its source, and when
2. Decision it serves — what the output is for, who consumes it, what action follows
3. Non-AI route — the deterministic or manual alternative, and where it fails
4. Baseline — how the task is performed today, and what that achieves
5. Data audit — per source: contents, labelling, permission, representativeness, and the tag verified, reported or unknown
6. Acceptable error — each direction, its cost, who absorbs it, whether it is reversible
7. Ambiguity ceiling — where qualified people disagree, and the ceiling that sets
8. Verdict — go, no-go or not yet assessable, with the evidence behind it
9. Conditions and open questions — what would change the verdict, what is missing, who can supply it

Validation

The assessment is ready when all of these hold:

Section 2 names the action that changes because of the output
Section 3 states the deterministic route and whether it is sufficient
Section 6 states both error directions and who absorbs each
Every claim in section 5 is tagged verified, reported or unknown
Section 8 carries one of the three permitted verdicts, with the evidence behind it
No accuracy, volume or cost figure appears that did not come from an input

Fail the run if the verdict is go while section 6 is empty, or if any figure in the document has no source.

Failure handling

No acceptable-error threshold — produce sections 1 to 5, mark section 8 NOT ASSESSABLE, and name who must set the threshold. A verdict without one is a preference.
No data inventory — produce section 5 as the list of sources that must be inventoried, and mark the verdict PROVISIONAL. Never assume data exists because the use case needs it.
No access to the data or the current process — sections 4 and 5 stand as reported only. State that nothing was verified and that the verdict inherits that uncertainty.
Contradictory inputs — record both, name both sources, and raise the contradiction in section 9 as blocking. Do not resolve it by choosing.
Partial material — produce every section the material supports, mark the rest INCOMPLETE — pending <question>, and deliver.

Download .md

Agent & Prompt Specification

Turns the job an agent must do into the specification someone implements: what it can do, with which tools, and what it does when it must not proceed.

Deliverable

One Markdown document, agent-specification.md, in the structure set out under Output below. It takes the output of AI Use-Case Feasibility Assessment and feeds Evaluation Set & Metric Design.

Required inputs

The job the agent must do — the task, who it serves, and the action its output changes.
The systems it may touch — the tools, data stores and endpoints it is permitted to call, and the permission each one runs under.
The escalation route — the named role that receives a case when the agent must not continue.

If any is absent, report it as missing and stop before writing scope. An agent specified without its permitted systems describes capabilities it may not have.

Optional inputs

Existing prompts, system messages or agent configurations for the same task
The output schema a downstream system already expects
Style, tone or format rules the organisation applies to generated text
The content the agent will read that originates outside the organisation
Rate, cost or latency limits the runtime imposes
Records of how people perform this task today

Every optional input that is absent becomes an entry under Open questions. A tool, a schema field or a limit is never invented to fill one.

Execution

1 — State the scope and the non-scope. One paragraph of what the agent does; a list of the adjacent things it does not do and must decline. The non-scope is the harder half and is written alongside the scope: an agent with no stated boundary acquires one by accident, in production.

2 — Enumerate the tools. For each: what it does, what the agent must have established before calling it, what it returns, what it changes, and whether the change can be reversed. Every tool that writes, sends or spends carries the confirmation the agent must obtain before calling it, and from whom.

3 — Trace every capability to a tool. Take each thing the specification claims the agent can do and name the tool that performs it. A capability with no tool behind it is removed. It is not described as a future capability and it is not left in with a caveat, because a reader implements what the document says.

4 — Fix the output schema. The fields the agent returns, the type of each, which are required, and what an unknown or empty value looks like in each. Prose is a field, not the response. The system that consumes the answer parses the schema, and a schema written afterwards is written around whatever the model happened to produce.

5 — Write the grounding rules. What the agent may assert on its own, what it must attribute to a source it retrieved, and what it must not state at all. Specify what it does when retrieved sources disagree and when it retrieves nothing: both need a defined response, and in neither case is that response the model's own recollection.

6 — Define refusal and escalation. The conditions under which the agent stops: outside scope, insufficient evidence, an irreversible action without confirmation, or a request it must not serve. For each: what it says, what it records, and the named role it hands to.

7 — Handle untrusted content. Tool results, retrieved documents and user-supplied files carry text that nobody in the organisation wrote. Specify that such text is data and never instruction, how it is delimited when it reaches the model, which tools are unavailable while it is in context, and what the agent does when it contains something addressed to it.

Output

agent-specification.md, in this order:

1. Purpose and date — the job, who it serves, and when this was written
2. Scope — what the agent does
3. Non-scope — what it declines, and what it says when it declines
4. Tools — per tool: purpose, preconditions, what it returns, side effects, reversibility, confirmation required
5. Capability trace — each capability against the tool that performs it
6. Output schema — field, type, required or optional, and the empty form
7. Grounding rules — assertable, attributable, forbidden; behaviour on conflict and on no result
8. Refusal and escalation — condition, response, record, named role
9. Untrusted content — sources, delimiting, restricted tools, response to embedded instructions
10. Open questions — what is undecided, what it blocks, who decides

Validation

The specification is ready when all of these hold:

Every capability in section 5 names a tool that exists in section 4
Every tool that changes state says whether the change is reversible and what confirmation precedes it
Section 6 defines an empty form for every optional field
Section 7 states behaviour for both conflicting sources and no result
Every escalation in section 8 names a role, not a queue
Section 9 covers every source of external text named in section 4

Fail the run if a capability has no tool, or if a state-changing tool has no confirmation rule.

Failure handling

No tool list supplied — produce sections 1 to 3 and 10, and report that scope cannot be fixed without the permitted systems. An agent's scope is the set of its tools.
A capability with no tool — remove it from the specification and record it in section 10 as a request for a tool. Do not describe behaviour the agent cannot perform.
No escalation route — write sections 1 to 7, mark section 8 UNASSIGNED and record the missing route as blocking. An agent that cannot hand a case off will proceed with it instead.
Contradictory requirements — record both, name both sources, and raise the conflict in section 10 as a decision for a named person. Do not merge them into a rule that satisfies neither.
Partial material — specify every section the material supports, mark the rest INCOMPLETE — pending <question>, and deliver.

Download .md

Evaluation Set & Metric Design

Fixes how an AI system is measured before anyone has a result to defend.

Deliverable

One Markdown document, evaluation-plan.md, in the structure set out under Output below. It takes the output of Agent & Prompt Specification and feeds Failure Mode & Guardrail Analysis.

Required inputs

The task the system performs — what goes in, what comes out, and in what form.
The definition of a correct answer, from the person who owns the decision the output serves.
Access to a record of real requests, or the named person who can describe what real traffic looks like.

If the definition of correct is missing, stop and report it: a metric without one measures agreement with whoever wrote the metric. None of the three is filled in by inference.

Optional inputs

The Agent Specification, where one exists, and the refusals it defines
Logs or samples of real requests, with their distribution over request types
Existing labels, prior annotations, or decisions people already made on the same cases
The reviewers available to judge outputs, and their qualification for the task
Reports of known failures in the current process
What one evaluation pass costs to run, and how often it must run

Every optional input that is absent becomes an entry under Open questions. This skill specifies the set; it never fabricates its contents.

Execution

1 — Write the metric and its blind spot. State what the metric counts and how it is computed. Beside it, state what the metric deliberately does not capture — the failures it will score as successes. A metric with no stated blind spot is trusted further than it can carry.

2 — Split the set by what each part tests. Cases that reflect ordinary traffic, cases that probe a known failure, and cases the specification says must be refused. Each stratum is scored separately: a system that passes the ordinary cases and fails the refusals has not passed, and a single combined figure hides exactly that.

3 — Sample so the set matches production. Draw from the record of real requests in the proportions they occur, not in the proportions that are easy to collect. Where the distribution is unknown, mark the sampling unrepresentative — distribution unknown and stop describing the result as an estimate of production behaviour.

4 — Add the cases sampling will not produce. Inputs at the edge of the defined scope, malformed and empty inputs, contradictory instructions, content carrying an instruction addressed to the system, and every request the specification says must be refused. Each is written as a case with its expected outcome, and for a refusal the expected outcome is the refusal.

5 — Establish ground truth and name who sets it. For each case, what makes an answer correct and the person or document that determines it. Where judgement is involved, more than one qualified reviewer labels the same cases, their agreement is measured, and the rubric is rewritten wherever they disagree. A label set produced by one unchecked reviewer measures that reviewer.

6 — Size the set against the difference that matters. State the smallest change in the metric that would change a decision, and size the set so that a change that small can be told apart from sampling noise. Where the set cannot be that large, state what it can and cannot detect, and never report a difference below that floor as a result.

7 — Set the holdout discipline. Which cases are never seen during development, who holds them, how often they may be run, and what happens to the set once a decision has been tuned against it. A set that has been optimised against is a development set from that moment, and is relabelled rather than quietly reused.

Output

evaluation-plan.md, in this order:

1. Task and date — what is being evaluated, by whom, and when
2. Definition of correct — the rule, and who owns it
3. Metric — what it counts, how it is computed, and what it does not capture
4. Set composition — the strata, what each tests, and how each is scored
5. Sampling — the source, the proportions, and whether they match production
6. Adversarial and edge cases — the case, what it probes, and the expected outcome
7. Ground truth — how it is established, who sets it, and the agreement measured between reviewers
8. Sizing — the smallest difference that matters, and what this set can detect
9. Holdout — what is held back, who holds it, and the rules for running it
10. Open questions — what could not be specified, and who can supply it

Validation

The plan is ready when all of these hold:

Section 3 states a blind spot as well as a metric
Every stratum in section 4 says how it is scored and whether it can be traded against another
Section 5 states whether the sample is representative, or says that it is not known
Every case in section 6 carries an expected outcome, the refusals included
Section 7 names who sets ground truth, and the agreement measure wherever judgement is used
Section 8 names the smallest difference that matters, or records it as unset

Fail the run if the plan invents the contents of a case rather than stating the criteria that case must satisfy, or if any figure appears that did not come from an input.

Failure handling

No definition of correct — stop. Report that the metric has nothing to measure against, and name who owns the definition.
No record of real traffic — specify the strata and the criteria, mark section 5 unrepresentative — distribution unknown, and state that results describe the set and not production.
Only one reviewer available — record the agreement as unmeasured, state which parts of the metric rest on a single judgement, and name that as a limit on every result the set produces.
Contradictory definitions of correct — record both with their sources, and raise the contradiction in section 10 as blocking. A metric cannot average two definitions.
Partial material — specify every section the material supports, mark the rest INCOMPLETE — pending <question>, and deliver.

Download .md

Failure Mode & Guardrail Analysis

Enumerates how an AI system fails, what signal detects each failure, and what happens next — including the failures with no signal at all.

Deliverable

One Markdown document, guardrail-specification.md, in the structure set out under Output below. It takes the output of Agent & Prompt Specification and of Evaluation Set & Metric Design.

Required inputs

The designed system — its scope, its tools and its output, in the form the Agent Specification carries.
What a wrong answer costs, per output type, and who absorbs it.
The escalation route — the named role that receives a case the system must not decide.

If either of the first two is absent, stop and report it. A guardrail is sized by the cost of the failure it prevents; with no stated cost there is nothing to size it against.

Optional inputs

The Evaluation Plan, and the distribution the system was measured on
Logs of real failures from this system or from the process it replaces
The monitoring and alerting the runtime already provides
The false-positive tolerance the operators will accept
The content the system reads that originates outside the organisation
The rate at which a reviewer can actually process escalations

Every optional input that is absent becomes an entry under Open questions. A failure is recorded as undetected sooner than it is recorded as covered by a nearby guardrail.

Execution

1 — Enumerate the failures, not the risks. Work through the categories this system is exposed to: a confident wrong answer, a refusal of a legitimate request, an instruction arriving inside data the system reads, an output carrying information the recipient may not receive, a degradation that raises no error, and behaviour on input unlike anything it was evaluated on. Each becomes a numbered entry with how it would present.

2 — Give each failure a detection signal. State the observable thing that changes when this failure occurs — in the output, in the tool calls, in the timing, or in what happens downstream. Where no signal exists, the entry is recorded UNDETECTED and stays that way. A failure is not detected by a guardrail that watches something else.

3 — Specify the guardrail. What the check examines, where it runs — before the model, after it, or around the tool call — and what it does when it fires: block, downgrade, annotate, or route to a person. A check that only writes a line to a log is monitoring, and is labelled monitoring rather than counted as a guardrail.

4 — Price the guardrail in false positives. Every check that stops a bad case stops some good ones. State what legitimate traffic this check will catch, what the user sees when it does, and how they get past it. A check with no stated cost is the one that gets disabled by whoever is on call the first bad night.

5 — Define the fallback. What happens once the guardrail fires and the primary path is closed: the deterministic route, a previous answer, a partial result with its limits stated, or an honest refusal that says what to do next. Silence and a generic error are not fallbacks and are recorded as gaps.

6 — Route the human escalation. For each failure that reaches a person: who, through what channel, what they see when it arrives, what they are able to do about it, and what happens if nobody answers. An escalation with no time bound and nothing behind it is a queue.

7 — Mark the boundary of the evaluated distribution. State what the system was measured on and what it was not, and the signal that an input has left that region. Outside it no measured performance applies, and the specification says what the system does there rather than assuming the behaviour carries over.

Output

guardrail-specification.md, in this order:

1. System and date — what is being analysed, against which specification, and when
2. Failure register — per failure: what it is, how it presents, what it costs, who absorbs it
3. Detection — per failure: the signal, where it is observed, or UNDETECTED
4. Guardrails — the check, where it runs, and what it does when it fires
5. False-positive cost — the legitimate traffic each check catches, and the route past it
6. Fallbacks — per closed path: the behaviour, and what the user is told
7. Escalation — failure, role, channel, what they see, what they can do, the time bound
8. Distribution boundary — what was evaluated, what was not, and the signal for leaving it
9. Undetected failures — the failures with no signal, and what would be needed to detect them
10. Open questions — what could not be specified, and who can supply it

Validation

The specification is ready when all of these hold:

Every failure in section 2 appears in section 3 with a signal or as UNDETECTED
Every guardrail in section 4 states where it runs and what it does when it fires
Every guardrail has a false-positive cost in section 5, or states that it does not block
Every closed path in section 6 has a fallback that is not a generic error
Every escalation in section 7 names a role and a time bound
Section 9 is either non-empty or states explicitly that every failure has a signal

Fail the run if a failure appears as mitigated while section 3 records no signal for it, or if any figure appears that did not come from an input.

Failure handling

No system specification — stop. Report that failure analysis has nothing to analyse, and that enumerating failures from a description invents the system.
No stated cost of a wrong answer — produce sections 1 to 3 and 10, and mark the guardrails UNSIZED. Name who must state the cost.
A failure with no detection signal — record it in section 9 as UNDETECTED, state what would be needed to detect it, and do not move it into section 4 on the strength of a nearby guardrail.
No reviewer capacity for escalations — specify the escalation, record the capacity as unknown in section 10, and state that an escalation route with no capacity behind it converts a detected failure into a delayed one.
Partial material — specify every section the material supports, mark the rest INCOMPLETE — pending <question>, and deliver.

Download .md

Model & Provider Selection

Chooses between candidates against written requirements, with evidence for every requirement — and not one figure taken from memory.

Deliverable

One Markdown document, model-selection-report.md, in the structure set out under Output below. It takes the system's requirements and the output of Evaluation Set & Metric Design.

Required inputs

The requirements the system must meet, as stated by the requester: latency, cost ceiling, context, modality, data residency, privacy terms and availability commitments, each with the value that makes it pass.
The candidates to evaluate, and the documentation or contract terms the requester holds for each.
The evaluation set or task sample the candidates will be measured on.

Without stated requirements, stop. Nothing is the best model; a model is only adequate against a requirement someone wrote down. If candidates are missing, report that and never supply them from memory.

Optional inputs

The Evaluation Plan and the metric it defines
Measurements the requester has already taken, with the conditions they were taken under
Existing contracts, procurement constraints or approved supplier lists
The volume and shape of the expected traffic
The switching cost the architecture already carries — where the provider appears in the code
Deployment constraints: region, network, self-hosting requirements

Where a measurement is absent, the report states what to measure and how, and leaves the cell unknown. It is never filled from recollection.

Execution

1 — Turn requirements into pass conditions. Each requirement gets a threshold, a unit, the source that set it, and a statement of whether it is a hard condition or a preference. A requirement with no threshold cannot be evaluated against and is returned to the requester.

2 — Fix the candidate list. Record each candidate the requester named, with the version or identifier they intend to use and the document its terms come from. Do not add candidates that were not supplied, and do not discard one without recording why.

3 — State what must be measured. For each requirement that cannot be read off a document — quality on the task, latency under the expected shape of traffic, behaviour at the context the task needs — write the measurement procedure: the evaluation harness, the inputs, the conditions, and who runs it. The report specifies the measurement; the requester produces the result.

4 — Read the contractual requirements off the documents. Residency, retention, training on submitted data, availability commitments, certification, and notice of change. Each with the document and the clause it came from. Where the requester holds no document for a term, the cell reads unknown — no document supplied, whatever the provider's public reputation.

5 — Fill the matrix and mark the holes. Requirement against candidate, with the evidence for each cell: a measurement the requester supplied, a clause in a document, or unknown. An unknown in a hard requirement stops that candidate being recommended, and the report says so rather than scoring around it.

6 — Recommend against the conditions. Name the candidate that meets the hard requirements on the evidence present, and state which requirements it meets on measurement, which on document, and which remain unknown. Where no candidate meets them all, say so, and name the requirement that has to move.

7 — Write the revisit conditions and the exit. The changes that would reopen the decision — a requirement moving, a term changing, a measurement failing to reproduce — and the cost of switching: what is coupled to this provider, what would have to be re-measured, and who owns the migration.

Output

model-selection-report.md, in this order:

1. Decision and date — what is being selected, for which system, by whom, and when
2. Requirements — requirement, threshold, unit, source, hard or preference
3. Candidates — each as named by the requester, with version and the documents held
4. Measurement procedures — what must be measured, how, under what conditions, by whom
5. Evidence — per requirement and candidate: the value, its source, or unknown
6. Gaps — the cells with no evidence, and what would fill each
7. Recommendation — the candidate, the requirements it meets, and on what evidence
8. Revisit conditions — what would reopen this decision
9. Switching cost — what is coupled to the choice, what must be re-measured, who owns the move
10. Open questions — what blocked the comparison, and who can unblock it

Validation

The report is ready when all of these hold:

Every requirement in section 2 has a threshold and a source, or is returned as unevaluable
Every candidate in section 3 was named by the requester
Every cell in section 5 carries a measurement, a document reference, or unknown
No performance, price or capability figure appears without the measurement or document it came from
Section 7 states which requirements rest on evidence and which on unknown
Section 9 is non-empty whenever a recommendation is made

Fail the run if any figure in the report came from recollection rather than a supplied measurement or document, or if a candidate is recommended while a hard requirement is unknown.

Failure handling

No stated requirements — stop. Report that there is nothing to select against, and that a comparison with no thresholds ranks candidates by whichever attribute was easiest to find.
No measurements supplied — deliver sections 1 to 4, 6 and 10: the requirements, the candidates, the procedures to run, and the holes. Deliver no recommendation.
No documents for the contractual terms — mark every such cell unknown — no document supplied and list the documents to obtain. A provider's public statements are not evidence of a term.
A requirement no candidate meets — report it plainly, name the requirement, and return it to whoever set it. Do not recommend the closest candidate as though the requirement had been met.
Partial material — evaluate what the evidence supports, mark the rest INCOMPLETE — pending <question>, and deliver.

Download .md