# Performance Investigation

Finds where the time actually goes, changes one thing, and proves the change with the same measurement that found the problem.

## Deliverable

One Markdown document, `performance-investigation.md`, in the structure set out under **Output** below. It measures against the thresholds set by **Performance Budget Definition**, and returns the evidence that the system is inside them again.

## Required inputs

- **The complaint as it was reported** — what is slow, for whom, doing what, and since when if that is known.
- **A way to reproduce it** — an environment, an account, a data set or a trace that shows the behaviour.
- **Access to measurement** — the profiler, the timing instrumentation or the traces the system exposes.

If any is absent, report it as missing and go no further than the step it blocks. Never proceed to a fix on the strength of a plausible explanation: the plausible cause and the actual cause agree far less often than anyone expects.

## Optional inputs

- The budget the interaction is supposed to meet, and who owns it
- Historical measurements from before the behaviour changed
- The change history covering the period the complaint describes
- The traffic shape and data volumes in production, as against those in the test environment
- The infrastructure the system runs on, and what it shares with other workloads
- Earlier attempts at the same problem, and what each of them measured

Where an optional input is absent, the report states what that leaves unconfirmed. It does not fill the gap with a hypothesis.

## Execution

**1 — Reproduce before changing anything.** Establish the conditions under which the behaviour appears: the input, the data volume, the environment, the concurrency, the cache state. An investigation that cannot reproduce the problem stops at this step and reports what is needed to reproduce it.

**2 — Take a baseline.** Measure the reproduced case as it stands, record the instrument, the conditions and the percentile, and repeat the measurement enough times to show its spread. Without a baseline there is nothing to compare a fix against, and every later claim about the fix is an opinion.

**3 — Measure where the time goes.** Use the profiler, the traces or the timing instrumentation to attribute the interval to its parts, and record the attribution as measured. A component nobody measured is not a suspect; it is unmeasured, and the report says which components those are.

**4 — Find the bottleneck rather than the suspicion.** Follow the measurement to the largest attributable cost and confirm it: remove it, bypass it, or vary its input and watch the measurement move. The bottleneck is the part whose change moves the number, and the proof is the measurement that moved.

**5 — Change one thing.** Make the smallest change that addresses the confirmed bottleneck, and only that. Two changes at once produce a result that cannot be attributed, and a result that cannot be attributed can be neither defended nor reverted with confidence.

**6 — Measure again under the baseline conditions.** The same instrument, the same conditions, the same percentile, the same repetition. Record the new figure beside the old one. If the number did not move, record that it did not, revert the change, and return to step 3 — a change kept because it ought to have helped is technical debt with a good story attached.

**7 — Lock it in.** Add the test or the check that fails if the behaviour returns, at the measurement point the budget names. A fix with no regression test is a fix with an expiry date on it.

## Output

`performance-investigation.md`, in this order:

- **1. Complaint and date** — what was reported, by whom, and when
- **2. Reproduction** — the conditions under which the behaviour appears, and what was needed to produce them
- **3. Baseline** — the measurement before any change: instrument, conditions, percentile, spread
- **4. Attribution** — where the time went, as measured, and what was left unmeasured
- **5. Bottleneck** — what it is, and the measurement that confirmed it
- **6. Change** — what was changed, and why that change addresses the confirmed bottleneck
- **7. Result** — the measurement after the change, under the baseline conditions, beside the baseline
- **8. Regression test** — what now fails if the behaviour returns, and where it runs
- **9. Not established** — the hypotheses that were not confirmed, and what would settle each

## Validation

The report is ready when all of these hold:

- Section 3 records a baseline taken before any change was made
- Section 5 names a measurement, not an inspection, as its confirmation
- Section 6 describes one change; a second change is a second investigation
- Section 7 reports the measurement after under the same conditions as section 3
- Section 8 names a test that fails when the behaviour returns
- Every cause stated in the document traces to a measurement in section 4

Fail the run if a cause is reported without a measurement behind it, or if the measurement after was taken under conditions other than the baseline's.

## Failure handling

- **The problem cannot be reproduced** — stop at section 2, report the conditions that were tried, and name what is needed: an environment, a data set, a trace from an affected user, or instrumentation in production.
- **No profiler or instrumentation available** — report the reproduction, and the baseline if the interaction can be timed end to end. State that attribution is impossible without instrumentation, and name what must be added.
- **The complaint and the measurement disagree** — measure at the point the user experiences, record both results with their conditions, and report the difference. Do not resolve it by choosing the measurement that is easier to take.
- **Several changes shipped together** — report the combined result and state that no single change can be credited. Re-run the investigation one change at a time before recording a cause.
- **Partial access** — measure what is reachable, list the components that could not be observed in section 9, and deliver. An investigation that names what it could not see is usable; one that omits it points at whatever it happened to measure.