Research & Analysis — Skills

Research Plan Design

Turns a question someone wants answered into a plan that can be run — with the analysis fixed before any data is collected.

Deliverable

One Markdown document, research-plan.md, in the structure set out under Output below. It is the input to Source Evaluation & Evidence Synthesis, to User Interview Analysis and to Data Analysis Brief.

Required inputs

The question someone wants answered — as it was asked, with its source and date.
The decision that depends on the answer — what would be done differently depending on how it comes out, and who makes that call.

If either is absent, stop and report it. A question whose answer would change no decision is reported as not worth researching, and the run ends there rather than producing a plan for it.

Optional inputs

The date the answer is needed by, and the budget available
Earlier work on the same question, and why it did not settle it
Access to the population, the records or the systems the method would need
The result the requester expects the research to produce
Ethical review, consent procedures or data protection obligations that apply
The unit of analysis, and any groups the decision distinguishes between

Every optional input that is absent becomes an entry under Open questions, and where it blocks the method it becomes a condition on the plan. None of them is assumed.

Execution

1 — Restate the question so it is answerable. Record the question verbatim with its source and date. Rewrite it so that it names the population, the thing being measured or observed, and the period it refers to. A question that cannot be rewritten this way is not yet a research question; write down what is missing and continue.

2 — Tie the question to the decision. State the decision, who owns it, when it is taken, and the options in front of them. State what each possible answer would change. If no answer changes anything, stop: report the question as not worth researching, and name the decision that would make it worth researching.

3 — Define what would count as an answer. Before choosing a method, write down what evidence would settle the question, and what observation would falsify the finding the requester expects. A finding that no observation could contradict is not a finding, and a plan that cannot produce one is not worth running.

4 — Choose the method and justify it. Name the method, what it can establish and what it cannot. State why it fits the question in step 1 rather than why it is convenient or familiar. A method that cannot produce the evidence named in step 3 is the wrong method, and is replaced rather than qualified.

5 — Define the sample and how it is reached. Who or what is sampled, how many, how they are selected, how they are contacted, and who that route silently excludes. Record the bias the route introduces — every route introduces one. Where the size cannot be set from the supplied material, write unknown — to be set with <name>.

6 — Fix the analysis before collection. State how the material will be analysed, which comparisons will be made and which will not, and how ambiguous cases will be treated. Analysis chosen after the data has been seen is selection wearing the name of analysis.

7 — Record the ethical and consent requirements. What participants are told, what they consent to, what is stored, for how long, who may see it, and what approval must be in place before collection starts. Collection does not begin until this section is complete.

Output

research-plan.md, in this order:

1. Question as received — verbatim, with source and date
2. Answerable question — population, what is measured or observed, period
3. Decision — the decision, its owner, its date, and what each possible answer changes
4. What would count as an answer — the evidence that would settle it, and the observation that would falsify the expected finding
5. Method — the method, what it establishes, what it cannot establish, and why it fits
6. Sample and recruitment — who or what, how many, selection, route of contact, who the route excludes, and the bias it introduces
7. Analysis plan — the comparisons that will be made, those that will not, and the treatment of ambiguous cases
8. Ethics and consent — what participants are told, what is stored, for how long, who sees it, and the approval required
9. Open questions — the question, what it blocks, who can answer it

Validation

The plan is ready when all of these hold:

Section 3 names a decision, its owner, and what each possible answer changes
Section 4 names an observation that would falsify the expected finding
The method in section 5 can produce the evidence section 4 requires
Section 6 names the bias its recruitment route introduces
Section 7 carries the date it was written, and that date precedes collection
No figure appears that was not supplied; unknown is a valid and expected entry

Fail the run if the method cannot produce the evidence section 4 requires, or if the analysis plan was written after collection began.

Failure handling

No question supplied — stop. Report that the plan has no subject and cannot invent one.
No decision depends on the answer — stop and report the question as not worth researching. Name the decision that would make it worth researching, and who would own it.
No access to the population or the records — produce sections 1 to 4, mark section 5 METHOD BLOCKED, and list in section 9 the access required and who can grant it.
Contradictory statements of the question — record both, name both sources, and raise the contradiction in section 9 as blocking. Do not merge them into a third question nobody asked.
Partial material — produce every section the material supports, mark the rest INCOMPLETE — pending <question>, and deliver. A plan that names its gaps can be reviewed; one that filled them cannot.

Download .md

Source Evaluation & Evidence Synthesis

Grades a body of sources on what each one actually observed, and synthesises only what they support.

Deliverable

One Markdown document, evidence-synthesis.md, in the structure set out under Output below. It takes the question written by Research Plan Design and feeds Competitive Landscape Analysis.

Required inputs

The question the sources are meant to answer — section 2 of research-plan.md is the expected form; any question naming a population, a subject and a period will do.
The sources themselves — the documents, records or pages to be graded, each one reachable rather than only cited.

Without both, stop. A source list with no question cannot be graded, because relevance is a property of the pair. Never substitute a source you were not given for one you cannot reach.

Optional inputs

The date each source was published, and the date it was retrieved
The method each source used, where it states one
Who funded or commissioned each source
Sources the requester expects to be dismissed, and why
Earlier syntheses of the same question, and what they concluded
The strength of evidence the decision requires before it can be taken

Where a source states no method, no date or no funder, the absence is recorded as unknown and counts against its grade. It is never filled in from the source's reputation or from what sources of that kind usually do.

Execution

1 — Fix the question and the inclusion rule. Restate the question, then write the rule that decides whether a source is in or out. Apply that rule to every source before reading any of them for content, so that inclusion is never decided by whether a source agrees with the expected finding.

2 — Classify each source by what it observed. Primary if it reports observations its own authors made; secondary if it reports observations someone else made; tertiary if it summarises a secondary account. A source that observed nothing and cites no one is opinion, and is labelled as such rather than quietly dropped.

3 — Trace each claim to its origin. Follow the citations back. Where several sources rest on one original, record them as a single line of evidence with that origin named. Repetition across sources is not corroboration, and counting it as corroboration is the most common failure in synthesis.

4 — Grade each source on what it actually did. Record the method, the population it observed, the period it covers, who funded it, and the distance between what it claims and what its own method could establish. A claim wider than the method that produced it is recorded as overreach against that source.

5 — Record the disagreements. Where sources conflict, state what they conflict about, what evidence each side rests on, and whether the conflict is about observation, about method or about interpretation. Disagreement is a finding; averaging it into a middle position destroys it.

6 — Synthesise only what the body supports. Each statement carries the sources behind it, the lines of evidence it rests on, and the strength of those lines. A claim no source supports does not appear, however plausible it is and however firmly the requester expects it.

7 — Name what the evidence cannot settle. The parts of the question the supplied sources do not reach, and the kind of source that would reach them.

Output

evidence-synthesis.md, in this order:

1. Question and inclusion rule — the question, the rule, and the date it was applied
2. Source register — one line per source: title or identifier, author or publisher, date published, date retrieved, and primary, secondary, tertiary or opinion
3. Grades — per source: method, population observed, period covered, funder, and any overreach found
4. Lines of evidence — the independent origins, and which sources trace back to each one
5. Excluded sources — the source, the rule it failed, and who supplied it
6. Findings — one statement per line, each with the sources behind it and the strength of the evidence
7. Disagreements — what is disputed, what each side rests on, and whether it is observation, method or interpretation
8. What the evidence cannot settle — the part of the question left unreached, and the kind of source that would reach it
9. Open questions — the question, what it blocks, who can answer it

Validation

The synthesis is ready when all of these hold:

Every statement in section 6 cites at least one source listed in section 2
Every source in section 2 carries a class and a date, or unknown where none was stated
Sources that share one origin appear as a single line of evidence in section 4
Section 5 lists every excluded source with the rule it failed
Section 7 is either non-empty or states explicitly that the sources did not conflict
No figure appears in section 6 that is not carried by a cited source

Fail the run if a finding cites no source, or if repeated citations of one origin are counted as separate lines of evidence.

Failure handling

No sources supplied — stop. Report that synthesis has nothing to synthesise, and name the kind of source the question requires.
A source cannot be reached — list it in section 2 as unavailable, exclude it from section 6, and record in section 9 who can supply it. Never summarise a source from its title or from another source's description of it.
Every source traces to one origin — deliver the synthesis with a single line of evidence in section 4 and state plainly that the body of evidence is one observation repeated. Do not present it as agreement.
Sources contradict each other flatly — record both positions in section 7 with their grades, and report the question as unsettled by this body of evidence. Do not resolve it by preferring the better-known source.
Partial material — synthesise what the supplied sources support, mark the rest INCOMPLETE — pending <source>, and deliver. A synthesis that names the sources it lacks is usable; one that closed the gap by inference is not.

Download .md

Competitive Landscape Analysis

Maps what is observably true of each player in a segment — sourced, dated, and with what is claimed kept apart from what was checked.

Deliverable

One Markdown document, landscape-analysis.md, in the structure set out under Output below. It takes the graded sources from Source Evaluation & Evidence Synthesis, and any question it cannot answer from them becomes an input to Research Plan Design.

Required inputs

The segment to map — stated so that a player is either inside it or outside it.
The dimensions that matter to the decision — the few attributes the decision actually turns on, and who owns that decision.

Without both, stop. A landscape drawn with no decision behind it maps everything and settles nothing. Never choose the dimensions to suit the players already in view.

Optional inputs

The players the requester already has in view, and who named them
Material on each player: documentation, published pricing, contracts, recordings of demonstrations
Access to a trial or evaluation version of a player's offering
The requester's own position on the same dimensions
Earlier landscape work on the segment, with its date
The date by which the map must be usable, since a landscape ages

Where no material is supplied for a player, every dimension for that player reads unknown and the player is marked unverified. An absent input never becomes an inference from the player's size, age or reputation.

Execution

1 — Fix the boundary. Write the rule that puts a player inside the segment or outside it, and apply it before any player is examined. Record the players it excluded and the rule each one failed, so that the map can be reviewed rather than simply trusted.

2 — Fix the dimensions. Take them from the decision, not from whatever the available material happens to describe. Each dimension carries a stated test: what would have to be observed for a player to count as having it. A dimension with no test cannot be checked, only asserted.

3 — Separate what is claimed from what is observed. For each player and each dimension, record the claim as the player states it, with the source and the source's date; then record separately what was verified, with the material that verified it. The two are never merged into a single entry.

4 — Record the gaps between claim and observation. Where a claim was tested and not confirmed, say so and say how it was tested. Where a claim could not be tested at all, mark it untested rather than promoting the claim to a finding because nothing contradicted it.

5 — Leave the unknowable unknown. Revenue, headcount, roadmap, internal metrics and customer counts are recorded as unknown unless the player published them, and then only as a dated claim by that player. A landscape whose interior numbers were inferred is worse than one with blanks in it.

6 — Date every entry. Each line carries the date of the material it rests on and the date it was checked. State the period after which the map should be redrawn, and the events that would trigger a redraw sooner.

7 — Write the differences that bear on the decision. Where the players genuinely differ on the dimensions from step 2, and where they do not. A dimension on which every player is alike is reported as not discriminating, which is itself a result the decision can use.

Output

landscape-analysis.md, in this order:

1. Segment and boundary rule — the segment, the rule, and the date it was applied
2. Dimensions — one line each: the dimension, the decision it serves, and the test for having it
3. Players — one line per player: the name as the material carries it, where the name came from, and verified or unverified
4. Claims — one line per player and dimension: the claim as stated, its source, and the source's date
5. Observations — one line per player and dimension: what was verified, the material that verified it, and the date of the check
6. Gaps — claims tested and not confirmed, and claims marked untested, each with how it was tested or why it could not be
7. Unknowns — what was not established for each player, and what material would establish it
8. Differences that bear on the decision — where the players differ, where they do not, and which dimensions did not discriminate
9. Expiry — the date after which this map should be redrawn, and what would trigger a redraw sooner

Validation

The map is ready when all of these hold:

Every player in section 3 was admitted by the rule in section 1
Every dimension in section 2 names the decision it serves and the test for having it
Every entry in sections 4 and 5 carries a source and a date
Nothing appears in section 5 without the material that verified it
Revenue, headcount, roadmap and internal metrics appear only as unknown or as a dated claim by the player
Section 9 carries a date

Fail the run if an entry carries no date, or if a player's claim has been recorded as an observation.

Failure handling

No decision behind the request — stop. Report that the dimensions cannot be chosen without one, and ask who owns the decision the map is for.
No material for a player — list the player in section 3 as unverified, fill every dimension with unknown, and record in section 7 what material would change that. Do not describe a player from memory.
Material that cannot be reached — record the attempt and its date, state what was blocked, and leave the entry unknown. Never substitute a second-hand description of the material for the material itself.
Sources disagree about a player — record both in section 4 with their dates, and leave the corresponding entry in section 5 unresolved until one of them is verified. The more recent source is not automatically the correct one.
Partial coverage — deliver the map for the players the material covers, list the rest in section 3 as unverified, and mark the analysis PARTIAL — <verified> of <total> players verified.

Download .md

User Interview Analysis

Turns interview material into findings that can be traced back to what a named participant actually said.

Deliverable

One Markdown document, interview-findings.md, in the structure set out under Output below. It takes the interviews run under the plan from Research Plan Design and feeds the decision that plan named.

Required inputs

The transcripts or notes — one record per participant, each attributable to the participant it came from.
The question the interviews were run to answer — the question, and the decision that turns on it.

Without both, stop. Material with no question behind it can be coded in any direction, and the direction will be the analyst's. Never code from a recording you were given no transcript or notes for.

Optional inputs

The interview guide, and how closely each interview followed it
How participants were recruited, and who that route excluded
What each participant was told the research was for
The participant characteristics the decision distinguishes between
Earlier rounds of interviews on the same question, with their findings
The consent each participant gave, and what may be quoted

Where the recruitment route is not supplied, the report states that the composition of the sample is unknown. An absent input is never replaced by an assumption about who the participants were or how typical they are.

Execution

1 — Prepare the material. One record per participant, each with an identifier, the date, the format, and whether it is a transcript or a note written afterwards. Transcripts and notes are marked differently throughout, because a note is already an interpretation of what was said.

2 — Code close to the words. Work through each record and apply codes that stay near what the participant said rather than near what the question hoped for. A code that cannot be pointed at a passage is not a code. Record every passage each code was applied to.

3 — Separate reported behaviour from stated intention. What a participant says they did, and when they did it, is evidence about behaviour. What they say they would do is evidence about intention and nothing more. The two never merge into one code, and never into one finding.

4 — Build themes from codes. A theme is a group of codes that recur together across records. For each theme, record the codes it is built from, the participants it appeared in, and the passages that evidence it. A theme that appears in a single participant is labelled as such and kept, not rounded away.

5 — Keep the disconfirming cases. For every theme, record the participants whose accounts contradict it and what they said instead. A theme that was never searched for contradiction has not been tested, and the search is recorded even where it found nothing.

6 — Count participants, never rates. Report how many participants a theme appeared in, out of how many took part. Never convert those counts into a percentage, a proportion or a projection onto a population the sample was not drawn to represent.

7 — State what the sample can and cannot support. Who was interviewed, who the recruitment route reached, who it never reached, and which parts of the question this sample cannot answer at all.

Output

interview-findings.md, in this order:

1. Question and decision — the question, the decision that turns on it, and its owner
2. Participants — one line each: identifier, date, format, transcript or note, and the consent given for quotation
3. Recruitment — how participants were reached, who that route excluded, and what is therefore unknown about the sample
4. Codebook — one line per code: the code, what it covers, and the records it was applied to
5. Themes — per theme: the codes behind it, the participants it appeared in out of those who took part, and the quotes that evidence it
6. Behaviour and intention — what participants reported doing, listed separately from what they said they would do
7. Disconfirming cases — per theme: the participants who contradicted it and what they said instead
8. What this sample cannot answer — the parts of the question the participants and the route cannot reach
9. Open questions — the question, what it blocks, who can answer it

Validation

The report is ready when all of these hold:

Every theme in section 5 names the codes behind it and the participants it appeared in
Every quote is attributed to a participant identifier and appears in that participant's record
No theme is attributed to a participant who did not express it
No count anywhere in the document is expressed as a percentage, a proportion or a projection
Section 7 is present for every theme, including where the search for contradiction found nothing
Reported behaviour and stated intention are never carried on the same line

Fail the run if a theme cites a participant whose record does not support it, or if any count appears as a percentage.

Failure handling

No transcripts or notes supplied — stop. Report that there is nothing to analyse, and name the form the material must take.
A recording with no transcript — list the participant in section 2 as not transcribed, leave them out of the coding, and record in section 9 what is needed. Never code from memory of a recording.
Consent does not cover quotation — record the theme and its participant count, mark the passage not quotable, and describe it without reproducing the participant's words.
Participants contradict one another — carry both accounts, as separate themes or as a theme and its disconfirming case, and report the question as unsettled by this sample. Do not prefer the more articulate participant.
Partial material — analyse the records supplied, list the missing ones in section 2 as pending, and mark the report PARTIAL — <analysed> of <total> participants analysed.

Download .md

Data Analysis Brief

Puts a question to a dataset, and states the limits of the answer at the same length as the answer.

Deliverable

One Markdown document, analysis-brief.md, in the structure set out under Output below. It takes the question and the analysis plan fixed by Research Plan Design.

Required inputs

The question — what is being asked of the data, and the decision that turns on the answer.
The dataset — the data itself, reachable rather than described, with whatever documentation exists for it.

Without both, stop. A dataset with no question attached invites the analyst to find something, and something is always there. Never analyse a dataset from a summary of it.

Optional inputs

The dataset's provenance: who collected it, how, and for what purpose
The period the data covers, and any gaps within that period
A data dictionary, schema or set of field definitions
Known collection changes, outages or instrumentation changes within the period
Earlier analyses of the same data, and what they concluded
The certainty the decision requires before it can be taken

Where provenance is not supplied it is recorded as unknown and every finding is marked conditional on it. A dataset whose origin cannot be established is reported as such rather than analysed as though it could be.

Execution

1 — Put the question in analysable form. Restate it so that it names the population present in the data, the quantity in question, and the period. A question the dataset's fields cannot address is reported as unanswerable by this data, together with the fields that would be needed to address it.

2 — Establish provenance before touching the data. Who collected it, how, for what purpose, over what period, and what changed during that period. Data collected for one purpose carries the shape of that purpose into every later analysis of it, and that shape is easiest to see before the data is.

3 — Profile the dataset as received. Record the count of records, the fields present, the missing values per field, the range of each field the question touches, and the duplicates found — every one of them counted from the data rather than estimated from a sample of it.

4 — Record every cleaning decision and what it removed. Each exclusion, correction and transformation, with the rule applied, the count of records it affected, and the reason. A cleaning step that cannot be justified in one sentence is reverted rather than kept and footnoted.

5 — Run only analysis you can describe. State the method, what it assumes, and whether this data meets those assumptions. Where it does not, change the method or report the question as unanswered by this data. An analysis whose assumptions the data breaks produces a number, not an answer.

6 — Report the finding with its uncertainty. The result, how it was arrived at, how precise it is, and how far it moves when the cleaning decisions in step 4 are made differently. A finding that changes under a defensible alternative cleaning choice is reported as unstable, with both results shown.

7 — Write the limits. What this data cannot answer, what else could produce the same result, and where a relationship in the data is being read as more than a relationship. This section is required rather than optional: a brief delivered without it is incomplete, whatever the finding says.

Output

analysis-brief.md, in this order:

1. Question — in analysable form, with the decision it serves and that decision's owner
2. Dataset and provenance — who collected it, how, for what purpose, the period covered, and unknown where that is what is known
3. Profile as received — records, fields, missing values per field, ranges, duplicates
4. Cleaning — one line per decision: the rule, the records it affected, and the reason
5. Method — the method, its assumptions, and whether this data meets them
6. Finding — the result, its precision, and how it responds to the cleaning decisions in section 4
7. Limits — what this data cannot answer, and what else would produce the same result
8. Causation — where a relationship is being read as a cause, and what would be needed to establish one
9. Open questions — the question, what it blocks, who can answer it

Validation

The brief is ready when all of these hold:

Section 2 names a provenance or states unknown, and section 6 is marked conditional wherever it is unknown
Every cleaning decision in section 4 carries its rule and the records it affected
Section 5 states the method's assumptions and whether this data meets them
Every figure in section 6 was computed from the supplied dataset and from nothing else
Section 7 is non-empty and names at least one question this data cannot answer
No relationship is described as a cause anywhere outside section 8

Fail the run if the brief is delivered without section 7, or if a figure appears that was not computed from the supplied dataset.

Failure handling

No dataset supplied — stop. Report that there is nothing to analyse, and name the fields the question would require.
The dataset cannot be reached — record the attempt and its date, deliver section 1 and section 9 only, and name who can grant access. Never analyse a dataset from a description of it.
The dataset cannot answer the question — where it is too small, too biased, or breaks the method's assumptions, report exactly that and what makes it so, and deliver sections 1 to 4 and 9 only. Do not deliver the number an unsuitable method produced with a caveat beside it.
The dataset contradicts another supplied source — record both, name what each rests on, and leave the question unsettled in section 9. Do not adjust the data until it agrees.
Partial data — analyse the period the data covers, state the gap and what caused it in section 7, and mark the brief PARTIAL — <period> only.

Download .md