Critical Thinking Audit for AI-Assisted Discovery

The programme manager looked at the printed pack and asked a reasonable question. The AI notes already had a process map, a Decisions heading, and a 48-hour refund SLA. Did we still need the two-hour workshop, or could people confirm by email?

Nobody in the room could say who had spoken the 48 hours. The pack looked finished. Elicitation had not started. Confirming by email would have filed a number nobody owned as a workshop outcome.

Workshop Sign-Off Is Not a Completeness Check

Elicitation is drawing information out of people and other sources. It is not the work of producing a tidy document. Professional practice still treats the output of a session as unconfirmed until someone compares it with its source and with other results, and until the people in the room agree that the capture is correct enough to use. If that word unconfirmed is new in this setting, the one-sentence version worth keeping is: structured notes are a draft of what was heard, not a decision.

A generated pack is very good at looking confirmed. It fills every agenda box. It invents a Decisions list. It supplies numbers that sound operational because they are specific. Completeness of coverage is not the same job as a named person taking a position.

The room often confuses the two. People read for whether every process has a box. They ask whether the notes look like last quarter's workshop output. They treat leftover questions as already answered because the headings left no blank space.

Completeness is the tell, not the comfort.

The name for this already exists. People defer to generated content that feels high quality and stop seeking other information — automation bias — which then makes confabulation easier to miss. In discovery work the bias looks ordinary: you stop asking who said the SLA, what exception was never raised, and what Finance would be locked into, because the pack already feels like a workshop that happened.

The fair objection is time. An hour of examinations before sign-off looks like delay when the pack arrived overnight. That objection is right about the hour. Where it stops holding is in what it counts. The hour is visible. A locked-in refund rule nobody owned is a quarter of exception handling and a quietly angry fraud team.

Three Examinations — Source, Silence, Commitment

You do not need a new auditor in the workshop. You need an hour on the pack you are about to treat as elicitation. The examinations are not spec probes. They do not ask how the design fails on a late file. They ask whether discovery happened.

The failure mode is quiet. The workshop gets shortened or skipped because the pack already looks like minutes. Three short examinations beat a longer "human in the loop" reread that only checks tone.

Each examination targets a different silence in a fluent pack.

Source — who actually said this

Compare every load-bearing claim with its origin. Practice already asks you to check elicitation results against their source — a stakeholder utterance, a prior document, a research note — before you commit resources to using them.

On an AI pack the origin is mixed more often than anyone admits. Some lines came from last week's call. Some came from a policy PDF. Some arrived because the model needed a number to finish a sentence.

Mark each Decisions item as said, sourced, or filled. Filled is not a crime. Treating filled as said is.

If you cannot find a speaker or a document for the 48-hour SLA within a minute, the SLA is unconfirmed. It does not belong under Decisions until someone in the room takes it.

Transcript origin helps this examination. It does not finish it. A model given a real transcript can still promote a passing remark into a heading, and it can still invent the number the transcript never contained. Empirical elicitation work keeps finding the same pattern: AI is strongest when it synthesises a discussion that already happened, and weaker when it generates the artefact from a problem statement alone. Source is how you tell which of those you are holding.

Silence — what the pack never asked

Silence is the examination most teams skip because the pack looks full.

Ask which exception, rejection, or "we tried that" never received a question. Models resolve underspecified context by inserting plausible assumptions so the prose stays coherent. Practitioners call the silent version of that move assumption injection: unverified context slips in so the artefact can look complete. If that term is new, keep this version: the pack answered a question the room never asked.

Industry voices on AI for requirements keep reporting the sibling pattern — specific stakeholder needs flattened into generic specifications that ignore organisational constraint. In discovery, silence is how that lands before a spec even exists. A "standard returns process" template has statuses and an SLA. It does not ask damaged versus change-of-mind versus fraud hold, because those distinctions are local and slightly embarrassing.

Write the unasked question in the margin. If nobody in the room can answer it, the workshop is not finished.

It has been formatted.

Commitment — what signing would lock

Approval is not an acknowledgement that a document exists. It is agreement that further work may proceed, given by people who understand and accept the content.

Read the pack as a set of locks. Who would be committed if this left the room as "agreed in workshop"? Which team would have to unwind the number? Which exception would become an incident because it was never owned?

Commitment is where discovery becomes expensive. Source and silence are cheap while the workshop is still a conversation. Once the pack is treated as decided, the same gaps travel into a specification with a date on them.

If a Decisions line has no named acceptor, it is not a decision. It is furniture.

Worked Walkthrough — Retail Returns to Refund

Take one AI-assisted discovery pack as a running example: retail returns flowing into refund in a familiar ERP and payments landscape.

What the pack said, in polished form: a linear process, statuses from received to refunded, a 48-hour SLA, exception codes that looked complete, and a Decisions block that listed the SLA and "auto-refund on receipt of warehouse confirmation."

Source found the filled number. Nobody in operations had said 48 hours. The figure was a round interval the model likes. We struck it from Decisions and wrote "SLA unconfirmed — ops manager to state the clock they actually run."

Silence found the missing split. The pack had one happy path. The floor runs three: change-of-mind (fast refund), damaged (inspect then decide), and fraud hold (Finance freeze, no auto-refund). The template never asked. We put the three paths on the agenda as the first hour of the workshop, not as a footnote.

Commitment found the lock. Auto-refund on warehouse confirmation would have committed Finance to paying out on a fraud-hold case the freeze exists to stop. We removed auto-refund from Decisions and named the fraud-ops owner as the acceptor for any exception that releases money.

None of that required a debate about whether analysts still have a job. It required treating the pack as unconfirmed elicitation.

Teams that get value from AI in analysis tend to invest in context first, then generate — and they still watch for verbose drafts that invent scope. The examinations are how you spend attention in the hour before the room is asked to sign.

After the Examinations — What Signed Off Means

Signed off means the three silences have answers on the page, or an explicit gap owned by a named person in the room until they do. It does not mean every heading is filled, the prose is elegant, or email confirmation closed the workshop because the pack looked like minutes.

Evaluative judgment stays human for the same reason elicitation was never a formatting task. Domain specialists still hold intuition and local constraint that a model cannot replace. The examinations are how that judgment is forced when the pack arrives already fluent.

If you only have time for one pass, run source first. Most workshops that rubber-stamp a tidy AI pack fail on a number nobody said, not on a missing process box.

A signed discovery pack does not need prettier notes. It needs a speaker for every load-bearing claim, a written question for every silence the template skipped, and a named acceptor for every lock the room is about to take.