The Perturbation Review

Your AI reads everything now. The proposal, the SOW variations, the architecture decision records, eighteen months of project chat, the tickets, the meeting transcripts, the final pack. It reconstructs the engagement with a completeness no human post-mortem has ever achieved, and it does it on the Friday the project closes rather than the following March. If you have installed this, you already know it works.

And here is the thing nobody says out loud at the steering committee: the firm has not changed. The wiki is bigger. The next proposal is faster. The delivery patterns are tidier. But if I asked you which commercial decision your firm made differently because of the last four engagements, you would tell me a story rather than name a change.

That gap is not effort and it is not model quality. It is instrumentation. And the reason it persists is embarrassingly simple: post-engagement learning has no customer this quarter. I have made this argument about the underwriting equivalent — the loss history row that decides whether anything compounds — and it holds here without modification: no client asks for it, no partner is measured on it, and it produces nothing this quarter, so it is always the thing that will be started properly next year.¹ Everything else in the review pays out immediately. This one pays out in engagement two, and only if you built it right.

Meanwhile the vendors have arrived at the answer they were always going to arrive at. Microsoft now tells every enterprise it needs to build “Owned Intelligence—institutional know-how that compounds over time, is unique to the firm, and is hard to replicate”, on the grounds that as agents take on more work, “they also generate valuable signals: what worked, what failed, where outcomes drifted”.² Read that second sentence carefully. Every noun in it is a recorded signal. It is a superb description of the harvest, and a complete description of its ceiling.

Push that ceiling forward five years and you get the outcome I keep warning firms about: the world’s best institutional memory for operating yesterday’s consultancy. Perfect recall of a business model that is being repriced underneath you. I have written the terminal-value argument elsewhere and I am not going to re-run it here — the one line worth importing is that declining terminal value and strong harvest returns can coexist, and the failure is not harvesting, it is harvesting while calling it a strategy.³

This piece is about the layer above the harvest. Not a better retrospective. An instrument.

1. The record is not the knowledge, and we have known it for twenty-four years

Before designing anything, look at what happened the last time a serious institution tried to industrialise project learning.

In January 2002 the US General Accounting Office audited NASA’s Lessons Learned Information System. The finding was blunt: “our survey revealed that lessons are not routinely identified, collected, or shared by programs and project managers”.⁴ Forty-three per cent of programme and project managers had not submitted a lesson in two years. Twenty-seven per cent had never heard of the system before the auditors asked.

But the sentence that should stop you is the one about where the learning actually lived:

“Instead, managers identified program reviews and informal discussions with colleagues as their principal sources for lessons learned.”

— GAO-02-195, NASA: Better Mechanisms Needed for Sharing Lessons Learned⁴

The knowledge was moving. It was moving through conversation, and the repository was not a participant. Twenty-four years later we have built a much better repository and the conversation is still where the material is.

Two more findings from the same audit, because they pre-specify two chapters of this method. On volume, one respondent said it was “difficult to weed through all the irrelevant lessons to get to the few ‘jewels’ that you need to find”.⁴ Capture without discrimination produces a haystack and calls it institutional memory. And on candour, a NASA programme manager put the reciprocity problem in one sentence: “People are never rewarded for telling about how they screwed-up and caused a problem/mistake…. This will continue to be a problem until a way is found to allow and encourage people to talk about their mistakes without feeling that they are risking their careers.”⁴

Now the structural reason a modern AI harvest inherits the same ceiling. A 2024 systematic review of two decades of lessons-learned system design found that of all the artefacts researchers had built, “all artefacts were intended to manage text-based explicit knowledge”.⁵ The review is sharper still about the assumption underneath: the omission of tacit knowledge “suggests an underlying assumption that tacit knowledge held by the staff can be easily explicated and converted into text-based formats effectively and accurately”.⁵

That assumption is false, and it has been known to be false since 1966, when Polanyi opened The Tacit Dimension by proposing to reconsider human knowledge “by starting from the fact that we can know more than we can tell”.⁶ The operational form of that, from the medical-education literature, is the one that matters for a project review: experts “are often unaware of the full scope of their knowledge when performing tasks”, and tacit transfer “requires sustained close interaction”.⁷

2. “Unrecorded” is four conditions, not one

Here is the design error that makes most post-project reviews low-yield, and it happens before anyone books a room. We treat “the stuff that wasn’t written down” as a single substance with a single extraction method — ask people. It is four different conditions, and they need four different sensors.

Walk each row, because the whole method falls out of them.

Compiled out. Every finished artefact structurally erases three things: the dated intent in the worker’s own words, the graveyard of considered-and-dropped approaches, and the work that was planned in detail and never built. There is no file for the third one — that is the point, and it is why the repository literally cannot contain it.⁸ Knowledge work adds three more: uncertainty, because publication rewards false closure; friction, because status reports reward green; and inheritance, because a document does not colour-code what survived from last year. The sensor here is not an interview. It is the deliberation record itself, which is why a firm that emits a structured work commit at each substantial session has already had most of this conversation.⁸

Performed but never articulated. Documents record what people say they do; event streams record what they do; the gap between the two is tacit knowledge.⁹ Ask someone to describe the process and you get the procedure. Watch them five times and the steps present in every trace are the procedure — and the variations are either personal style or, far more interestingly, undocumented branches nobody ever wrote down.⁹ No interview reaches this, because the holder cannot narrate it. That is not evasion. It is what tacit means.

Recognised only afterwards. This is the one an interview is genuinely for, and it is a narrow band: changed judgment, political meaning, consequence the participant only understood in hindsight. Notice how little of the total this is. Notice also that this is the only row where the defensible claim is about the participant’s present interpretation, not about what happened.

Not yet knowable. Adoption, persistence, avoided cost, transferability. These are future facts. They must be null at close, and a review that fills them in anyway is not reporting, it is forecasting with the confidence of a receipt.

3. Freeze the account before anyone speaks

Now the move. Everything else is downstream of it.

Before you interview a single person, the machine produces a trace-only account and you freeze it. Not a summary. A reconstruction, from the record alone, covering: the intended outcome and initial assumptions; the intervention actually delivered; deviations, exceptions and contested decisions; evidence of adoption and consequence; unresolved questions; candidate explanations; and — the field that does the work — what the documents and traces cannot establish.

From that last field you compile the negative-space map: decisions whose rationale is thin; moments where the recommendation changed; repeated workarounds or exceptions; disagreements left unresolved; planned work never undertaken; assumptions with no outcome evidence; mismatches between the declared process and observed behaviour; and apparent learning candidates supported by only one source.

The output is not “the lessons”. It is a map of what the project record cannot yet explain.

Then, and only then, you talk to people — and you score the conversation on one thing: what changed relative to the frozen account.

Why the sealing is not bureaucracy

Because the alternative is unmeasurable, and because the human memory you are about to consult has already been rewritten by the outcome.

Baruch Fischhoff established this in 1975 and named it creeping determinism. Outcome knowledge increased the postdicted likelihood of reported events “regardless of the likelihood of the outcome and the truth of the report”, and — the part that ends the argument — “Judges were, however, largely unaware of the effect that outcome knowledge had on their perceptions.”¹¹ When he and Beyth asked people to recall the probabilities they had personally assigned before an event, subjects “remembered having given higher probabilities than they actually had to events believed to have occurred and lower probabilities to events that hadn’t occurred”.¹¹

Read that as a specification. The pre-outcome state of belief does not survive in anyone’s head. If it exists anywhere, it exists in the record — the dated intent, the option that was killed, the estimate before it was revised. The frozen account is not paperwork. It is the only surviving witness to what the project believed before it knew how it ended.

I have argued the engineering version of this before without noticing it was the same problem. When you replay a system against its own history to test whether a policy would have caught something, the rule that makes it science rather than cosplay is the Future-Leakage Rule: do not replay the final period against a corpus that already knows what happened during that period — otherwise you are not testing discovery, you are testing whether the system can read a spoiler.⁴⁶ Future leakage is the silent invalidation mode for historical replay: name it, forbid it.⁴⁶

A post-project interview is a replay, and outcome knowledge is the spoiler. The transposition is exact: do not interview a participant against an account that already knows how it turned out. You cannot un-know it for them — Fischhoff settled that — so instead you freeze what was knowable beforehand and measure everything against it.

Fischhoff also supplies the sentence that should be printed on the cover of every lessons-learned programme:

“the very outcome knowledge which gives us the feeling that we understand what the past was all about may prevent us from learning anything from it.”

— Fischhoff, Hindsight ≠ Foresight, 1975¹¹

The frozen account is a preregistration

The cleanest way to understand what you are doing is to borrow from science. Preregistration exists because “the distinction between postdiction and prediction is appreciated conceptually but is not respected in practice”.¹² A frozen trace-only account converts the interview from postdiction into a test of a stated prior. That is the whole mechanism, and the vocabulary maps one to one: postdiction and prediction become the frozen account and the delta.

And the payoff is measurable in the domain that pioneered it. Comparing the standard psychology literature with Registered Reports — where peer review and the decision to publish happen before results are known — researchers found 96% positive results in standard reports but only 44% in Registered Reports.¹³

4. Questions that stress an account, rather than harvest a lesson

A conventional retrospective asks for a shared story: what went well, what went badly, what should we do differently. That format rewards tidy hindsight and organisationally safe consensus. The perturbation review does the opposite — it preserves incompatible views long enough to learn from them.

So the frozen account is not context you send in advance. It is the stimulus. You put the machine’s reconstruction in front of someone and ask them to break it.

1. What is wrong in this reconstructed account?
2. Where did actual behaviour depart from the stated process?
3. What changed because of the intervention — and what would probably have happened anyway?
4. Which apparent success depended on an exceptional person or a local condition?
5. What failed despite the deliverable being accepted?
6. What judgment still could not be compiled, and why?
7. What appears general here but would fail in another environment?
8. Who sees a different edge of this outcome?
9. What next observation would kill our preferred explanation?

That is not oral history. It is causal-account stress testing.

Alongside those, a second set that reconstructs episodes rather than inviting general lessons. The difference is not stylistic. “What did we learn?” asks a person to perform synthesis under hindsight, which is precisely the operation Fischhoff showed is corrupted. “At what exact moment did you stop believing the original plan?” asks for an episode with a timestamp, which you can then check against the record.

• At what exact moment did you stop believing the original plan?
• What cue did you notice that someone reading the final pack would miss?
• Which option looked attractive, and why was it killed?
• What did people actually do when the designed process failed?
• What became possible only after the project?
• What now looks causal but might merely have accompanied the outcome?
• What evidence would make you reverse this lesson?

The one question that manufactures new evidence

Most review questions retrieve. One class of question creates evidence that was never in anyone’s head, because the question had not existed before. My favourite, asked of the client sponsor:

“What part of this engagement would you now refuse to pay another consultancy to do, because you’ve seen how it can be done?”

Nothing in the project record contains that answer. It is not a delivery question — it is an industry-economics question wearing a delivery costume, and it crosses an altitude in a single sentence. Two more in the same family: what did we do that you thought you were purchasing, and what did we do that you didn’t realise you needed until you saw it? And to the delivery lead: which piece of work felt like delivery but was really compensation for a bad commercial boundary? Nothing in Jira says that.

The instrument is not neutral, and you must design for that

Loftus and Palmer showed in 1974 that the verb inside the question rewrites the memory. Asking how fast the cars were going when they smashed into each other produced higher speed estimates than collided, bumped, contacted or hit — a nine mile-per-hour spread from a single word. A week later, the subjects who got smashed were more likely to say yes to “did you see any broken glass?” — and there was no broken glass in the film.¹⁴

This is not a curiosity. It is the mechanism of the self-confirming flywheel, demonstrated experimentally fifty years before anyone built a canon dense enough to supply the verb. Your frameworks are the verb. Which is why the record keeps observed fact, exact phrase and marked inference in separate fields — memory launders inference into fact, and the discipline is separation, not completeness theatre.¹⁵ I am not going to re-teach that capture object here; it is published, and this review imports it unchanged.¹⁵

5. Four positions, never averaged

“Interview the client, the users, delivery and management” is directionally right and structurally wrong. It sorts people by org chart. Sort them by what they were in a position to know instead:

The fourth position is the one firms skip, and it is where the transferable material concentrates. “Management” reproduces the authorised narrative. A downstream recipient, a risk owner, a team that inherited a new manual step, a non-adopter — those people saw the externality. Safety investigation has been fighting this exact battle for decades, and the Australian Transport Safety Bureau names both the payoff and the resistance in the same passage: “the most effective findings for safety enhancement are often the most difficult to justify”, and “ATSB often encounters attitudes such as ‘why is the ATSB looking at that, it had nothing to do with the accident’”.¹⁶ If nobody in your firm ever asks why you are interviewing the team next door, you are not running this method.

Run them separately wherever hierarchy could suppress disagreement. The goal is not consensus. The goal is to give disagreement an address. And it is worth knowing that the suppression is measurable rather than folkloric: a meta-analysis of 89 studies, 124 samples and 303 effect estimates on sensitive-question techniques found that changing the structure in which a question is asked significantly improves response validity — and that the effect is heterogeneous, meaning you cannot correct for it with a constant factor.¹⁷ You have to change the room.

Hunt the off-diagonal

Once you have four accounts, do not average them. Compare formal importance against lived attention and go looking for the cells where they disagree — the disagreement between the two rankings is diagnostically more valuable than either alone.¹⁸ Applied to a completed engagement:

One discipline governs all four cells, and it is the same one that governs weak-signal sensing generally: chatter should be a prior, never a verdict; the signal must perturb, not command.¹⁹ High friction does not equal a real problem. Consensus does not equal correctness. A cell is a nomination for investigation, not a finding.

6. Some of the most important fields must be null

Adoption. Persistence. Avoided cost. Transferability. Whether the operating-model change actually removed the bottleneck. These are future facts, and at project close the honest value is null.

The cross-domain model here is accident investigation, and it is worth being precise about why. A flight recorder preserves sequence and system state. Witnesses contribute perception and intent. Later physical evidence tests both. None of the three is an oracle. That is exactly the relationship between your trace-only account, your four interviews, and the outcome you have not observed yet.

The investigative bodies are explicit about how they weight the human class. The NTSB investigator’s manual instructs that witness interviews be obtained as soon as feasible because “long delays between the witnesses’ observations and the interviews increase inaccuracies in their statements”, and that witnesses “should be urged to relate only their personal observations and refrain from passing on information they have derived from some other source”.²⁰ Both rules transfer directly. Run the interviews close to close. And when someone tells you what the client thought, that is hearsay in your record, not evidence.

The engineering version of the null field is a design I have argued for in attention systems, and it is the field almost nobody implements: a first-class nullable later outcome that a subsequent process fills, by human label or delayed evidence join. The line that justifies it is uncomfortable and correct — without backfill, review is only aesthetics; with it, learning becomes a supervised problem.²¹

Which produces one concrete change to how an engagement ends:

Close-out should schedule evidence events, not merely finish documentation.

“Project completed” and “outcome closed” are different states. Put the second date in the calendar while the first one is still being celebrated, because the measurement window closes at project close and does not reopen. And note the environmental squeeze: CEO planning effort has compressed — half of it now sits inside a horizon of less than a year, up from 43% the year before²² — while the outcome clock on your engagement is exactly as long as it always was. That widening gap is the thing outcome backfill exists to close.

7. Type the delta, then let the claim climb — slowly

Every difference between what the conversation produced and what the frozen account already contained gets a type. Six of them, and the list is deliberately unflattering:

Then the ladder. “Captured” is not a terminal state; a claim has an altitude, and the only real question is whether it has earned the right to climb.

1. Episode evidence — what happened, was said, or was observed in this engagement.
2. Client operating prior — a client-approved interpretation that should alter the client’s next decision.
3. Transfer hypothesis — a de-identified, rights-cleared possibility that the pattern recurs elsewhere.
4. Field pattern — seen across suitable cases, or strong enough to justify an explicit test.
5. Firm doctrine — a transferable distinction that has survived challenge, changed future action, and earned human admission.

Most material should stop at levels one or two. That is not failed compounding. It is correct discrimination.

To see why the ladder is not bureaucracy, walk one observation up it. A data reconciliation took much longer than expected. At practice level that says: we need a better reconciliation harness. At commercial level: clients shouldn’t be paying implementation rates while we discover their data estate. At offer level: perhaps certainty about the estate is itself the product. At industry level: maybe much of “consulting delivery” is monetised uncertainty created by an inability to cheaply inspect reality. At doctrine level: cheap cognition may move professional services from producing answers to bounding and proving decisions.

Same project. Six altitudes. A competent harvesting agent will find the first two increasingly well and will keep getting better at them. The scarce capability is recognising when an observation deserves to climb — and, far more often, when it does not. As I put it when I first stumbled on this: the altitude is what lets you see what’s new, and what’s trivial. The second half is the harder half.

Evidence ceilings, borrowed from people who publish theirs

An unusual n=1 can move fast to a transfer hypothesis when its consequence is large. It still carries its evidence ceiling. “This may expose a general mechanism” is a different claim from “this is how firms work”, and a dense canon is exactly the thing that makes the second one feel earned.¹⁵

Safety investigation solved this by splitting two ideas most firms never separate: the standard of proof (how sure you must be) from the standard of evidence — “the quantity or quality of evidence required before a decision maker can be satisfied that the relevant standard of proof has been met”.¹⁶ The ATSB then does something almost nobody in professional services does: it publishes the threshold. The term “probably”, it says, means “more than 66 per cent likelihood”, and evidence burden scales with consequence — “the more significant the issue to be determined… the clearer and more persuasive the evidence needed”.¹⁶ It also notes, drily, that “most organisations that conduct safety investigations do not clearly specify the standard of proof (or standard of evidence) they use”.¹⁶

Neither does your firm. Write yours down before the next review, not during the argument about a particular finding.

Two more constraints, both imported rather than invented. Nomination is not publication, and publication is not automatic — parking a recurrence is a success mode, and contested facts should stay contested rather than be flattened into a false consensus.²⁴ And a durable write is justified only when understanding changes in a way that should affect future judgment; if the author cannot state what understanding changed, the write belongs on a case trajectory or nowhere.²⁵

8. Three artefacts, kept apart

A review that produces one document has already collapsed the architecture. Three outputs, three owners, three different burdens of proof.

1. The Outcome Closure Receipt — client-owned, and the thing they actually paid for

The schema is not new and I am not going to rebuild it: proposal, execution, observation, learning, kept as separate blocks.²⁶ What matters here is what the review has to put in it, which is mostly things a normal close-out report is designed to remove:

• contradictions, rather than a smoothed retrospective;
• unresolved attribution, rather than invented causality;
• remaining operating risks;
• the decisions and the help the client still needs;
• a later observation date wherever the outcome cannot yet be known.

Without those fields you get folklore — “that vendor is difficult”, “that playbook is outdated” — none of it addressable, none of it replayable, none of it safe to promote.²⁶ With them you have something the client can take to its own board without you in the room, which is the actual test.

2. The Field-Pattern Nomination — the firm’s separate, rights-cleared record

Anonymised operating context; the local intervention and the value the client received now, not after productisation; the domain prior and the process prior learned; the exception and failure shape; the repeated invariant versus the local variation; evidence pointers including an explicit confession of what was not verified; confidentiality and reuse status; the transfer test; and a named promotion owner with a decision, a date and a revisit trigger.²⁷

If the only artefact is “we built a connector for Client X”, you have a services receipt, not product discovery.²⁷ And the failure mode on the other side is just as expensive: promoting one client’s workflow into the shared platform after a single deployment is not discovery, it is importing one client’s interface politics into everyone else’s operating system.²³

3. The Machinery Change Receipt — the one that decides whether any of this was real

“Feed it back into the machinery” is where most learning programmes go to die, because the sentence has no object. Give it seven fields. For every mutation record: what changed; which evidence caused it; which prior it replaces or qualifies; the applicability conditions; the owner; the next transfer test; and the rollback or demotion condition.

A legitimate change is concrete — a new standing question, an altered offer-eligibility rule, a new acceptance test, an exception taxonomy, a delivery scaffold, a risk threshold, a reusable evaluation, a software primitive, or a documented rejection rule.

The measure is not whether IP was written. It is whether future behaviour changed.

9. The dangerous version: a canon marking its own homework

Now the part I have to argue against myself, because it is the strongest case against everything above and it is not hypothetical.

The failure chain runs like this. I arrive with a dense canon. That canon determines which anomalies look interesting and which questions I ask. Participants reconstruct events already knowing the outcome. The AI finds several of my frameworks that explain the resulting story. I recognise the fit and approve promotion. The new “learning” enters the same canon that shaped its discovery.

The loop then compounds its own priors rather than reality. And here is the vicious part: greater corpus density and better synthesis make the error more coherent, not less. That is not compounding judgment. It is compounding confidence.

The comfortable answer is a human gate. The comfortable answer is wrong. If the same taste proposes the frame, interprets the evidence and admits the result, then “human approved” can only mean “the founder remained persuaded”. And before anyone reaches for experience as a control, the 2025 study of patient-safety incident investigators is worth sitting with: worsening outcomes increased judgements of staff responsibility and produced more punitive recommendations from the same scenario, and while those with safety expertise showed the effect “to a lesser extent”, they still showed it.²⁸

Epistemic separation of powers

The remedy is structural, because personal virtues fail under pressure — especially the pressure of an idea you are already in love with.²⁹ Seven passes, seven artefacts:

1. The harvesting pass reconstructs, and is not permitted to promote.
2. The interviewer records testimony, and does not silently translate it into doctrine.
3. A challenger generates rival explanations — and its list must include “client-local”, “outcome bias”, “no reusable learning” and “this contradicts existing doctrine”.
4. The client confirms, conditions or rejects claims about its own world.
5. A rights and transferability review decides what may leave that world.
6. Someone other than the interviewer authorises promotion.
7. Later outcomes retain the power to supersede all of it.

A two-person practice will have one person walking several of those passes. That is fine — but the passes and the artefacts must still be separated, and the system must be able to show where observation ended, interpretation began, the client spoke, the challenge occurred and promotion was authorised. This is the same structural instinct as separating epistemic access from causal authority in AI systems generally,³⁰ and it inherits the same warning from the audit side: the machinery that compressed the ambiguity cannot independently certify that no important ambiguity was lost.³¹

Four dispositions, and the one almost nobody has

Every claim about the canon gets one of four dispositions: confirms, qualifies (survives only under narrower conditions), contradicts (the existing pattern must be challenged or demoted), and alien — the observation that fits no current frame and must not be discarded merely because it is difficult to classify.

Alien is the disposition that breaks the flywheel, and it needs a budget rather than good intentions. Reserve a fixed allocation for high-significance observations with zero intersection with your existing map, and then measure how often those later become canon — if the answer is never, the lane is wrong, not the idea of the lane.³² Pair it with a scheduled suppression audit: sample the candidates the system chose not to surface and score the misses, because learning only from what you kept is how echo chambers become code.³² And never accept a fluent paragraph as causal proof — if you cannot re-run the judgment from stored inputs, you do not have a judgment history, you have a story.³²

The reason a dense canon needs all three is that the density is simultaneously the asset and the hazard: diffing new evidence against your accumulated map is the power and the trap, because the map can become a suppression shield around unfamiliar fields.³³

The kill path has to win sometimes

Somewhere in the promotion process there must be a live, first-class option to promote nothing — and it cannot be a token seat. If the kill-path never wins, it isn’t a guard; it’s set dressing.²⁹ The test you can run on yourself this afternoon is the one I use: when did your review process last kill a lesson you liked? If you cannot remember, you have not been running an evaluation. You have been running a ratification ceremony, and the confident yes it produces is worth exactly what a yes is worth from something that was never able to say no.²⁹

What survives when you give up the flattering claim that human origination is irreplaceable is narrower and much more durable. A panel cannot supply the criterion it is being scored against, and it cannot carry the risk — “It’s not a gavel at the end. It’s a star overhead and a name on the line.”³⁴ Owning the evaluation function and the consequence boundary is the job. Everything between those two poles is delegable, and getting more so.

10. Reciprocity is a production constraint, not an ethic

I want to be precise about why this section exists, because it is usually written as a values statement and it is not one. If the return path fails, the sensor degrades. That is an engineering claim, and the evidence for it is the NASA manager’s sentence about risking careers.⁴ Fear of blame does not make people lie. It makes them stop volunteering, which is worse, because the record still looks complete.

The patient-safety literature says the same thing in the same shape: fear of punishment or blame “often discourages healthcare professionals from reporting errors and near-misses, leading to missed opportunities for improvement”.³⁵ Aviation’s answer was to design for it — a just culture, meaning an atmosphere of trust in which people are encouraged, even rewarded, for providing essential safety-related information.³⁶ Note that the NTSB manual insists the format is an interview, not an interrogation, conducted “on a basis of courtesy, cooperation, and neutrality”.²⁰ That is a design decision about signal quality, not manners.

Fischhoff, again, from 1975, on why hindsight makes this worse: “When second-guessed by a hindsightful observer, his misfortune appears to have been incompetence, folly, or worse.”¹¹ A review that knows the outcome will read every in-flight judgement as a mistake unless it is designed not to.

The rule

Every signal extracted from people must return either help, corrected authority, reduced friction, a better decision, or acknowledged learning. Without that return path, the process is vendor extraction under a collaborative costume.

Six commitments, each checkable by a participant rather than by you:

• the client receives the Outcome Closure Receipt and the resulting actions;
• participants receive a short synthesis showing what their contribution changed;
• client evidence stays client-controlled;
• reusable learning is de-identified and promoted only under agreed rights;
• a refusal to permit reuse does not reduce the client’s closure benefit;
• participant statements never become individual performance evidence.

That last one is contract-grade, not a setting. The equivalent rule in behavioural capture is the sharpest version I have written: the system records procedure shape, never individual performance; it exists to scaffold you, not to rate you; no productivity metrics, ever, contractually. The day a manager asks for time-per-transaction by staff member is the day the product has failed its own contract — and the boundary is not a limitation on the feature, the boundary is the feature.⁹

Consent is not theoretical either. A July 2026 Pollfish survey of 500 US workers commissioned by Kolmogorov Law found that one in three employed Americans say an AI notetaker or transcription bot has been present in their work meetings, while only 34.7% say they were always asked for permission.³⁷ It is a single commissioned panel survey and I would not build a strategy on the exact figures — but as a description of the room you are walking into, it is directionally undeniable. People already suspect they are being recorded and already suspect nobody asked.

The membrane control that makes this implementable

One practical move does more than any policy: where possible, promote a synthetic reproducer of the failure shape rather than the original case. The shared kernel gets a regression test; the client’s evidence never travels. And keep the direction of travel honest — local delivery can be a gravel road, correct for this workflow and this approval role, while promotion is a paved road with an entirely different burden. Automatic upward movement from client exhaust into shared IP is not compounding; it is a breach of the architecture.³⁸

All of which is downstream of one commercial fact, and it is the line I would hold in a negotiation: the paid product is the client’s closure, not your IP harvest. Get that wrong and the pitch becomes “pay me to use your projects to make my IP smarter”, which is both unattractive and, as a sensor design, self-defeating.

The test for whether you have actually handed something over is one I set for engagement close generally, and it is unforgiving: if the client cannot operate the world without your private kernel as a hidden dependency, you did not deliver a world — you delivered a leash.⁴⁸ The same chapter supplies the rule for the gate on the other side, which is the one most firms quietly fail: approve and reject are both success outcomes of a working gate; a queue that only ever approves is not a gate, it is a conveyor.⁴⁸

11. The strongest case against this — including the parts that argue against me

The strongest counter-case is not that interviews produce nothing. It is that trace-first analysis may already produce nearly all the valuable learning, more accurately and more cheaply.

Post-project interviews can introduce recency and hindsight bias; polite agreement with the interviewer’s thesis; attribution of systemic outcomes to the visible project; management narrative dominating operator experience; your existing frameworks forcing alien evidence into familiar categories; attractive abstractions that never change delivery; and client fatigue with unpaid participation. Every one of those is real, and several of them are experimentally demonstrated in the sources above rather than merely plausible.

Now the honest counterweight, which cuts the other way. Properly conducted structured debriefs do work: a meta-analysis of 46 samples covering 2,136 participants found that “organizations can improve individual and team performance by approximately 20% to 25% by using properly conducted debriefs”, with effects holding across teams and individuals, simulated and real settings, medical and non-medical.³⁹

The evidence from my own history, which does not flatter this idea

I have been doing versions of this for a long time, and the record is instructive in the wrong direction. A 2006 project retrospective outline of mine proposed exactly the conventional thing — hear the client’s specifics, ask what went well and what should improve, reset the process. That establishes precedent for review. It does not establish a compounding IP system.

More sharply: in 2003 a client refused further review meetings until the product had reached a point where it added value. And feedback on a portal project around the same period rated “learning as you go” as expected quality rather than a premium selling point, while emphasising the importance of listening to the workers rather than only the boss.

The correction those three records force is the design constraint I hold today:

Clients will not value a conversation because it improves the supplier. They may value a bounded closure product that improves their own operating system.

And one dataset that argues against the whole posture

The ANA and the 4As found that clients without mandatory review periods averaged 8.1-year agency relationships, while those with frequent reviews ran as low as 3.8 years.⁴⁰ Read plainly: removing the periodic falsification test lengthens the relationship. Which is the opposite of what a killable review programme does on purpose.

I am not going to reinterpret that into support. What it measures is relationship survival, not relationship value — two different quantities that are easy to confuse because one of them is easy to count. Running a review that can kill itself is a bet that a relationship designed to be falsifiable is worth more per year than one designed to persist. It might be wrong, and those are the numbers that would show it.

Meanwhile the buyer side is moving the other way. The UK National Audit Office is unambiguous that consultants “should only be used where they represent best value for money and not to replace capability required inside the civil service”, and recommends “ensuring that civil servants learn from consultants while they are working together by building knowledge transfer agreements into contracts”.⁴¹ When one of the largest buyers of advisory services in a G7 economy writes transfer into the paperwork, an instrument whose output is a client-owned closure asset stops being a nice gesture.

12. Run it as a falsification programme, with the kill conditions written first

This is the part that makes the difference between a method and a ritual, and it has to be pre-committed — before the first review, not after the first disappointing one.

The causal chain under test is:

interview → material semantic delta → transferable mechanism → machinery change → better subsequent outcome

Four links. Each can fail separately, and knowing which one failed is most of the value.

The protocol — seven steps, for the next few engagements

1. Freeze the trace-only account before any interview. Timestamp it. Nobody edits it afterwards.
2. Run the role-separated perturbation reviews — four positions, separate rooms.
3. Record only what changed relative to the frozen account. Restatement earns nothing.
4. Classify each delta into the six types.
5. Have someone other than you disposition promotion.
6. Test promoted candidates against an unrelated later engagement.
7. Return later to observe whether the client outcome actually persisted.

What to measure

• material contradictions found only through conversation;
• candidate patterns that survive an unrelated transfer test;
• machinery changes with evidence and version history attached;
• client decisions or operating improvements caused by the closure asset;
• downstream movement in rework, speed, risk or scarce-expert demand;
• participant burden and refusal rate;
• whether routine closure becomes possible without you.

Directions and comparisons across cycles — never fabricated percentages. Shape, not invented magnitudes. And if you cannot measure a row at all, that is itself the finding: the process is not instrumented enough to claim what it is claiming.⁴²

The freeze protocol carries one more borrowed test, and it is the sharpest kill condition in the set. When you freeze a replay you are supposed to record the as-of timestamp of world knowledge, which events are allowed after it, where the decision traces are stored, and what would count as contamination — and then: “If any answer is ‘the model just knew,’ you do not have replay. You have a demo.”⁴⁶ Transposed to this method: if the interviews restate the frozen account, you do not have a sensor. You have a ceremony.

And the review of the deltas should look like a suppression audit rather than a highlights reel. The four-bucket structure I specified for attention systems transposes almost unchanged: what the frozen account already claimed (precision); what it nearly explained — thin rationales, changed recommendations, unresolved disagreements (boundary calibration); what one position confidently classified as fine that another experienced as friction; and the observations with no place in the current frame at all.⁴⁷ The third bucket is the one nobody builds, and it has the best name in the corpus: a case dismissed not because the evidence was weak but because the worldview had no place to hang the significance does not show up as a near miss. It shows up as silence with a clean conscience.⁴⁷ That is exactly what a confident trace-only reconstruction produces.

The vocabulary for what “good” looks like already exists, and it is unforgiving: falsification throughput is the rate at which you eliminate options, each elimination carrying the named evidence that killed it. Not candidates generated. Not initiatives launched. Eliminations, with receipts.⁴³ Applied here: a review that produces findings without eliminations is generating candidates and calling it learning. Activity does not move this clock. Evidence moves it.⁴³

13. One review, compressed

Designed specimen — composite, not an executed client engagement

What follows is a designed walkthrough, fused from patterns rather than drawn from one client. No metric below is a measurement, and none is presented as one. Its job is to show the shape of the outputs, not to prove an effect.

The engagement. A reporting-and-reconciliation programme for a mid-sized enterprise. Delivered, accepted, invoiced, closed. Everyone reasonably pleased.

Stage one — the frozen account. The machine reconstructs the engagement from proposal, SOW variations, decision records, project chat, tickets and the final pack. Frozen and timestamped. The negative-space map that falls out of it has, among others, four entries worth following: a design recommendation that reversed in week six with a thin rationale; a repeated manual reconciliation step appearing in delivery chat but in no document; a scope item planned and silently dropped; and an adoption assumption with no outcome evidence whatsoever.

Stage two — four rooms. Authority, operator, delivery, and one adjacent sceptic: the team downstream who inherited the reports and was never in a project meeting.

Stage three — the delta table. Only what the frozen account did not already contain:

Read the second row again, because it is the one that pays for the review. Everything about that engagement was formally successful. The report shipped, acceptance passed, the invoice cleared. And a downstream team is still doing a weekly manual step that the closure documentation says was automated. That is the delivery confidence, no observable client consequence cell, and no amount of trace analysis would have surfaced it, because the trace ends where the project ends.

Stage four — the alien lane. One observation fits nothing. The operator mentions, almost as an aside, that the team stopped trusting a particular dashboard not because it was wrong but because it had once been right in a way that embarrassed someone. There is no frame in my canon for “accuracy as a political liability”. The temptation is to file it under change management and move on. Instead it goes into the alien lane with its exact phrase preserved, and it stays there until either a second instance arrives or the lane’s scheduled review retires it. Not classifiable is a classification.

Stage five — the three artefacts. The client gets an Outcome Closure Receipt with the contradiction preserved, the unresolved attribution stated as unresolved, and an observation date set six months out. The firm gets one Field-Pattern Nomination — the “agreeing what the numbers mean” hypothesis, de-identified, marked as a transfer hypothesis with an evidence ceiling and a named test. And one Machinery Change Receipt: a new eligibility question added to the qualification checklist, with the evidence, the prior it qualifies, the applicability condition, an owner, and a demotion trigger if it fails on the next two engagements.

Stage six — the backfill that reverses a lesson. Six months later, somebody returns to the observation date. The promoted eligibility question has fired twice. Once it caught a genuine problem. Once it disqualified an engagement that a competitor took, delivered without difficulty, and that showed none of the predicted failure shape. The transfer hypothesis is demoted from field pattern back to transfer hypothesis, and the demotion is recorded with its evidence.

14. Monday

Five things, and none of them requires a budget line.

One. Place your current process on the four-condition table. Which of the four do you actually sample? Most firms sample condition three, badly, and assume it covers the other three. If you cannot name your sensor for “performed but never articulated”, you do not have one.

Two. On the next engagement that closes, freeze the account before anyone talks. It does not have to be sophisticated. A dated reconstruction and an explicit list of what the record cannot explain is enough to start measuring. You cannot retrofit this — the baseline has to exist before the conversation.

Three. Add the fourth chair. Find one person who saw a consequence outside the declared success perimeter and was never in a project meeting. That single interview will tell you more about whether this method is worth running than any amount of internal reflection.

Four. Put a date in the calendar for the outcome you cannot see yet, and put it in the closure document where the client can see it too. Close-out schedules evidence events.

Five. Write your kill conditions down and give them to the client. Six of them are drafted above. Publishing them costs you nothing if the method works, and saves you a year if it does not.

Where this leaves my own role, and yours

Not every project earns this. Most closures should be run by the client’s own people, the delivery team and the machinery. A project earns the intervention when it has some combination of material strategic consequence, an unresolved contradiction, a high likelihood of transferable learning, an unusual failure shape, evidence that formal success and lived operation disagree, or a live question in the strategic portfolio.

Which means the role has to erode on purpose. The honest renewal test is the one I hold myself to: is the organisation getting better at closing known classes of outcome without me, while I keep finding consequential new classes? If engagement ten requires the same routine interview as engagement one, nothing compounded — I just installed another founder dependency with a recurring invoice attached. That is the same discipline as transfer the known, earn renewal at the frontier,⁴² and the same proof standard as everywhere else in this body of work: engagement two is the only proof that matters.⁴⁴ The compounding test is one sentence long — did we leave merely another codebase, or a sharper corpus that makes the next project easier and better?⁴⁵

One adjacency I will name and leave alone, because it belongs to a different argument with a different instrument: whether I personally get more capable while my firm becomes no more transferable is a distinct failure with its own ablation test. Different question, different apparatus — named here so you know I have not confused the two.

Here is where I have landed, after arguing myself out of the flattering version twice. AI harvesting projects, materials and documents is awesome. It does a fantastic job. But it doesn’t seem to go the next step — and the next step is not a smarter model or a bigger wiki. It is a frozen baseline, four sensors instead of one, a claim lifecycle with a ceiling, a separation of powers, and a written-down reason to stop.

AI can harvest what the project recorded. The higher-value act is discovering what the project taught and never recorded. But the version of that sentence I will actually defend is longer and less quotable: AI reconstructs what the engagement made legible; human testimony and behavioural evidence probe what remained tacit, contested or unasked; and later outcomes decide what was genuinely learned.

Every project should leave two things behind: the result the customer bought, and a better organisation than the one that started it. Most firms are now excellent at proving the first. The second one needs an instrument, and the instrument has to be able to tell you when it isn’t working.