AI Architecture · LeverageAI · Failure Telemetry

The “This Answer Is Wrong” Button: Flight Recorders for Organisational AI

A one-click dispute on a wiki-backed answer auto-captures a flight recorder — question, answer, receipts, walk, page versions as-of — and decomposes “the AI was wrong” into four addressable classes with different owners and fixes. Thumbs-down is a mood. A walk-backed ticket is a repro.

Scott Farrell · LeverageAI · An extender of File Back the Walk — the disputed-answer path

Someone asks the internal assistant how car expenses work on novated leases. The answer comes back polished, confident, and aimed at the wrong policy region — general car allowances, not the salary-packaging path they actually needed. They can feel it is wrong. They may even click through a citation and see a page that doesn’t match the question they asked. What happens next, in most deployments, is organisationally useless: a thumbs-down, a free-text rant into a black hole, a rephrase, a Slack ping to a human who already answered this last quarter, or quiet abandonment. The system learns nothing. The same wrong turn is waiting for the next person.

Now imagine a different control on that same answer surface: a button that says, simply, I think this answer is wrong. One click. No form essay. The system already holds everything that makes the moment investigable — the question as typed, the answer as shown, the receipt trail of which wiki pages were offered, the scout’s walk through the map, and the page versions as they stood at that instant. The click does not collect a mood. It materialises a structured defect report, auto-populated at the moment of failure. That is a categorically different object from a thumbs-down. One is a repro. The other is a shrug with telemetry cosmetics.

Thumbs-down is a mood. A ticket is a flight recorder.

Enterprise teams ship feedback widgets because they know, correctly, that silent failure is how trust dies. Research on chatbot service failures shows people do not always quarantine blame to a single interaction; they generalise — one bad AI experience stains the category.¹ McKinsey’s work on AI trust has long recorded the mismatch: large shares of organisations name explainability as a material risk, while a much smaller share actually works the problem.² So the industry builds buttons. The buttons usually capture the wrong artefact.

The deceptively small UI is hiding a categorical shift. Compare what the black-box equivalent collects: a thumbs-down and a transcript. One package is a reproduction case written by the system itself. The other is a mood. Support teams know the difference in their bones: a bug report without steps-to-reproduce is a complaint; the same bug with environment, inputs, and a stack trace is work.

There is a second reason the ticket matters that is easy to miss if you still think of the assistant as an oracle. On a wiki-backed navigator, the model is not the knowledge store. It is reading and walking a map with owners, history, and edges — the stance we have argued elsewhere as Witness, Not Oracle and as receipts replacing pure tenure in What Does the Wiki Say?. When the navigator is wrong, the post-mortem does not terminate at a shrug. Every error has an address — wrong page, wrong page content, wrong synthesis over a right page, or a genuine hole in the map. That is what makes a dispute meaningful rather than a complaint against a black box. The oracle asks for faith. The navigator offers receipts. The button is how the user pulls the receipt into a repair loop without writing a ticket system by hand.

Why the walk makes the ticket actionable (and RAG logs mostly don’t)

RAG systems do keep logs. That is not the differentiator. The differentiator is what the trace is made of.

A typical RAG trace is a handful of embedding queries and a pile of scored chunks. The why of every ranking step is a number with no reasons: a cosine, a fused score, a reranker float. Why did chunk 31 outrank chunk 7? The geometry says so. You can re-run the search. You can even re-index. What you cannot do is disagree with the step at a place a human can name. There is nothing to point at and say: there — that is the wrong turn, and here is why the signage failed. Similarity isn’t an argument. Microsoft’s own GraphRAG writing has been candid that baseline RAG “struggles to connect the dots” when answers need relationships across disparate facts — the multi-hop shape of real organisational questions.³ A log of dot products does not become an argument by being stored carefully.

A walk trace is a different medium. Started at the leave hub. Read these edges. Chose “car allowances” over “salary packaging.” Opened page X, abandoned page Y, synthesised from Z. Every step is a place a human can put a finger and say wrong turn. The scout-and-senior pattern that produces that walk is its own story — see The Scout and the Senior — but for this article only one property matters: the trace is written in the same medium as the fix. Diagnosis and repair happen in plain language on the same map. That is the deep reason the ticket has teeth.

A RAG trace is five searches and forty scored chunks. A walk trace is a sequence of legible decisions. Only one of those is something a domain expert can dispute without learning embedding math.

Four failure classes, four owners, four fixes

Run the trace backwards from a bad answer and the failure decomposes. Industry agent-failure work already admits that production breakdowns do not collapse into a single “model wrong” bucket — Microsoft’s agent failure-mode taxonomy is one public attempt to name the plurality.⁴ For wiki-backed answers, the decomposition we care about is operational, not academic. It yields four addressable stages, each with a different owner and a different fix.

That table is the product. The black-box version has one failure class and one non-fix. Here, a synthesis fault is not a map emergency; a coverage gap is not a prompt tweak; a navigation fault is not “the model is dumb.” Organisations already know how to own pages and queues. What they lacked was a failure object that routes into those ownership lines instead of into a generic AI apology macro.

Notice the compounding line under navigation. When you fix a missing edge between leave and salary-packaging, you are not closing one ticket. You are deleting a wrong turn from the future. That is the same instinct as path testing in a broader sense — grade the path, not only the answer — applied at dispute time. Content and coverage fixes compound too: one corrected policy page and one newly ingested SOP change every agent that walks those regions, immediately, without a fleet-wide prompt redeploy. Contest → trace → correct → propagate, in the language of documents organisations have governed for a century.

The ticket schema (what auto-populates)

Here is the minimum artefact the button should create. None of this requires the user to be a good bug reporter. The system was already holding the flight recorder; the click only freezes it and opens a case.

Two details earn their keep. Page versions as-of is what makes the ticket forensic rather than folklore. Organisational questions are often time-bound (“was this compliant when lodged in 2024?”). If the ticket only stores page titles, next week’s edit rewrites the evidence. If it stores versions, the investigation still sees what the scout saw. Classification with owner_route is what prevents the queue from becoming a second black hole: tickets do not sit in “AI team mystery pile”; they land on the desk that can actually change the system.

Clustered tickets are demand-side failure telemetry

One ticket repairs one incident. A month of tickets is a map of pain.

Cluster by hub, by edge, by page, by failure class. The ranking you get is not the janitor’s supply-side guess about what might be stale. It is demand-side failure telemetry — ambiguous hubs people actually hit, missing crosslinks that keep producing navigation faults, stale claims users bother to dispute, coverage holes that surface as honest “we don’t have this” rather than confident hallucination. The users become the wiki’s QA department without knowing it, one annoyed click at a time.

That is the disputed-answer twin of a move already made for successful traffic. In File Back the Walk, every query is treated as a write in disguise: hard answers can be filed back as typed cache, and all walks can be mined for missing edges, dead ends, and cold pages so the map improves from being used, not only from being fed. This article does not re-litigate that passive mining design — the parent owns it. The extension is specific: when the user disputes the answer, you get a labelled failure event with a forced freeze of evidence, not only an unlabelled walk. Passive telemetry says where walkers went. Dispute tickets say where walkers (and readers) were failed. Together they are the full query-improves-system loop: use improves the map; disagreement prioritises the repair queue.

If your organisation only ever ships the parent half — mine successful walks, ignore disputes — you will improve connectivity and still miss the pages users actively reject. If you only ship dispute tickets without walk evidence, you are back to moods. The extender is the disputed path with the flight recorder attached.

Just culture, not blame theatre

Aviation has a name for the organisational property we are trying to buy. A just culture, in Reason’s framing as used in safety practice, is an atmosphere of trust in which people are encouraged — even rewarded — for providing essential safety-related information.⁵ Sidney Dekker’s sharpening is useful here: just culture is forward-looking and change-oriented rather than backward-looking and retributive.⁶

Translate that into organisational AI without the cosplay. When a leave-bot is wrong, the useful move is not “who do we blame — the model vendor, the prompt author, or the employee who clicked?” The useful move is: freeze the recorder, classify the failure, route the fix, verify on a re-walk. Users will only click the button if clicking is cheap and safe. Teams will only investigate if the artefact is rich enough to act on. Leaders will only fund the loop if tickets close into map changes rather than into apology macros.

That is what receipt-plus-walk architecture gives organisational AI for the first time: incidents with flight recorders. The industry’s current default answer to “the AI said something wrong” is a support script and a shrug. Yours can be a ticket that knows which page to blame — or which edge was missing — and a fix that means nobody ever files that ticket again.

RAND’s work on AI project failure is a useful external reminder that these programmes often fail for organisational and governance reasons, not because the underlying models are incapable.⁷ A just-culture dispute loop is governance you can operate daily, not a slide about “human in the loop” that never touches the map.

What this piece is not

Two boundaries, stated so they stay clean.

• Passive walk telemetry — mining every walk for missing edges, dead ends, and cold pages without a user dispute — is owned by File Back the Walk. This article assumes those walks exist and attaches them to disputed answers; it does not re-specify the telemetry pass.
• Receipts UX — how citations are laid out, two-click verification chrome, pretty page previews — matters, and it is a different design problem. Here receipts appear only as fields inside the ticket schema.

Also out of scope: turning the button into a performance-review weapon against staff who dispute answers, or into a vanity CSAT score. The button exists to improve the map. If you use it to score employees, you will train them never to click it, and you will lose the flight recorder on purpose.

Close the loop

Put the pieces in order. A wiki-backed assistant is a navigator, not an oracle. Navigators can be challenged. Challenge is only useful if it produces an investigable artefact. On a walk-backed system, that artefact almost builds itself: question, answer, receipts, walk, versions as-of. Classify into synthesis, navigation, content, or coverage. Route to the owner who can actually change that layer. Cluster the month. Fix the map. Re-walk.

The parent article taught the successful-walk half of query-improves-system. This extender adds the disputed-answer half. Together they mean something most internal AI programmes still cannot say with a straight face: errors make the system better, because every error has an address, and the people who feel the pain can open the case with one click.

The button is small. The shift is not. Stop collecting moods. Start freezing flight recorders.