Mid-afternoon. Design choices had been made; changes had been applied; the coding agent was A/B testing over existing WikiWalks. Then it said something ordinary and ridiculous: we needed some users to go and see whether the new pattern held. We needed more walk data.

Which was so funny, because the AI had not really noticed that it was the main user of the walks.

So I opened ChatGPT, asked a few real questions, and made it do a few walks through the MCP. Then I went into the other coding window and told it to review those walks — what ChatGPT as a user was using and not using, which code paths it followed, which patterns it preferred. We reorganised how we worked around those user patterns, in a really tight loop, the same afternoon.

The AI was not simulating a user. It was the user.

That sentence is the hinge. You do not need to recruit humans to generate more WikiWalks when the intended consumer of the system is principally an AI navigator. ChatGPT using the MCP was legitimate production-like traffic. Not a persona. Not a synthetic shopper. Not a focus-group puppet. The user class the product was built for, generating authentic behavioural evidence on demand.

The reader question

How do you usability-test and evolve a system whose users are AI agents?

Classical usability answers: recruit humans, watch them struggle, redesign the interface. That answer is correct for human products. It is the wrong ontology for an MCP server, an agent-facing API, or a wiki whose principal navigator is a model. Waiting for human subjects is not only slow — it tests the wrong creature.

After this book you should be able to run a three-mode ladder:

• Inspect recorded walks for design defects;
• Replay the same queries across design variants from a pinned Git state;
• Generate missing evidence by dispatching real AI users —

while grading paths, not just answers.

What this book owns

We mint Reflexive Agent Design: one artefact (a corpus of AI usage traces over a Git-versioned system) and one engine (the intended user is an AI, so representative usage can be generated, inspected and replayed on demand).

What we deliberately do not own:

• The full design-evolution loop, candidate breeding and build boundary — that is Replay-Driven Design Evolution.
• Wiki self-improvement from query exhaust — that is File Back the Walk; we reference it, we do not re-explain it.
• Path Testing’s origin story — applied here, not re-derived from RAG Was Built for Chatbots.
• The product of the signal-case queue and cascade ledger — siblings 143 and 144; we need only their trace shapes.

Later siblings in this run — generative design patterns, the open-source shortcut trap, the moat is the memory, experience is compressed priors — are named in prose only. They are not this book.

Thesis in one breath

When the production user of a system is an AI, usability research inverts: real usage can be generated on demand, every walk is an inspectable behavioural trace in the same medium as the repair, and competing designs can be replayed against identical usage — so the architecture evolves through AI-generated traffic instead of waiting for humans.

The rest of Part I defines the framework and the path-quality target. Part II develops the WikiArmy afternoon as a full flagship case. Part III gives you dual streams, the progressive ladder, the multi-model guard, and an operating protocol you can run this week.

Name the practice so you can demand it. A system is under Reflexive Agent Design when six conditions hold together. Drop any one and you have ordinary logging, ordinary dogfooding, or ordinary offline evals — useful, incomplete.

The six conditions

1. AI is a real user of the system. Not a mock. Not a persona script that pretends to be your human customer. The intended consumer class is already a model.
2. Its interaction path is recorded in an inspectable form. Steps, tools, branches, searches — legible later, not collapsed into a final string.
3. AI can analyse those paths as behavioural evidence. A second instance (or the same class of model) can review the walks the way a usability researcher reviews a session recording.
4. It can generate more representative usage when evidence is thin. Dispatch more real AI users; do not wait for a human recruitment cycle.
5. Competing designs can be replayed against the same usage. Same walks, restored world state, different tool contracts or map shapes.
6. Findings update the system the AI will subsequently use. The loop is closed only when the next walk hits the redesigned affordances.

That is stronger than ordinary dogfooding. Engineers using their own product still leaves you recruiting humans for “real” usability. Here the real user is already non-human. The human job does not disappear; it concentrates where judgment belongs: what a good walk feels like, which surprises become criteria, which designs to kill.

The tight loop

AI uses system
   ↓
system records its walk
   ↓
AI reviews the behavioural trace
   ↓
AI identifies navigation/design defects
   ↓
design changes
   ↓
AI uses the changed system again

Every arrow is operational. If you only store traces and never review them, you have surveillance without learning. If you review but never change the system, you have a book club. If you change the system without shared usage to compare, you have vibes. Reflexivity is the closed circuit.

From map janitor to architecture falsification

We already had language for half of this. File Back the Walk treats the answer and the path as two different artefacts: the answer records what the walk concluded; the path records what the walker actually did, including abandoned branches and untouched regions. Querying improves the map because missing edges, dead ends and cold pages become janitor work rather than discarded exhaust.

Reflexive Agent Design extends that idea from a content maintenance doctrine to a design doctrine:

Use AI-generated traffic to evolve the architecture through repeated observation and replay.

Page-level janitor work remains useful. This book is about falsifying designs — tool contracts, boot profiles, graph shapes, queue decision rules — under authentic agent traffic.

Where this sits relative to its sibling

Reflexive Agent Design
├── Trace-Driven Design Review
└── Replay-Driven Design Evolution

Trace-Driven Design Review is what you can run after this book. Replay-Driven Design Evolution — the full candidate-breeding loop, future-leakage-safe racing, and the build boundary — is the sibling altitude. The artefacts differ: usage traces of a live system versus candidate designs raced on a harness. Explicit fences both ways. We supply the instrumentation RDDE consumes; we do not smuggle RDDE into these chapters.

Imagine an agent returns the correct claim about a source or a framework. Your eval suite marks the run green. In the walk log, the agent never opened the canonical page. It searched too broadly, ignored the graph, and synthesised from a secondary mention plus model prior. The answer was right. The path was a liability.

That is not a pedantic distinction. It is the difference between a system that will keep working when the model changes and a system that is secretly a prior with a search UI painted on top. The distinction is load-bearing for Reflexive Agent Design because every mode of the ladder — inspection, replay, generation — needs a target. If the target is only “final string matches,” you will ship designs that look healthy until the lucky prior stops holding.

The Path Testing 2×2

We already named this matrix in earlier work on agentic wikis. Here we apply it to design evolution rather than re-deriving the concept from first principles.

The third cell is invisible when you only test outputs. It is the cell Reflexive Agent Design is built to make expensive to ignore. Output-only suites celebrate it. Path-aware suites treat it as a failure of architecture dressed up as a win.

Work each cell until the repair is obvious

Good path, right answer

The agent opens the canonical page, follows edges that carry scent, spends searches where the map is thin, and synthesises only after the evidence is in hand. This is the only cell you should call success without apology. File these walks as regression goldens: when a redesign shortens them into thrash, you have regressed even if answers stay green.

Good path, wrong answer

Navigation worked; judgment failed. The agent reached the right neighbourhood and still mis-synthesised, over-weighted a weak source, or missed a contradiction sitting on an opened page. Fixing this with more edges is the wrong instrument. Fix synthesis rules, verification prompts, or how conflicting claims are presented. Confusing this cell with a navigation defect wastes a week rearranging the graph.

Bad path, right answer — the invisible cell

This is the load-bearing failure mode. A system can return the right answer while never opening the canonical page, searching too broadly, relying on model priors, ignoring the graph, or stumbling onto the answer from a secondary mention. That looks successful until the lucky prior stops holding — a model upgrade, a domain shift, a missing rumour the prior was quietly using. Then the “working” design collapses and nobody can point to the walk that warned them.

Inspection questions that surface this cell: Did the agent open the canonical source before synthesising? How many search calls preceded the first useful page? Which edges were available and unused? If you cannot answer those from the log, you are not grading paths yet.

Bad path, wrong answer

Navigation failed and the answer failed. Often the same design defects as the lucky cell — illegible boot profile, missing tool, wrong grain of pages — plus genuinely missing content. Do not treat this as “the model is dumb.” Treat it as the application teaching a bad walk, then failing to recover. Scott’s phrasing from walk review applies here as much as anywhere: the users are doing dumb things because we wrote the application dumb — or because we need to allow for a behaviour the design currently punishes.

What to score on every walk

Path quality is not a single number. Score a small dashboard that forces the 2×2 into view:

• Path length in steps (not wall-clock alone)
• Search calls and redundant reads
• Canonical-source reach: opened before synthesis? (yes/no, with page id)
• Abandoned branches (entered then left without use)
• Time and token cost
• Answer quality (right / wrong / partial)
• Explicit 2×2 cell classification

Without the cell label, teams optimise path length into silence (short walks that still never open the canon) or optimise accuracy into luck (long thrash that eventually guesses right). The label is the honesty device.

The queue translation — same 2×2, different surface

For a news-queue system the equivalent is not merely: did the interesting signal eventually enter the queue? That is the output-only question. The path-aware questions are harder and more useful:

• Correct reasons: Did the decision cite independent corroboration, first-party confirmation, or trajectory — or did it fire because a single noisy aggregator spiked?
• Right case: Did the event join the existing developing signal case, or did it spawn a duplicate case for the same story under a new name?
• Appropriate canonicality: Was the provisional anchor (often a secondary discussion) later promoted when a primary source arrived — without deleting bronze evidence?
• Right time: Did the case enter watch / alert / resolve windows that match urgency, or did everything sit in a FIFO sludge until a human noticed?

Map those questions onto the same four cells:

Output-only success (“it showed up somewhere”) is the queue’s version of the lucky answer. A duplicate case that later gets merged by hand still “worked” if you only ask whether a human eventually saw the topic. A Reddit-first discovery that never promotes the later primary still “worked” if you only ask whether something entered the queue. Those are bad paths with right-looking outputs. Part III instruments both surfaces: WikiWalks for graph legibility, queue-evolution traces for serial attention. The 2×2 is the shared scoring philosophy for both streams.

Industry is moving toward trajectories

Agent evaluation platforms increasingly capture full trajectories: tool selection, arguments, loops, retrieval paths, latency and cost — not only final strings.¹ The recommended engineering loop looks like: observe production, identify failure, add the trace to a versioned dataset, change prompt or tool or workflow, replay, compare, deploy.²

That is necessary infrastructure. It is still incomplete if trajectories only validate a change you already decided. Reflexive Agent Design uses the same raw material to discover what the design should become — including new evaluation criteria that did not exist when the first walks were recorded. Path length, canonical-source reach and queue stability often appear as metrics only after inspection makes them impossible to ignore.

Hope is not path quality. A green final answer is not a cognition receipt. Grade the cell, then fix the surface the cell points at.

Part I gave you the name and the scoring target. Part II is the receipt: one afternoon on the WikiArmy project where the method appeared fully formed before it had a name.

Setup

We were getting AI to A/B test over existing data and existing WikiWalks. Design choices had been made. Changes had been applied. The work felt like ordinary agentic development — until the coding agent asked for users.

Not metaphors. Not “imagine a user would…” Literally: we need some users to see if the new pattern holds; we need more walk data. The request was reasonable inside a classical usability frame. It was absurd inside an agent-native frame, because the coding agent and its cousins were already the primary walkers of the system.

Action

I went into ChatGPT — named here as a public product, with no claims beyond observed behaviour — and asked a few questions. I made it do a few walks through the MCP. Same tools. Same graph. Same boot surface production agents would see.

Then I switched to the coding window and asked for a review of those walks: what ChatGPT as a user was using and not using, which code paths it followed, which patterns it preferred. The review was not a vibe check. It was behavioural evidence in the same text medium as the design itself.

We reorganised how we worked around those user patterns the same afternoon. Not a roadmap. Not a research sprint. A tight loop: generate authentic traffic, read it, change the system the traffic will hit next.

As ChatGPT as a user, we worked out what it was using and not using — and reorganised around those patterns in a really tight loop.

Why the walks already existed

On WikiArmy we had kept session walks to replay because we thought we might ingest them later as a different form of data. We still might. What we found first was more useful: replaying new features over retained walks to see whether we got better walks, better quality, faster walks. The retention decision was half accident, half instrumentation instinct. It is the kind of accident you should industrialise.

What “production-like” means

Production-like does not mean “looks realistic in a slide.” It means:

• the same tool contracts as production agents;
• the same graph state (or a known Git pin of it);
• the same boot profile and affordance order;
• questions that resemble real work, not toy prompts;
• recording every step as if you will redesign from it.

If your agent API is only tested by humans clicking through a UI, you are not testing the product. You are testing a tourist brochure of the product.

Implication for queue-shaped systems

The same afternoon also clarified what to record next. Wiki walks remain useful — and for wiki-native work we already know how to think about them. The scarcer instrumentation for a news radar is the serial nature of the queue: how attention is allocated as events arrive, how cases form and merge, how canonical anchors move. You can keep WikiWalks. You should also record queue decisions as they come, with enough Git context to replay from a known position when you redesign.

That dual-stream instinct becomes Chapter 7’s schemas. The point here is simpler: the WikiArmy loop proved that authentic AI traffic plus same-day review is enough to move architecture, not only content.

You do not need a counterfactual harness to start. Mode 1 of Reflexive Agent Design is pure trace inspection: give an AI a collection of recorded walks and ask the questions a good usability researcher asks while watching people struggle with an interface.

Questions that force design defects into the open

• Why did walkers repeatedly perform broad searches?
• Where did they backtrack?
• Which tool or edge was consistently ignored?
• Which paths were unexpectedly long?
• Where did they reach the right answer for the wrong reasons?
• What behaviour does the system appear to be eliciting?

That last question is the sharp one. Systems train their users. Agent-facing systems train models. If every walk starts with thrashing search, the boot profile or the tool descriptions are teaching thrash. If a critical edge is never followed, the graph is illegible at that junction — or the edge does not exist where the model expects scent.

Concrete defect found by inspection alone

On WikiArmy, ChatGPT as a real MCP user repeatedly reached for broad search and secondary pages while under-using edges and tools that should have been the shortest path to canonical material. The coding-window review made the defect obvious without a full counterfactual race.

The diagnosis arrived in plain language — the same plain language Scott had already heard from walk review more than once:

The users are doing dumb things. We wrote the application dumb. Or we need to allow for that to happen.

Either branch is a design change. If the behaviour is thrash, fix the affordance. If the behaviour is legitimate, stop punishing it. What is not allowed is pretending the walk is the user’s moral failure.

Why the medium is the superpower

Each step of a WikiWalk is a legible decision in the same medium as the repair. A reviewer can point at the exact wrong turn and change the map or the tool contract at that point. You are not reconstructing intent from opaque clicks. You are reading a transcript of cognition against a text substrate you control.

That is why AI was “pretty good at just reviewing the walk” without even replaying it — straight out inferring the way a human usability observer would, except faster and in the same tokens the system already speaks.

When inspection is enough

• The pattern repeats across multiple walks.
• The defect is local: wrong tool description, missing edge, bad boot order, misleading page title.
• You can name the change without needing a controlled A/B under identical history.

Escalate to counterfactual replay when you need discrimination between designs under the same usage, or when inspection patterns are ambiguous. Jumping straight to a full replay harness for every question is process theatre when the walks already name the problem.

Mode 2 begins when inspection cannot discriminate, or when you need fair comparison under controlled restore. The move is simple to state and easy to corrupt: restore a known world state, replay the same queries under alternative designs, and score the paths.

Wiki commit A + tool design A + walks
Wiki commit A + tool design B + same walks
Wiki commit A + tool design C + same walks

Then compare path length, search calls, redundant reads, canonical-source reach, abandoned branches, time and token cost, answer quality, and unsupported lucky answers. This is counterfactual design replay over AI cognition, not only over software outputs.

The replay tuple

A replay begins from a precise tuple:

wiki_commit
+ queue checkpoint
+ event-log offset
+ agent/tool configuration

Restore the wiki commit. Restore the queue checkpoint. Branch the replay. Feed events from offset N. Record decisions and walks. Compare with baseline. You do not re-ingest ninety days for every experiment.

The infrastructure is boring on purpose:

• periodic queue checkpoints;
• append-only incoming event logs;
• Git-versioned wiki state;
• versioned prompts, tools and model identifiers.

With those four, a replay can cover the last two hours, two days or seven days from an exact historical position. The queue remains a materialised serial working state, but it is reconstructable from the event stream and wiki snapshots.

Worked comparison: tool design A versus B

Hold the world fixed. Let the designs differ only where you claim they matter.

Score each walk:

1. Path length (steps)
2. Search-call count
3. Canonical source opened before answer? (yes/no)
4. Final answer correct? (yes/no)
5. 2×2 cell classification (Chapter 3)

What you typically learn (qualitative shape, not invented percentages): Design A often still lands right answers — with high search counts and low canonical reach. Those runs cluster in the bad-path/right-answer cell. Design B tends toward shorter paths, higher canonical reach, and fewer unsupported luckies. The decision is not “B is prettier.” The decision is: ship B’s tool contract, and keep A’s failed walks as regression corpus so the lucky cell cannot quietly return.

Cousins, not substitutes

LangSmith-style platforms turn production traces into versioned evaluation datasets and replay new agent versions over them, comparing trajectories as well as outputs.² Reflective optimisers such as GEPA maintain candidates, capture full execution traces, diagnose failures in language, mutate targeted components, and retain a Pareto frontier rather than collapsing everything to a single scalar reward.³

Those systems prove that readable traces are load-bearing. The harder problem in open design work is that the evaluation function itself is being discovered: while studying walks, you realise that canonical-source reach, path quality or queue stability should have been measured in the first place. The harness co-evolves with the design. That co-evolution is why a pure optimiser is insufficient, and why human judgment about what a good walk feels like remains scarce.

On the radar work that sits beside this book — the signal-case queue and its cascade ledger — we retained two related but distinct replay streams. Product siblings own the queue design itself: case grain, cascade re-observation, influence learning. This chapter owns the trace schemas: what you must record if you want Reflexive Agent Design to falsify either surface under AI use.

Orientation only — not a product re-teach: the wiki holds accumulated understanding; the queue holds unresolved attention. Scott’s instinct on the project was explicit: keep wiki walks if useful, but the scarcer new instrumentation is how we queue and how we understand the queuing. You need both streams because they falsify different claims. Either alone leaves a blind spot that path-quality scoring (Chapter 3) cannot see.

Stream 1 — WikiWalk: is the graph legible?

The WikiWalk stream records how the agent interpreted the graph on a single question or task. It is the natural extension of File Back the Walk’s path telemetry into a design-falsification corpus.

question_or_task:
wiki_commit:
pages_opened:
edges_followed:
searches_run:
branches_abandoned:
canonical_sources_read:
result:

What each field is for. question_or_task anchors the walk so you can replay the same intent later. wiki_commit pins the map version — without it, path comparison across days is confounded by content drift. pages_opened and edges_followed are the route. searches_run catches thrash: high search counts with low edge use usually mean the boot profile or tool descriptions are teaching search-first habits. branches_abandoned is dead-end telemetry: places the agent entered and left without use. canonical_sources_read is the honesty field for the bad-path/right-answer cell — if this is empty and the result is still correct, you are looking at luck. result is the answer artefact, graded separately from the path.

Use this stream to improve semantic navigation and interpretation: tool affordances, map structure, scent trails, boot orientation. A thrashing WikiWalk is usually a graph or tool-contract defect, not a “smarter model” defect. Inspection (Chapter 5) reads these fields in language; counterfactual replay (Chapter 6) holds wiki_commit and the question set fixed while you change tool design.

Stream 2 — Queue evolution: does serial attention behave?

The queue-evolution stream records the serial sensing history — decisions about cases over time, not only navigation of pages. This is the stream Scott flagged as more interesting for the radar once the wiki was “good enough.”

event_offset: 1842
observed_at: ...
source_object: reddit.post.123
wiki_commit_before: abc123
queue_state_before: checkpoint-47

decision:
  action: create_case
  case_id: signal.agent_wikis
  reason: new_relevant_development

wiki_mutation_commit: def456
queue_state_after: checkpoint-48
next_review_at: ...

Why this shape. event_offset is the position in an append-only log — the handle that lets you restart a replay without re-ingesting history. source_object is bronze: what was observed, not what you concluded. wiki_commit_before and queue_state_before capture world state at decision time. The decision block is the path: action, case identity, reason. wiki_mutation_commit and queue_state_after close the transition. next_review_at records the attention schedule the design chose — so you can later ask whether re-observation was too eager, too late, or never.

Later events in the same case lineage might say:

action: promote_canonical_anchor
from: reddit.post.123
to: x.post.karpathy.456

or:

action: merge_case
losing_case: signal.karpathy_reddit_discussion
surviving_case: signal.agent_wikis

Those later actions are where the queue 2×2 becomes concrete. A create_case that later needs merge_case may have been wrong grain. A create that never promotes a better primary when one arrives is a canonicality defect with a green “something entered the queue” score. A promote without retaining bronze as propagation evidence is a different design failure. The schema does not invent the product rules; it makes the decisions inspectable and replayable so the product rules can be falsified.

What each stream falsifies

Running only WikiWalks on a radar product is like usability-testing the map while ignoring the attention scheduler. Running only queue traces is like testing the scheduler while agents thrash the graph underneath. Replay-Driven Design Evolution races full design candidates on these streams; this book defines what to record so those races have ground truth rather than vibes.

One tuple binds both streams to history

Both streams attach to the Git-pinned replay tuple from Chapter 6:

wiki_commit
+ queue checkpoint
+ event-log offset
+ agent/tool configuration

Restore wiki commit, restore queue checkpoint, branch the replay, feed events from offset N, record WikiWalks and queue decisions, compare with baseline. Periodic checkpoints, append-only event logs, Git-versioned wiki state, and versioned prompts/tools/model identifiers are enough to cover the last two hours, two days or seven days without re-ingesting ninety days of bronze. The queue remains a materialised serial working state, reconstructable from the event stream and wiki snapshots. Without the tuple, dual streams are two folders of anecdotes. With it, they are a laboratory.

You do not need to replay everything continuously. Continuous full replay of every experiment is not maturity — it is often avoidance of reading the traces you already have. Reflexive Agent Design climbs a progressive ladder that keeps both passive and active learning in play, with escalation rules that protect attention.

Rung 1 — Inspection (cheapest)

Review existing walks
→ infer defects
→ change design

This is Mode 1 from Chapter 5: no restore, no A/B harness, no new traffic. You already have WikiWalks and, if instrumented, queue-evolution lines. Give them to an analyst model with the inspection questions — broad search thrash, backtracks, ignored tools, long paths, right answer for wrong reasons, elicited behaviour. Force a named defect. Change the affordance. Let natural next traffic exercise the fix.

Stay on this rung when the pattern repeats across walks, the defect is local (tool description, edge, boot order, label), and you can name the change without needing controlled comparison. Leave this rung when two designs look equally plausible under inspection, or when stakeholders will not accept a change without side-by-side path metrics on the same usage.

Rung 2 — Counterfactual replay (medium)

restore snapshot
→ replay existing walks/events
→ compare

This is Mode 2 from Chapter 6. Restore the Git-pinned tuple — wiki commit, queue checkpoint, event-log offset, agent/tool configuration. Replay the same questions or the same event window under design A and design B. Score path length, search calls, canonical-source reach, abandoned branches, token cost, answer quality, and the 2×2 cell. Keep losers as regression corpus.

Stay on this rung when the corpus already exercises the behaviour under dispute. You are not short of walks; you are short of fair comparison. Leave this rung when the existing corpus simply never touches the new tool, the new queue action, or the new boot profile. Replaying the past cannot invent traffic the past never produced.

Rung 3 — Active evidence generation (dearest)

generate new real AI usage
→ collect more traces
→ inspect / replay

Escalate only when the existing corpus does not exercise the new behaviour. Generation is not a substitute for reading. It is coverage when coverage is genuinely missing.

The twelve-question automated usability protocol

When analysis says “we need more user walks,” dispatch actual AI users. The content of the method is concrete enough to run as a standing protocol:

Run these 12 representative questions
through ChatGPT, Claude and a cheap utility model.
Record every walk.

Build the twelve questions deliberately. They are not random prompts. Compose a battery that covers:

• two to three questions that should hit known canonical pages (happy paths);
• two that require multi-hop edges (relationship questions);
• two that use vocabulary outside your wiki’s house jargon (naive-user probes);
• two that are deliberately underspecified (force search vs clarification behaviour);
• two that target recent design changes (the behaviour the old corpus never saw);
• one adversarial or near-miss question (similar entities, easy to thrash);
• one “should refuse or escalate” question if your tools support that contract.

Hold the battery fixed for a generation cycle so results are comparable. Version the battery in Git next to tool configs. When you add a thirteenth question because a new defect class appeared, that is evaluator co-evolution — record why, do not silently churn the set mid-comparison.

Run across models, not one favourite. ChatGPT and Claude are named here as public products used as actual users; a cheap utility model is the third leg that exposes designs that only work when the model is strong enough to compensate for bad affordances. Chapter 9 expands the full variation protocol; the minimum for Rung 3 is already multi-model.

Record every walk into the WikiWalk schema (and queue-evolution lines if the task is a queue decision). Then drop immediately back to Rung 1: inspect the new corpus for defects before you invent more traffic. Only if two fixes compete do you climb to Rung 2 on the new material.

That is not merely synthetic testing. It is automated usability research for an AI-native interface. Nothing pretends to be a human shopper. The subjects are models; the product is an MCP or agent API; the traces are ground truth for that user class.

Shared walker economics

The shared walker makes Rung 3 particularly powerful: query, ingestion and janitor roles can be tested through one path harness when they use the same navigation machinery. One instrumentation investment covers several roles. If ingestion walks and query walks diverge wildly under the same map, you have learned something about role-specific boot profiles — a design fact, not a model fact.

Escalation rules on one card

Inspection first. Replay second. Generate only when the corpus fails to exercise the behaviour you care about.

Teams reverse this order constantly. They spin up synthetic batteries before reading a single production walk. They rebuild harnesses before naming a defect. The ladder is a discipline against that waste. It is also a discipline against false confidence: if you only ever inspect, you may never discover that two designs look the same until you pin the world and race them. Climb in order. Come back down as soon as you have a discriminating answer.

Because AI is the intended user, optimising for AI behaviour is correct. The main risk is subtler: one AI teaching the system its own habits. A beautiful ChatGPT session walk is not the same as robust affordances. Tool order, vocabulary, and boot profiles can lock in one model’s quirks until the next model generation makes the design look stupid overnight. Reflexive Agent Design without a variation protocol is ChatGPT-native design wearing a general label.

What overfitting looks like in agent UX

You optimise tool descriptions until one frontier model walks cleanly. Path length drops. Canonical reach rises. You ship. A utility model, or a second frontier with different tool-following bias, still thrash-searches and skips the edges you thought were obvious. Or a fresh context — no warm chat memory of your wiki vocabulary — cannot parse house jargon that the first session had already absorbed. The design did not become legible to the class of agents. It became legible to one session.

That failure mode is especially easy to miss under Reflexive Agent Design, because the loop is so tight. The same model family that generates walks may also review them and propose fixes. Without deliberate variation, the loop is a hall of mirrors: one set of habits diagnosing itself.

The multi-model variation protocol

Use variation across every axis that changes how an agent meets your affordances:

• Frontier and utility models — ChatGPT and Claude as named public products used as actual users; plus a cheap utility model that cannot paper over bad tool contracts with clever compensation.
• Fresh contexts — no warm conversational memory of the wiki, no prior turns that taught house names.
• Terse and conversational questions — keyword-ish asks versus full prose; both appear in real agent traffic.
• Direct and indirect formulations — “open the page on X” versus “what do we believe about X given Y?”
• Vocabulary-familiar and vocabulary-naive users — agents that know your framework names versus agents that only have public language.
• Tool descriptions and boot profiles — deliberately swap order, rename tools, shorten or lengthen descriptions; measure path change.
• Deliberately delayed or missing information — drop a page, delay a canonical source, remove an edge; see whether the walk fails honestly or luckily invents.

The aim is not to make one ChatGPT session walk beautifully. It is to make the system’s affordances legible across the class of agents intended to use it.

Worked protocol you can run

1. Fix a twelve-question battery (Chapter 8) and a Git pin (wiki commit + tool/boot version).
2. Run the battery on Model F (frontier), Model F′ (second frontier), and Model U (utility), each in a fresh context.
3. For one model only, re-run a subset under three prompt styles: terse, conversational, indirect.
4. For one model only, re-run under two boot profiles: search-first versus canonical-tool-first.
5. Score path metrics and 2×2 cells per run (Chapter 3).
6. Where models or styles diverge, refuse designs that depend on one model’s search bias or vocabulary preference.
7. Change tool contracts, map labels and boot order until path quality holds across the set — not until the favourite model looks pretty.
8. Keep the diverging walks as regression: the next redesign must not reintroduce one-model specialisation.

That is not optional polish. It is part of the method. Rung 3 of the ladder already multiplies models; this chapter is the full guard that also multiplies prompt style, vocabulary familiarity and boot profile. Skip it and you will industrialise one vendor’s habits into your architecture.

Why the synthetic-user critique does not bite here

Teams increasingly use LLMs as synthetic users to generate test conversations and broaden evaluation coverage for human products. Research warns that LLM-simulated users can systematically differ from humans: less diverse behaviour, weak correlation with human preference, shallow personalisation, and occasionally incoherent feedback.⁴ Related work on AI-simulated evaluation stresses that commercial synthetic-user tools and heuristic simulators need rigorous comparison against human evaluation before they substitute for real observation.⁵

That warning is real. It applies when the product user is a human and the LLM is a proxy. It is the right critique of “replace your usability lab with ChatGPT personas.”

It applies much less strongly to WikiArmy-style systems. ChatGPT was not pretending to be a human customer. The real production user of the wiki is an LLM operating through the MCP. Its tool choices, searches, backtracking and page reads are genuine usage behaviour from the intended user class. Nothing is simulated. The validity critique dissolves by ontology, not by hand-waving.

Hold both truths at once. Do not dismiss the synthetic-user literature when someone ships a human app tested only with personas. Do not let that literature block you from generating authentic agent traffic for an agent-native interface. The multi-model protocol is how you stay honest after the ontology inversion: ground truth is still not the same as one-session beauty.

Belief

We optimise for agents because agents are the users. We refuse to optimise for a single vendor session. Class-level legibility beats one pretty walk every time. If your ladder only ever climbs with one model, you are not doing Reflexive Agent Design — you are doing monologue.

You now have the joke that is the finding, the six conditions, the path 2×2, the WikiArmy receipt, dual streams, the progressive ladder, and the multi-model guard. This chapter is the operating protocol you can actually run — a checklist with enough detail to start this week, not a calendar fiction and not a vague aspiration.

Operating protocol

0. Preconditions (half a day once)

• An agent-facing surface worth testing (MCP, tool API, agentic wiki).
• Walk logging that can capture the WikiWalk fields from Chapter 7.
• If you have a queue: event-log offset, queue checkpoints, decision lines.
• Wiki (or world state) in Git; versioned prompts, tools and model identifiers.
• A human who will remain architect and judge — not a committee of miniature architects.

1. Instrument

• Record every substantive walk: question_or_task, wiki_commit, pages, edges, searches, abandoned branches, canonical sources, result.
• If applicable, record queue-evolution lines: event offset, source object, commits before/after, decision action/case/reason, next_review_at.
• Refuse to delete walks after the answer is green. Retained walks are the corpus.

2. Inspect (Mode 1) — default weekly and after every design change

• Sample recent walks across at least two models.
• Ask the inspection questions: thrash search, backtrack, ignored tools, long paths, right answer for wrong reasons, elicited behaviour.
• Classify each sampled walk into a 2×2 cell (Chapter 3).
• Name one design defect visible in paths only. Write it as a sentence: “Agents skip X because boot profile advertises Y first.”
• Change that affordance (tool contract, edge, boot order, label, case-decision rule).
• Do not climb higher if the defect is already discriminating.

3. Replay (Mode 2) — when two designs need a fair fight

• Restore the Git-pinned tuple: wiki_commit + queue checkpoint + event-log offset + agent/tool configuration.
• Hold the question set or event window fixed.
• Race design A versus design B (only the disputed surface changes).
• Score path length, search calls, canonical reach, abandoned branches, tokens, answer quality, 2×2 cell.
• Ship the winner’s contract; keep the loser’s walks as regression corpus.

4. Generate (Mode 3) — when the corpus cannot see the new behaviour

• Freeze a twelve-question battery (happy path, multi-hop, naive vocabulary, underspecified, change-targeted, near-miss, refuse/escalate if relevant).
• Dispatch the battery through ChatGPT, Claude and a cheap utility model, each in a fresh context.
• Record every walk into the schemas.
• Return immediately to Mode 1 on the new material. Climb to Mode 2 only if two fixes still compete.

5. Guard (every serious design decision)

• Multi-model variation: frontier, second frontier, utility.
• Prompt-style and boot-profile variation on a subset (Chapter 9).
• Refuse single-model beauty contests. If only Model F walks cleanly, the design is not done.

6. Close the loop

• Confirm the next walks hit the redesigned affordances.
• If they do not, the change was incomplete — inspect again, do not declare victory on the code review alone.
• Promote stable happy paths to regression goldens; keep lucky-cell walks as permanent counterexamples.

A one-afternoon starter (WikiArmy shape)

If you have never run the ladder, copy the founding afternoon rather than building a platform:

1. Dispatch five real questions through your MCP with one frontier model; record walks.
2. In a second window, ask a coding agent to review those walks for defects.
3. Name one defect; change one affordance the same day.
4. Re-walk two of the questions; confirm the path moved.
5. Only then invest in dual streams, Git pins and the twelve-question battery.

The method is real at that scale. Infrastructure multiplies it; it does not invent it.

What you leave able to do

• Inspect recorded walks for design defects.
• Replay queries across variants from a pinned Git state.
• Dispatch AI users for missing evidence.
• Grade paths, not only answers — on both graph and queue surfaces.

Boundaries

Framework position

Reflexive Agent Design
├── Trace-Driven Design Review   ← you can run this now
└── Replay-Driven Design Evolution ← next altitude

The deeper loop

The writer rehearses the read, the reader leaves a trace, the trace tests the design, and the redesigned system changes the next read.

File Back the Walk already says that querying improves the system because missing edges and cold pages become work rather than exhaust. The extension is sharper:

When AI is the principal user, it can cheaply generate the usage, inspect its own behavioural evidence, and run counterfactual versions of its future working environment before the design is committed.

That is not a testing trick. It is a design practice for a world where the user is already a model. Climb the ladder in order. Keep the human as architect and judge. Let AI be user, analyst and design assistant. Instrument both streams. Vary the agent class. Ship affordances that remain legible when the next model arrives. When the coding agent asks for more users, remember the joke — and dispatch one.