You are halfway through designing something that does not fit a template. Not another CRUD app. Not a thin wrapper around a vendor API. A system with competing patterns, fuzzy data relationships, and a grain problem that refuses to stay put when you name it.

In our case it was a personal AI news radar: wiki-grounded, signal-aware, quiet when silence was the right answer. The product details matter later. What mattered first was the mid-conversation realisation that keeps showing up whenever coding agents get good:

The design is the complex bit. The coding is the easy bit.

That line is not a slogan for keynotes. It is an operating diagnosis. If it is true, most of the AI-native workflow people are running is pointed at the wrong scarce resource.

Chat is not a design harness

Talking the architecture through with a strong model is genuinely useful. It formalises half-thoughts. It surfaces priors you forgot you had. It holds more competing factors in working memory than a whiteboard session on a Thursday afternoon.

It still fails a basic test.

It does not give you the feel of different ideas under pressure. It does not make two architectures collide with the same ugly history. It does not force a candidate to lose on detection latency, duplicate rate, or canonical-source correction. You leave the chat with a document that feels coherent — and no evidence that a different coherent document would have behaved worse on the same week of reality.

That gap is where technical debt now hides. Not in missing unit tests. In untested grain.

The problem this book owns

The reader question is simple and hard:

How do I evolve a genuinely novel system design with AI when neither of us can fully specify it upfront — and why doesn't blasting sub-agents at it work?

After these chapters you should be able to:

• hold one design steward and dispatch probes as instruments, not miniature architects;
• generate a few materially different candidate designs and race them against recorded reality;
• freeze world state so replay cannot cheat with future knowledge;
• read a measures table that makes architectures lose honestly;
• recognise the build boundary — when the next experiment must be a thin end-to-end system;
• keep learning in the design harness (corpus, ADRs, rejections), not only in the first codebase.

What this book deliberately does not own

Scope fences matter. Sibling work from the same publishing run already owns pieces of the product:

• The Signal-Case Queue is the product grain that survived the method — the queue as an active case index over a wiki graph. Read that artefact for queue doctrine; here it is the flagship case of design evolution, not a second queue manual.
• The Cascade Ledger is influence-as-receipts. Cameo only.
• Reflexive Agent Design — AI-as-user walks, usability trace analysis — is adjacent prior art. We use Git-pinned replay; we do not re-explain the usability methodology.
• Whether to fork open source as a full Extender essay lives elsewhere. We only need the economics: free code can trap you inside an alien North Star.
• Content-kernel Nuke and Regenerate economics are a reuse allowance, not a chapter.

If a paragraph wants to become a product tutorial for the radar, it is in the wrong book.

We do not know what we do not know

Open-ended design has a property that breaks ordinary task decomposition:

We don't know what we don't know at the start.

That is not romantic uncertainty. It is structural. The subtasks are not on the ticket yet. The quality criteria are incomplete. A discovery about case identity can reframe three other “finished” decisions. Methods that assume a clear definition of done — including many multi-agent coding workflows — silently change the problem into something easier and wrong.

The rest of this book installs the loop that respects that structure: Replay-Driven Design Evolution. Competing designs. Historical replay. No future leakage. Human recognition of new criteria. A harness that compounds.

Chapter 2 draws the old loop next to the new one — and thickens the left plate of the barbell.

Before coding agents, the expensive middle of software was implementation. Design documents were aspirational. Tests were late. The only honest teacher was a running system, so teams learned by building.

AI did not delete that teacher. It changed the price of the middle. You can regenerate a service from a better specification in an afternoon. What you cannot cheaply regenerate is a wrong object — the unit of work, the serial state model, the grain of attention — once it has colonised schemas, APIs, dashboards, and habits.

So the loop has to move left.

The loop most teams still run

Even with agents, the common cycle is still:

design → build → review → learn → update design → nuke → rebuild

Nuke-and-regenerate is already an upgrade over endless patch nursing. It says the generation recipe is the asset and the output is disposable. Good. Incomplete.

That loop still discovers design through implementation thrash. Each rebuild teaches you something — and each rebuild also encodes the previous misunderstanding into a larger surface of code that must be re-understood before the next insight lands.

When implementation was expensive, that cost was unavoidable. When implementation is cheap, paying it by default is a choice.

The faster loop

Because coding is easy relative to architectural judgment, the interesting iteration is design iteration before the product spine exists:

intent
  → generate competing designs
  → replay each against historical reality
  → compare behaviour
  → mutate / reposition designs
  → reject weak designs with reasons
  → repeat
  → only then code

Compressed:

Generate → Replay → Measure → Diagnose → Mutate → Replay
                                              ↓
                                        stable enough
                                              ↓
                                             Code

We call that pattern Replay-Driven Design Evolution (RDDE).

Candidate designs are synthetic futures — stand-ins for systems that do not yet exist — but they are grounded in real event streams and scored by an external harness rather than by a model admiring its own prose.

Counterfactual Design Replay is the instrument, not the whole factory

If you have already met Counterfactual Design Replay (from the Governance Barbell work), you know the testing question:

Had this design existed during these historical situations, what would it have seen, changed, missed, or made worse?

That is unit-testing an idea before the project exists. Generate → replay → measure → compare → human judge. Never generate → admire → approve.

RDDE is one level up. Replay does not merely approve or reject a design. It produces the next design population. Failures become mutations. Surprises become new measures. Rejected alternatives stay visible with reopening conditions. The loop is evolutionary, not a one-shot bake-off.

The thickened barbell

The Governance Barbell already said AI coding shifts effort toward the ends: heavy design challenge before commitment, cheap middle delivery, heavy independent verification after.

RDDE thickens the left plate:

HEAVY DESIGN EVOLUTION
  - competing architectures
  - synthetic design futures
  - historical replay
  - arrival-order perturbation
  - failure-shape tests
  - preserved rejections
          ↓
THIN IMPLEMENTATION
          ↓
HEAVY LIVE VERIFICATION

The governing line:

Do not iterate the code to discover the design when you can iterate synthetic designs against recorded reality first.

Or even more tightly — the line that titles this book:

The old AI loop regenerated implementations. The new loop regenerates designs before implementation exists.

What changes in practice

Three habits die:

1. One schema, many vibes. You stop treating the first plausible data model as the design. You require two to five materially different candidates.
2. Argument without a stream. You stop debating grain in the abstract when a week of bronze events could host the race.
3. Amnesiac iteration. You stop deleting losers silently. Visible rejection is how the next session does not re-litigate settled failures.

Three habits replace them:

1. A design workshop repository — cases, probes, decisions, rejections, open questions — not only a long chat.
2. A freeze protocol so replay cannot leak the future (Chapter 5).
3. A measures table that makes “feels right” lose to “behaved better on the same history” (Chapter 6).

Design remains the source of truth

None of this is an argument for infinite design documents and zero contact with reality. Later chapters install the build boundary: when remaining questions require operating behaviour, you build the smallest end-to-end spine that can falsify the current model. That spine is still design work. Its primary output is evidence, not a product launch.

The doctrine holds:

The design remains the source of truth, but it is empirically compiled — reality is allowed to edit it, and learning must return from code into design.

Chapter 3 attacks the other false shortcut: if design is hard, blast more agents at it.

The other tempting shortcut is parallelism without framing.

Coding agents got good at orchestrator–worker patterns. Agent Teams can fan work across a shared codebase. Research systems can dispatch specialised workers. So when design feels stuck, the reflex is: spin up more agents and “design the architecture in parallel.”

For open-ended design, that reflex usually produces expensive essays, duplicated assumptions, and a fusion job nobody budgeted for.

When multi-agent systems actually help

Anthropic’s engineering write-up on multi-agent research systems is clear about the fit: multi-agent approaches help mainly when tasks can be parallelised, context isolation is valuable, or workers have genuine specialisation. They also document the cost side — multi-agent versions commonly consume on the order of 3–10× more tokens than single-agent approaches and often lose important context through handoffs. Tightly coupled or sequential work is usually better kept in one persistent context.¹

Their broader guidance on effective agents prefers simple composable patterns; complexity should be earned.²

So the industry’s own best public guidance already tells you: parallelism is not free intelligence. It is a bet on independence.

Design fights independence

Open-ended design has several properties that fight ordinary sub-agent delegation:

• The subtasks are not known at the beginning.
• Discoveries change the framing of other discoveries.
• Important context is global and cumulative.
• Evaluation criteria evolve during the work.
• Alternatives cannot be judged independently of the whole system.
• The valuable output may be a newly noticed relationship, not a completed task.

A broad instruction such as “design the queue architecture” causes each agent to silently choose its own assumptions. Their outputs are then different answers to different questions. You did not get four peer designs. You got four partially specified problems.

Scott’s instinct from the source session is the right centre of gravity:

Blasting a bunch of sub-agents is not likely to help. Sub-agents work when you can explain the task, have a clear definition of done, and strong verification. Design usually does not fit that pattern.

What other people are almost doing

Before naming our topology, map the near neighbours so RDDE is not mistaken for any one of them.

Trace evals (LangSmith and peers)

Teams take production traces, add them to versioned evaluation sets, replay new agent versions, and compare trajectories — tool selection, arguments, loops, retrieval paths, latency, cost — not only final answers.³⁴

Close. Most practice still uses traces to validate a known change. RDDE uses them to discover what the design should become.

Reflective evolution (GEPA / DSPy)

GEPA maintains multiple candidate versions of an LLM program, runs them on examples, captures full execution traces and textual feedback, diagnoses failures in natural language, mutates targeted parts, and retains a Pareto frontier of complementary approaches. Reported results show strong gains versus some RL-style and prompt optimisers with far fewer rollouts.⁵

Closest optimiser loop. Still assumes a reasonably stable evaluation process. Hard design discovers new quality criteria mid-replay — path quality, queue thrash, canonical correction — so the scorer co-evolves.

CEGIS

Counterexample-guided inductive synthesis: propose → verify → counterexample → revise.⁶

Brilliant when the spec is formal. RDDE is CEGIS where counterexamples also teach you what the specification was missing. Human recognition remains scarce.

Design-space exploration

Architecture-as-search with numerical fitness works when you already know how to score throughput, latency, coupling, cost. Semantic behavioural objectives for attention systems are often discovered while watching walks. Pure optimisation without that recognition step is insufficient.

Session “dreaming”

Managed agents that periodically review prior sessions and curate memory point the industry toward treating traces as learning material. Present scope is usually memory hygiene, not redesign of the architecture that produced the sessions.

Pieces exist. The combined operating loop is still rare.

The topology that works

ONE DESIGN STEWARD
holds intent, current model, open questions and design history
        ↓ dispatches
SMALL TARGETED PROBES
generate a counterexample
test one hypothesis
analyse one trace family
challenge one design assumption
measure one property
        ↓ return evidence
SHARED DESIGN STATE
candidates, replay results, rejected alternatives,
newly discovered criteria, unresolved questions
        ↓
HUMAN MERGE

The sub-agents are not miniature architects. They are experimental instruments.

The memorable line:

Sub-agents search the evidence. The design evolves in one shared mind.

Or in Discovery Workshop language: one bench, many test tubes — not ten separate workshops producing ten final designs.

Good probes versus poor parallelism

Good:

Probe A: Find examples where queueing source objects creates duplicate cases.
Probe B: Replay cases where the canonical source arrives late.
Probe C: Try to prove the queue can be entirely rederived from the graph.
Probe D: Find serial state that would be lost under that design.

Poor:

Four agents: independently design the whole system.

You can also parallelise evaluators over the same replay — path quality, canonicality, cost, queue stability, adversarial “find the mishandled case” — because those lenses are genuinely separable. They do not each need to carry the entire design process.

Practical loop (operating view)

1. Hold one current design model.
2. Generate only a few materially different candidates.
3. Restore a Git-pinned wiki and queue state.
4. Replay recorded events (and walks when relevant).
5. Evaluate outputs and paths.
6. Let AI inspect traces for patterns not covered by existing metrics.
7. Convert surprises into a new counterexample, evaluation, design mutation, or open question.
8. Preserve rejected designs and reopening conditions.
9. Repeat until the design stops changing structurally.
10. Compile it into code.

This is still Discovery Workshop posture. Maturity is knowing when you have stabilised enough to switch into a PR Factory for implementation — not pretending factory metrics measure discovery quality.

Chapter 4 puts five real candidates into the harness.

Abstract argument about architecture is cheap and usually wrong in the same way chat is wrong: it never forces two models to process the same ugly week.

The flagship case for Replay-Driven Design Evolution was the queue at the heart of a wiki-grounded AI news radar. The product doctrine that survived — signal cases as queue grain, cascade influence as receipts — is published separately as the Signal-Case Queue and Cascade Ledger. This chapter owns only what the method needs: five competing designs raced through one harness.

Why the queue was the right object to evolve

A news radar is not interesting because it scrapes. It is interesting because it must allocate scarce human attention across developing stories whose identity, importance, and canonical anchors change as evidence arrives.

That means the queue is not a FIFO of posts. It is a temporal working set over a semantic graph. Getting the grain wrong produces immediate operational pain:

• duplicate alerts for one story arriving via Reddit and X;
• thrash when a better source appears;
• “concepts” that never resolve;
• silent loss of serial state if you pretend the graph snapshot is enough.

Those failure modes are perfect for design evolution: they show up in replay, they are measurable, and they mutate the model.

The design workshop is a repository, not a chat

Before the race, stop living only in conversation history. Put the work in a design workshop layout:

/design
  north-star.md
  current-model.md
  question-ledger.md
/cases
  reddit-first-source-later.md
  two-independent-origins.md
  false-viral-rumour.md
  old-idea-new-populariser.md
  one-concept-many-cascades.md
/probes
  queue-grain.md
  canonicality.md
  influence-graph.md
  review-scheduler.md
  briefing-surfaces.md
/decisions
  ADR-001-....md
/rejections
  queue-every-post.md
  queue-the-concept.md
  global-influencer-score.md
/open
  how-to-detect-independent-convergence.md
  when-does-a-case-resolve.md

Each parallel probe returns a design card, not an essay:

Question
Current hypothesis
Alternative models
Worked example
Failure case
What evidence would change the view
Implications for the data model
Unresolved questions

Essays hide assumptions. Cards force falsifiers.

Five candidates

Race at least these five. Names are disposable; differences must be material.

Design A — Queue every interesting source object

Unit: the tweet, the Reddit post, the GitHub repo, the newsletter item.

Promise: Simple. Faithful to arrival. Easy collectors.

Structural failure: One developing story becomes many queue entries. Reddit discussion of a Karpathy tweet, the tweet itself, a YouTube recap, and a repo all demand the same neighbourhood load and eventually discover they are one cascade. Duplicate rate explodes. Human attention pays the merge tax the system refused to pay.

Design B — Queue concepts

Unit: “agent-maintained wikis,” “tool use,” “eval harnesses.”

Promise: Semantic stability. One place for a theme.

Structural failure: Concepts can stay interesting for years. They do not naturally resolve. Review scheduling becomes meaningless. Heat never dies. You reinvent a second wiki under a queue label.

Design C — Queue signal cases with mutable anchors

Unit: a bounded evolving claim or development whose significance is not fully resolved.

Example:

SIGNAL CASE
“Agent-maintained wikis becoming a mainstream AI-agent pattern”

The case connects to concept nodes, people, posts, repos, counterevidence — but the queue item is the case ID, not any single observation. Canonical anchors can change. Discovery anchors can differ from canonical anchors. Review targets are the objects rescraped; the case is the identity that persists.

Promise: Middle grain — more durable than a post, more bounded than a concept.

Risk: Case boundary errors (merge/split/create) become the load-bearing failure mode. That risk is good: it is falsifiable in replay (Chapter 8).

Design D — Derive the queue from the graph every run

Unit: whatever a pure graph query says is “active” right now.

Promise: One source of truth. No second store to drift.

Structural failure: The queue is produced by a serial sensing process. Its state depends on arrival order, last observation time, review cursors, failed expectations, expected information gain, closing windows. Those are temporal and operational facts, not only semantic relationships. A single graph snapshot under-represents them.

Design E — Persist a materialised temporal working set

Unit: an active index row pointing into the wiki — case pointer, reasons active, next review, observation cursors — rebuildable from graph + event ledger + unresolved case state + policy if lost.

Promise: Operationally real without becoming a second epistemology.

Risk: If you let the materialised index become authoritative meaning, you have created a shadow wiki. The rule must stay: wiki holds meaning; queue holds active attention state.

The expected combination is not a substitute for the race

A thoughtful prior says C + E should win: signal cases plus a materialised temporal working set. Write that down as a hypothesis, not as a coronation.

The purpose of the harness is to let the favourite lose honestly if the stream says so. Maybe your domain’s “cases” thrash worse than slightly over-duplicating posts for a month. Maybe pure derivation works if your event ledger is rich enough and recompute is cheap. Race it.

How the race is run (preview)

Chapter 5 freezes the world and forbids future leakage. Chapter 6 installs measures. Chapter 7 walks the mutation that killed Design A in the Reddit-first story.

For now, only the discipline:

Generate design
    ↓
Replay Reddit-first scenario
    ↓
Replay tweet-first scenario
    ↓
Replay false-source scenario
    ↓
Replay two-independent-origin scenario
    ↓
Compare on shared measures
    ↓
Human chooses + records rejection reasons

That is Intent-Preserving Layer Shift applied to mechanism: same intent (allocate attention to developing significance), different grain and storage, scored before you generate the product.

Replay without a freeze protocol is cosplay science. The system looks brilliant because it already knows what mattered.

Scott’s bootstrap instinct from the source session is the right shape: roughly ninety days of historical ingestion, then replay the last seven days with controlled arrival-order variants so you do not always see the tweet before the Reddit post. That experiment only counts if you obey one non-negotiable rule.

The future-leakage rule

Do not replay the final seven days against a wiki that already knows what happened during those seven days.

If the gold graph already contains the later Karpathy tweet, the repositories that appeared on day six, and the engagement path that proved the story real, the queue will “detect” importance with hindsight smuggled in as prior knowledge. You are not testing discovery. You are testing whether your system can read a spoiler.

Future leakage is the silent invalidation mode for every historical replay method — agent evals, counterfactual design, decision regression. Name it. Forbid it.

Protocol: five steps

1. Bootstrap the historical substrate (~first 83 days)

Ingest selected sources into an immutable bronze layer. Materialise useful working resolutions into silver if you use a medallion split. Promote recurring navigable meaning — people, concepts, provenance, durable relationships — into gold.

No live queue is required yet. Historical ingestion builds understanding; the queue will hold only the active frontier later.

The bronze/silver/gold split matters here: bronze keeps everything so you can always re-interpret; gold holds only what earned graph citizenship.

2. Freeze the world at day 83

Create a versioned snapshot:

wiki_as_at_replay_start

Pin it in Git (or equivalent immutable versioning). Record:

• wiki commit hash
• configuration / prompt / model versions that will judge during replay
• policy hashes for review scheduling if they exist

The replay system is forbidden to see future observations. Treat violations as test contamination, not as clever caching.

3. Replay the final seven days serially

Feed observations one at a time in a defined order:

08:03 Reddit post
08:47 secondary newsletter
09:15 original tweet discovered
11:30 tweet engagement update
16:00 Reddit top-comment update
next day GitHub repository appears

Each event may:

• update bronze
• mutate a disposable replay branch of the wiki
• create or modify a signal case
• reprioritise the queue
• schedule future review

Do not batch the week into one bulk “ingest.” Serial state is the phenomenon under test.

4. Run controlled arrival-order variants

Keep the real order as baseline. Then create deliberate permutations:

• Reddit first
• original tweet first
• aggregator first
• primary source delayed six hours
• one source temporarily missing
• engagement burst before technical corroboration

This is not random scrambling for theatre. It tests whether the architecture is robust to realistic discovery order — the exact ambiguity that made the radar design hard in the first place.

5. Compare designs on the same corpus

Designs A–E (Chapter 4) each run the same freezes, the same serial feeds, the same permutations. Only then do measures (Chapter 6) mean anything.

You do not need to replay the whole wiki

Replaying every raw API payload for ninety days is usually too expensive in tokens and time, and it is the wrong grain.

Apply the grain rule to replay itself:

The unit of replay is not necessarily the raw API response.

For high-value cases, retain event deltas:

post first observed
canonical source discovered
engagement snapshot changed
top comments changed
new influential participant entered
repository appeared
first-party confirmation appeared

A useful corpus might be:

• all final-seven-day observations in compact event form
• 10–20 hand-labelled important cases
• 10–20 convincing noise cases
• several canonical-source-late cases
• several hot-then-dead cases
• several slow-burn cases that initially looked unimportant

Enough to evolve the design cheaply. Not a second internet.

Exact replay state

When you later add walks or decision traces, the pin expands to:

wiki commit
+ queue checkpoint
+ event-log offset
+ agent/tool configuration

Append-only events, versioned prompts/models, and checkpoints turn “we tried something” into a scientific instrument.

Bootstrapping the live frontier after the race

The freeze protocol is for evaluation. Going live is a different motion:

90-day ingest → historical understanding
recent aggregator / hot-source scan → active frontier
active frontier × wiki → initial queue

You do not need to wait a week for the queue to “warm up” if the wiki already knows what is old. New observations jump into cases almost immediately because novelty is a diff against existing knowledge, not a blank calendar.

Reddit and aggregators are strong bootstrap sensors precisely because they solve early discovery before the influence graph knows whom to watch on X. Over time, learned influence can shift collection effort toward high-signal people without hard-coding a permanent VIP constitution — that product story belongs to the Cascade Ledger; the method point is that bootstrap sensors and live sensors need not be identical.

What “pass” looks like for the protocol

A freeze protocol passes when a skeptical engineer can answer:

1. What is the as-of timestamp of world knowledge?
2. Which events are allowed after that timestamp?
3. Which design variants saw which permutations?
4. Where are the decision traces stored?
5. What would count as contamination?

If any answer is “the model just knew,” you do not have replay. You have a demo.

Chapter 6 installs the scoreboard that makes winners and losers legible.

Without measures, design races become rhetoric. Each candidate ships a narrative. Humans pick the story that matches their prior. The harness must make architectures lose on shared numbers — and still leave room for the scorer itself to grow.

The measures table

For the news-radar queue race, compare designs on questions like these:

You do not need all nine on day one. You need enough that a favourite can fail. Three is a minimum honest set for a first race: duplicates, canonical correction, and detection latency (or cost). Add precision/recall when you have labels for what later mattered.

What each measure falsifies

Detection latency falsifies designs that are semantically pretty but slow to notice developing significance — often concept queues that wait for too much evidence, or pure derivation that recomputes too late.

Duplicate rate is the executioner of Design A. If source objects are queue units, multi-channel cascades produce multi-entry noise. Humans experience it as “this system keeps yelling about the same thing.”

Canonical correction falsifies designs that treat the first observation as destiny. The Reddit-first story is the textbook case: provisional anchor must be allowed to demote without destroying the case identity or the propagation evidence.

Attention precision/recall are the adult metrics. Precision without recall is a quiet system that misses the story that mattered. Recall without precision is a firehose. Labels are expensive; start with a small hand-labelled set from the seven-day window and grow.

Revisit efficiency catches the design that “works” by rescraping everything constantly. Token and API budgets are part of architecture.

Influence learning connects to the Cascade Ledger sibling: did the graph surface a cascade root without a VIP whitelist? If your design only works when Karpathy is hard-coded, you have not learned influence — you have smuggled a seed list into the test.

Queue stability falsifies thrashy heat functions. If cases flip hot/cold every hour without new evidence, operators stop trusting the system and start babysitting it.

Resolution quality falsifies eternal queues. Dead stories must expire without erasing bronze history or useful wiki meaning.

The evaluator is also being discovered

Here is the part pure optimisers under-serve.

While watching walks and decision traces, you notice failures your original scorecard did not name:

• right answer, bad path (lucky retrieval that will not generalise);
• case boundary errors that look fine on day one and poison corroboration by day four;
• silence that was correct (not a miss) versus silence that was a miss;
• independent convergence mis-tagged as amplification.

Those are not “implementation bugs.” They are missing specification.

This is why GEPA-like reflective evolution is necessary but not sufficient. GEPA can mutate candidates against a scorer and keep a Pareto frontier. Hard design work also mutates the scorer when human judgment recognises a new quality criterion in the traces.

Call it:

CEGIS where the counterexamples teach you what the specification was missing.

The synthesiser (design generator) and the verifier (harness) co-evolve. The scarce input is still human recognition of what “a good walk feels like” and what “a good queue decision” means in this domain.

How to grow the harness without gaming it

1. Start with structural measures that do not require deep labels (duplicates, thrash, cost).
2. Add semantic measures only with explicit label sources and as-of dates (so labels cannot leak future knowledge either).
3. Version the measures alongside designs. When you add “path quality,” record the date and the motivating counterexample.
4. Keep minority metrics on a Pareto view. One design may win on cost and lose on recall; premature single-score collapse hides complementary survivors.
5. Refuse vanity. “Model said this design is elegant” is not a measure.

Worked expectation for the five candidates

Without pretending to a single numeric result for every domain, the shape of expected failure is part of the method:

C+E is the hypothesis that usually survives — if case boundaries are disciplined and the index remains a pointer, not a photocopy of the wiki.

From measures to mutations

A measure that fails is not a scoreboard insult. It is a generator:

failed measure
  → concrete counterexample case
  → design mutation or new open question
  → possibly a new measure
  → re-race

Chapter 7 walks one mutation in full: the move to signal cases forced by Reddit-first arrival.

This is the worked mutation the brief demands: a design change forced by a reality collision, not by abstract taste.

The scenario that broke “queue the post”

You are reviewing sources. A Karpathy tweet about agent-maintained wikis is obviously queue-worthy — if you see it first.

But discovery order is not destiny. Sometimes Reddit surfaces the conversation first: a post that talks about the tweet before your collectors see the tweet. Later you find the original. Later still, repositories and commentary appear.

If the queue unit is the source object, you face a bad menu:

1. Create a queue item for the Reddit post, then another for the tweet, then more for descendants — and teach humans to merge by hand.
2. “Kick out” the Reddit item when the tweet arrives — and throw away propagation evidence (stronger technical discussion, new people, implementation links) that still matters.
3. Invent ad-hoc merge logic while pretending you still have a simple model.

All three are symptoms of the wrong grain.

What the harness sees

Replay the week twice under Design A (queue source objects):

Permutation 1 — tweet first. One primary item. Reddit arrives as… another item? Related item? Comment? The model improvises. Duplicate rate and review load rise as channels multiply.

Permutation 2 — Reddit first. A provisional item exists for the Reddit post. When the tweet arrives, either:

• the system treats it as new (duplicate developing story), or
• it overwrites/kicks the Reddit row (loss of discovery anchor and discussion value), or
• it grows a special-case “related posts” structure that is a signal case in denial.

Measures light up: duplicate rate, canonical correction (if you even have a place to record promotion), attention precision (noise of multi-entry stories).

Design B (queue concepts) fails differently: “agent wikis” becomes an eternal heat blob. The specific cascade never gets a lifecycle. Resolution quality collapses.

Design D (derive from graph) can describe relationships after the fact but struggles to hold “we last looked at the Reddit comments at 16:00 and the next scrape is due because expected information gain is high” — serial operational state.

The mutation: the queue unit is a signal case

The design change:

Not the tweet. Not the Reddit post. Not the broad concept. The queue contains an evolving signal case.

Example identity:

signal.agent-wiki-mainstreaming
current claim: "Agent-maintained wikis are becoming a mainstream agent pattern"

Step 1 — Reddit discovered first

signal_case_id: signal.agent-wiki-mainstreaming
canonicality_status: provisional
discovery_anchor: reddit.post.456
canonical_anchor: reddit.post.456

The Reddit post is not declared historical origin. It is the best current anchor.

Step 2 — Original tweet found

Wiki edges record reference and authorship. Canonicality is a judgment with reasons — not merely another retrieval hit.

signal_case_id: signal.agent-wiki-mainstreaming
canonicality_status: settled_for_current_cascade
discovery_anchor: reddit.post.456
canonical_anchor: twitter.post.123

The queue item is not replaced. The queue item was never the Reddit post. Its anchor was replaced.

The Reddit post remains as propagation evidence: discussion quality, criticism, older links, new people, implementations. Demoted from canonical anchor — not deleted from meaning.

Review targets are not the case

The scheduler needs objects to rescrape:

review_targets:
  - object: twitter.post.123
    reason: origin conversation
    cadence: 30 minutes
  - object: reddit.post.456
    reason: technical community interpretation
    cadence: 3 hours
  - object: github.search.agent-wiki
    reason: implementation uptake
    cadence: 24 hours

So:

• signal case = queue unit (identity)
• review targets = things rescraped
• wiki neighbourhood = interpretation

That is cleaner than putting every object independently in the queue.

Bronze, wiki, and queue keep distinct jobs

The mutation only stays clean if layers stay honest:

Observations go into bronze. Relationships and conclusions go into the wiki. Developing stories become signal cases. The queue holds pointers to active signal cases and schedules which evidence objects deserve another look.

Visible rejection: what lost and why

Record the losers so the next session does not re-open them casually:

Rejected: queue every post Reason: Multi-channel cascades produce duplicate attention units; merge tax externalised to humans. Reopen if: A domain has truly independent items per URL with no cascade structure (rare for news intelligence).

Rejected: queue the concept Reason: Concepts lack natural resolution and bounded review lifecycles. Reopen if: The product is a topic browser, not an attention system for developing episodes.

Rejected: pure graph derivation with no materialised index Reason: Serial sensing state (cursors, last observed, expected failures, next review) is not a pure semantic snapshot. Reopen if: Event ledger + recompute is proven complete and cheap enough that materialisation is pure cache with identical semantics.

Preferred hypothesis: signal case + materialised temporal working set Survived because: Case identity stable under anchor promotion; operational state explicit; rebuildable if lost; measures improve on duplicates and canonical correction in the Reddit-first race.

Concept, case, cascade (boundary preview)

The mutation raises the next hard problem — intentionally:

CONCEPT  — enduring subject in the wiki
CASE     — bounded developing episode currently interpreted
CASCADE  — observable propagation / influence lineage

They may not need three storage engines. They need three conceptual jobs so the case does not become too broad to resolve or too narrow to accumulate meaning. Chapter 8 schedules collisions that force those boundaries into a contract.

Why this chapter is the proof of RDDE

Nothing in the signal-case move required a full product build first. It required:

• competing candidates,
• a freeze-safe serial replay,
• measures that care about duplicates and canonical correction,
• a human (with AI help) recognising that the identity of the queue unit had to change.

That is Replay-Driven Design Evolution in miniature: replay did not only score designs — it produced the next design.

If you look back at how a real design conversation advanced, the main jumps rarely came from “break the problem into ten tickets.”

They came from perturbation.

Outside products showed that clustering, influence detection, and momentum sensing already exist separately. Wiki priors clarified that the graph holds meaning while the queue holds unresolved attention. A worked multi-source example forced the move from posts to signal cases. Serial-state arguments killed pure derivation. Human recognition named what still felt wrong when the prose looked fine.

Each important outside influence forced a redraw of the object.

So the next move after a partial victory (signal cases) is not another general design discussion. It is another collision with reality.

Decomposition is the wrong metaphor

Decomposition assumes the whole is known well enough to partition. Open-ended design violates that assumption. Subtasks appear after perturbations. Criteria appear after failures. The method must schedule stimuli that change the model, not only tasks that implement the current model.

Scott’s line from the session:

You definitely needed perturbation.

AI is strong at holding competing factors, formalising shape, and bringing historical priors into the conversation. It still needs collisions — outside systems, worked examples, replay surprises, human taste — to stop polishing the wrong object.

The next collision: case boundary through time

“Signal case” is useful because it is flexible. That flexibility hides the next load-bearing problem:

What exactly makes two observations part of the same case, and when does one case become two?

Consider variations:

• Reddit discusses a Karpathy tweet: same case.
• A GitHub repository implements the tweet: evidence, or a new implementation case?
• Twenty repositories appear: still one diffusion case, or an ecosystem case beneath it?
• Karpathy tweets about agent wikis again a month later: reopen, continuation, or new episode?
• Another influential person independently reaches the same idea: merge, or preserve two lineages?
• The theme branches into personal memory, enterprise knowledge, coding-agent memory: one case becomes three movements?
• Original conversation dies; six months later a major product ships it: reopen or new adoption episode?

Working hypothesis:

A queue case is an open hypothesis about a bounded developing episode — not the timeless concept and not an individual source object.

Example claim:

“Karpathy’s formulation is moving agent-maintained wikis
from a niche memory pattern into mainstream AI-agent practice.”

Tweet, Reddit, YouTube, repos are evidence testing that hypothesis — not the hypothesis itself.

Ten falsification scenarios

Take roughly ten real developments from the bootstrap window. Convert them into chronological event streams. Include messy shapes:

1. Secondary source arrives before primary.
2. Two independent sources converge.
3. Viral claim later proves false.
4. Quiet source becomes important days later.
5. One influential post generates several implementation branches.
6. Old concept suddenly reactivates.
7. One story gradually turns into two distinct stories.
8. Several apparent stories later collapse into one.
9. Large engagement but no substantive adoption.
10. Small engagement but consequential implementation.

Feed each event serially. Force the queue to choose one action:

ignore
attach to existing case
create case
promote canonical anchor
merge cases
split case
resolve case
reopen case

At every step, ask:

• What is the case currently claiming?
• What changed because of this observation?
• Why is it still in the queue?
• What would make this a separate case?
• What evidence would resolve it?
• What is worth observing next?

That exposes requirements faster than designing more fields in the abstract.

The artefact this collision should produce

Not a database schema yet. A Case Boundary and Lifecycle Contract:

What creates a case?
What attaches as evidence?
What causes a merge?
What causes a split?
What resolves a case?
What reopens it?
How are parent and child cases represented?

You will likely need to keep three conceptual layers distinct even if storage collapses some of them:

CONCEPT  — enduring subject in the wiki
CASE     — bounded developing episode
CASCADE  — propagation / influence lineage

Distinguishing them stops the case from becoming either too broad to resolve or too narrow to accumulate meaning. Full cascade-ledger product doctrine is the sibling book; here the contract is a design-evolution output.

Scheduling the next collision (operating habit)

After every design card session, write one line:

Next collision: _______________
Because the model is still soft on: _______________

Examples:

• Next collision: ten case-boundary streams — soft on merge/split identity.
• Next collision: cost-limited revisit policy on a hot week — soft on efficiency vs recall.
• Next collision: probe build on serial create/attach — soft on whether the decision trace is even capturable.

Perturbation is not randomness. It is deliberate contact with the class of reality your current model is least entitled to assume.

Discovery Workshop posture again

This chapter is the human-as-sensor doctrine applied to design time. Parallel probes can generate candidate boundaries or adversarial cases. The steward holds the contract. The human notices when Probe A’s merge rule destroys Probe B’s independent-convergence evidence.

Do not automate away the collision sensor while the unit of production is still understanding.

Chapter 9: when even these paper collisions stop teaching, you have reached the build boundary.

There is a moment when more design cards produce diminishing truth.

Not because the design is finished — because the remaining questions require behaviour only an operating system can produce.

Serial arrival. Uncertain case identity. Delayed canonical sources. Mutable importance. Merge and split. Changing observation targets. Attention scheduling. Interaction between event history and graph meaning.

You can describe those. Eventually you must watch them happen.

Two bad extremes

Keep designing until the whole architecture feels settled

Build the full product and discover the architecture through technical debt

The useful middle:

Build the smallest end-to-end system capable of falsifying the current design.

That is more than a mock-up and less than a product. Its primary output is not functionality. Its primary output is design evidence.

current design
    ↓
thin working system
    ↓
real serial events
    ↓
queue decisions and traces
    ↓
AI + human review
    ↓
revised design
    ↓
regenerate the affected system

This is the design flywheel. Implementation becomes an instrument for discovering the design — not a commitment that freezes wrong grain into a six-month codebase.

Still design-first — not design-only

Design remains the durable intellectual asset; code remains regenerable expression. That Design-Compiler doctrine is still right.

There is a limit to what a document reveals without contact with native data. For attention systems, native difficulty is serial and stateful. So the upgraded model is:

coarse design
→ design-probe build
→ behavioural evidence
→ better design
→ regenerated build

The resolution of the false fight:

The design remains the source of truth, but it is empirically compiled — build early enough to learn, never let the code become the only place where the learning lives.

Empirically compiled design is what “vibe coding evolved” looks like when you are honest: a real North Star, contact with reality, learning written back into design documents, regeneration of the spine when the model changes — not only local patches that bury insight in diffs.

Vertical slice checklist

Build a chunk — but only one spine. Include enough architecture to expose serial shape:

1. Append-only incoming event log
2. Bronze source objects
   Reddit post, tweet, GitHub item, aggregator item
3. Minimal wiki integration
   lookup concepts, people, prior cases and sources
4. Signal-case mutation
   create, attach, promote anchor, merge, split, resolve, reopen
5. Materialised queue
   active case, reason active, next observation target
6. Full decision trace
   what arrived, what the system read, what it decided, why
7. Replay runner
   wiki Git commit + queue checkpoint + event offset
8. Minimal inspection surface
   files/CLI first — not a polished UI

That is a substantial working spine. It is enough to crash the architecture into real data.

It does not initially need:

• every social source
• sophisticated influence scoring
• perfect engagement collection
• complete briefing interface
• production scheduling infrastructure
• final ranking formulas
• microservices theatre
• multi-user support

The falsifiable question:

Given a serial stream of observations and an existing wiki, can the system maintain a useful set of evolving signal cases and explain every queue transition?

Code as laboratory apparatus

Sometimes code is a product output. Here, early code is closer to a scientific instrument:

The event log creates the experiment.
The queue engine runs the hypothesis.
The traces are the observations.
The replay harness repeats the experiment.
The design document records what was learned.

The implementation is allowed to be wrong because its job is to make the wrongness visible.

Mission-shaped software thinking applies: the tool takes the shape required by the present obstruction; the tool remains disposable while the recovered understanding is the asset.

Why not start from someone else’s GitHub project?

Code is not neutral. A mature repository embodies thousands of design decisions: unit of work, state model, data grain, source assumptions, audience, latency, UI, storage, success definition, deliberate ignorance.

That is its compiled North Star.

You can delete features, but deeper assumptions remain in schemas, module boundaries, names, tests, abstractions, dependencies, control paths. Adopting often means:

their North Star
+ your desired behaviour
+ adapters between them
+ growing explanation debt

The technical debt is not necessarily poor code. It is good code organised around a different purpose.

When software development was expensive, accepting that mismatch was rational. With cheap AI coding, it can be cheaper to extract useful ideas and regenerate around your own design than to spend weeks unbending somebody else’s architecture.

Use GitHub as a parts and priors library

Wrong question: which project should we fork?

Right question: what did each project learn that we should carry into our design?

Extract connectors, event schemas, dedupe techniques, scheduling strategies, graph algorithms, API peculiarities, operational failure modes, tests, fixtures, terminology, things they tried and later removed. Borrow commodity libraries where the contract is truly commodity.

Copy discoveries, not destiny.

The design flywheel (named)

SHAPE     Hold North Star and current architecture.
BUILD     Generate a thin but complete working spine.
EXPOSE    Run real events (and real AI users when relevant).
TRACE     Capture transitions, walks, decisions, costs.
INTERPRET AI detects patterns; human recognises changed meaning.
REVISE    Update boundaries, invariants, schemas, evaluation criteria.
REGENERATE Change or replace implementation from improved design.
REPLAY    Run the same reality against the new design.

The improving object is specifically the system design:

system use → behavioural trace → design learning
→ design mutation → regenerated system → changed use

The loop compounds only when each cycle changes the substrate the next cycle operates on — not when you merely accumulate observations in a graveyard of chats.

Chapter 10 lists what must remain when the first probe build is gone.

If the first probe build burned down tomorrow, what should still exist?

If the answer is “only some generated code,” the method failed. The durable outputs of Replay-Driven Design Evolution are the harness and the history — not the first codebase.

The durable artefacts

From a serious design-evolution cycle you should be able to point at:

design.md / north-star.md
invariants.md
open-questions.md
case-lifecycle.md          (boundary contract)
replay-corpus/             (events, labels, permutations)
event-log/                 (append-only observations)
walks/                     (if AI-user traces matter — sibling topic)
decisions/                 (ADRs)
rejections/                (with reopening conditions)
probes/                    (design cards)
measures.md                (versioned scoreboard)

The code matters because it runs the experiment. The corpus, invariants, decision history, and discovered distinctions matter because they make the next implementation better — and the next human session faster.

That is the asset that appreciates.

Two assets at once

A co-design session on a novel system produces:

1. The actual system under design (here: a news-signal / queue architecture and its probe build).
2. A reusable method for co-designing new systems with AI when neither party can fully specify the answer upfront.

The second may ultimately travel further than the first. Every future project inherits the steward/probes topology, the freeze protocol, the measures habit, the rejection discipline, and the build-boundary instinct.

You are not merely using AI to implement a design. You and AI are jointly observing how the design comes into existence:

rough human shape
→ AI supplies priors and formal structure
→ outside examples perturb the model
→ human recognises what is still wrong
→ thin implementation meets reality
→ traces expose hidden requirements
→ AI helps reinterpret them
→ design changes
→ implementation morphs

The project is a case study and an instrument for discovering the design method itself.

Monday operating checklist

1. Write the North Star and coarse model in a design repo — not only in chat.
2. Generate two to five materially different candidates, not paraphrases of one schema.
3. Assemble a replay corpus with an explicit freeze point; forbid future leakage.
4. Define measures before the race (at least duplicates, one semantic correction, latency or cost).
5. One steward; probes as instruments with discriminating questions.
6. Convert surprises into counterexamples, new metrics, mutations, or open questions.
7. Preserve rejections with reopening conditions (Visible Rejection / John West applied to designs).
8. When traces stop teaching, build the smallest falsifying end-to-end spine.
9. Write learning back into design and regenerate affected implementation — do not let the codebase become the only memory.
10. Switch posture to PR Factory only when structure has stabilised enough that the unit of production is changes, not understanding.

What to refuse

• Arguing abstractly whether a schema “feels right” with no shared stream.
• Blasting whole-architecture sub-agents and calling the collage a design.
• Replaying history against a wiki that already knows the ending.
• Forking a near-miss open-source project to avoid design work, then spending months unbending its North Star.
• Shipping learning only as code comments and lost chat threads.

Closing the loop

The old AI loop regenerated implementations. It was rational when code was scarce.

The new loop regenerates designs against recorded reality — competing candidates, freeze-safe serial replay, measures that force honesty, probes under one steward, probe builds when paper stops teaching, harness and history as the compounding assets.

Counterfactual Design Replay asks whether a design would have worked. Replay-Driven Design Evolution uses the answer to breed the next design and treats the scoreboard as capital.

Chat is not enough. Parallel essays are not enough. Morphing code without a harness is amnesia with a Git log.

Race the designs. Freeze the world so you cannot cheat. Keep the scoreboard. Compile when the structure stops thrashing.

The design is the complex bit. The coding is the easy bit.

Act like both halves are true.