LeverageAI · Design method

Replay-Driven Design Evolution: The Old Loop Regenerated Implementations — the New Loop Regenerates Designs

Because AI made code cheap, iterating implementations to discover a design is now the slow path. The fast path races competing designs against recorded reality — with a freeze protocol that forbids cheating — and treats the harness that scored them as the durable asset.

Scott Farrell · LeverageAI · July 2026

Somewhere in the middle of designing a personal AI news radar, the real problem stopped being “can we code this?” It became: what is the grain of the queue, how does arrival order change meaning, and how do we evolve a design when neither human nor model can fully specify it upfront?

The blunt line is the one worth keeping:

The design is the complex bit. The coding is the easy bit.

If that is true — and for novel architectures in 2026 it usually is — then the default AI workflow is backwards. We still spend our scarce judgment arguing about schemas in chat, or regenerating large implementations until something “feels right,” or spinning up parallel agents that each invent their own silent assumptions. None of those moves makes competing designs collide with the same reality under the same measures.

This article names the loop that does: Replay-Driven Design Evolution (RDDE). Counterfactual Design Replay is the testing mechanism — run a proposed design against historical traces. RDDE is the larger generative loop that uses those results to breed the next design population, and that treats the scoring harness as the asset that appreciates.

The old loop and the new loop

The loop most teams still run, even with coding agents, looks like this:

design → build → review → learn → update design → nuke → rebuild

It is not stupid. When implementation was expensive, you had to learn through builds. Nuke-and-regenerate even became rational once AI could rewrite a subsystem from a better kernel. But the expensive step moved. What burns time now is discovering that the object was wrong — the queue unit, the case boundary, the serial state you never modelled — after a large volume of code has already compiled that mistake into schemas, tests, and muscle memory.

The faster loop, once code is cheap, is:

intent
  → generate competing designs
  → replay each against historical reality
  → compare behaviour
  → mutate / reposition designs
  → reject weak designs with reasons
  → repeat
  → only then code

Or more tightly:

The old AI loop regenerated implementations. The new loop regenerates designs before implementation exists.

That is a thickening of the left plate of the Governance Barbell: heavy design evolution (competing architectures, historical replay, arrival-order perturbation, preserved rejections), thin implementation, heavy live verification. You do not iterate the code to discover the design when you can iterate synthetic designs against recorded reality first.

Why blasting sub-agents fails design

The instinct is understandable. Coding agents got good at orchestrator–worker patterns. Anthropic’s multi-agent research systems show real gains when work can be parallelised, context isolation helps, or workers have genuine specialisation.¹ Prefer simple composable patterns; complexity should be earned.²

Open-ended design is a different animal. Subtasks are not known at the start. Discoveries reframe each other. Evaluation criteria evolve mid-flight. Alternatives cannot be judged independently of the whole system. The valuable output may be a newly noticed relationship, not a completed task.

So a broad instruction like “design the queue architecture” sends four agents to write four plausible essays against four different silent assumptions. You then pay the fusion tax. Anthropic’s own guidance is clear that multi-agent versions commonly consume on the order of 3–10× more tokens than single-agent approaches and often lose important context through handoffs; tightly coupled or sequential work is usually better kept in one persistent context.¹

The topology that works is not ten miniature architects. It is:

That is Discovery Workshop posture, not PR Factory posture: the human is the collision sensor, not a latency to eliminate. Parallelism is welcome after you have framed a discriminating question — “find examples where queueing source objects creates duplicates,” not “independently design the whole system.”

What other people are almost doing

RDDE sits next to several public practices. Naming the difference is the point of the framework.

LangSmith-style trace evals. Teams turn production traces into versioned datasets, replay new agent versions, and score trajectories — tools, arguments, loops, latency, cost — not only final answers.^3,4 The workflow is observe → identify failure → add to eval set → change → replay → deploy. That is close. The usual purpose, though, is to validate a known change. RDDE uses the same machinery to discover what the design should become.

GEPA (Genetic-Pareto) around DSPy. GEPA maintains multiple candidate LLM programs, runs them on examples, inspects full traces, reflects in natural language, mutates components, and keeps a Pareto frontier of complementary approaches. It can beat heavy RL-style optimisers with far fewer rollouts on reported benchmarks.^5,6 The loop shape is almost RDDE. The assumption is different: GEPA needs a reasonably stable evaluation process. In hard design, the evaluation function is also being discovered — you realise mid-replay that canonical-source reach or queue thrash should have been measured in the first place.

CEGIS. Counterexample-guided inductive synthesis proposes a candidate, verifies it, returns a concrete counterexample, and revises.^7,8 Brilliant when the specification is formal. RDDE is CEGIS where the counterexamples also teach you what the specification was missing. The scarce input is human recognition of new quality criteria in the traces.

Design-space exploration. Architecture-as-search is old in systems engineering. It works when fitness is known (latency, cost, coupling). Semantic objectives — did the system see one developing story rather than four duplicate posts? — are often discovered while watching the walk. Pure optimisation without that recognition step is insufficient.

The design harness is the durable asset

For a news-signal system — or any attention system over a graph — the harness eventually holds scenarios like: primary tweet first; Reddit first; aggregator first; primary never found; two independent origins; famous amplifier of an obscure source; viral signal later false; quiet post becomes important three days later; related-looking posts that are distinct cases.

Each design variant runs through the same event stream. You compare behaviour on measures that force honesty:

That is better than arguing whether a schema “feels right.” Rejected designs stay on disk with reopening conditions. Architecture Decision Records capture what survived. The first codebase is allowed to be wrong because its job, early on, is laboratory apparatus — to make wrongness visible.

Store the work in a design-workshop repository, not only a chat log: north star, current model, question ledger, cases, probes, decisions, rejections, open questions. Each probe returns a design card — question, hypothesis, alternatives, worked example, failure case, falsifier, data-model implications — not an essay.

Flagship: five queue architectures, one freeze protocol

The running case was a wiki-grounded AI news radar. The product objects of that work — the Signal-Case Queue and the Cascade Ledger — are siblings. What this article owns is the method that evolved the queue grain.

Five candidates raced the same stream:

• Design A — queue every interesting source object (tweet, Reddit post, repo).
• Design B — queue concepts (“agent wikis”).
• Design C — queue signal cases with mutable anchors.
• Design D — re-derive the queue from the graph every run.
• Design E — persist a materialised temporal working set.

A and B fail for structural reasons you can state abstractly — too narrow (duplicates) or too broad (never resolves) — but the harness is what lets them lose with numbers. D under-represents serial state (what arrived, when last observed, review cursors, failed expectations). The winner that survived honest replay was roughly C + E: a signal case as the stable identity of a developing episode, plus a materialised active index that can be rebuilt from graph + event ledger + policy if lost.

The Reddit-first / tweet-later story is the signature mutation. Queue the Reddit post as the unit and you either thrash or “kick out” valuable propagation evidence when Karpathy’s tweet appears. Queue a signal case and the case ID stays stable while the canonical anchor promotes. The queue item was never the Reddit post; its anchor was. That is a design mutation forced by a reality collision — not a preference about YAML shape.

The future-leakage rule

The bootstrap experiment that makes the race fair:

1. Ingest ~83 days into bronze/silver/gold. Build people, concepts, provenance. No live queue required yet.
2. Freeze the world at day 83: wiki_as_at_replay_start. The replay system is forbidden to see future observations.
3. Replay the final seven days serially — one event at a time: Reddit post, newsletter, original tweet, engagement updates, later GitHub repo.
4. Run controlled arrival-order variants: Reddit first, tweet first, aggregator first, primary delayed, engagement before technical corroboration.
5. Compare designs on the measures table — and let the architecture lose honestly.

Do not replay the final seven days against a wiki that already knows what happened during those seven days.

That is the future-leakage rule. Without it the system looks brilliant because it already knew Karpathy mattered and which signals survived. With it, you are testing discovery under realistic ignorance — the only ignorance that counts.

You do not need to replay the whole internet. Use grain on the corpus: event deltas for high-value cases, a handful of hand-labelled important stories, convincing noise, canonical-source-late cases, hot-then-dead cases, slow burns. Enough to evolve the design cheaply.

Perturbation, not decomposition

Look back at how a real design conversation advances. It is rarely “break the unknown into ten tasks.” It is outside products showing that clustering and momentum sensing already exist separately; wiki priors clarifying that the graph holds meaning while the queue holds unresolved attention; a worked Reddit/Karpathy example forcing the move from posts to signal cases; a recognition that serial state cannot be computed from a single graph snapshot.

Each important outside influence forces you to redraw the object. So the next move after a design card is not another general discussion. It is another collision: take ten real multi-source stories and force the queue, event by event, to choose ignore / attach / create / promote / merge / split / resolve / reopen. That produces a case-boundary contract faster than another abstract schema workshop.

The build boundary

There is a point where remaining questions require behaviour only an operating system can produce: serial arrival, uncertain case identity, delayed anchors, merge/split, observation targets, attention scheduling, interaction of event history and graph meaning. You can describe those forever. Eventually you need to watch them happen.

Two bad extremes: keep designing until the whole architecture feels settled, or build the full product and discover architecture through technical debt. The useful middle:

Build the smallest end-to-end system capable of falsifying the current design.

A design-probe vertical slice might include: append-only event log; bronze source objects; minimal wiki lookup; signal-case mutation ops; materialised queue; full decision trace; replay runner (wiki Git commit + queue checkpoint + event offset); a boring CLI. It does not need every social source, perfect influence scoring, or a polished briefing UI.

Early code is laboratory apparatus. The event log creates the experiment; the queue engine runs the hypothesis; the traces are observations; the replay harness repeats the experiment; the design document records what was learned. The implementation is allowed to be wrong because its job is to make the wrongness visible.

That resolves the false fight between design-first and build-first:

The design remains the source of truth, but it is empirically compiled — build early enough to learn, never let the code become the only place where the learning lives.

On open source: with expensive coding, a near-miss GitHub project was gold. With cheap AI coding, free code often includes an alien North Star — ontology, state model, success definition, historical compromises. Mine projects for compressed experience (connectors, schemas, failure modes, tests). Copy discoveries, not destiny. Regeneration around your own design is frequently cheaper than weeks of unbending someone else’s.

What to run on Monday

Two assets come out of a serious co-design session: the system under design, and a reusable method for evolving designs with AI when neither party can fully specify the answer. The second may travel further than the first.

Chat is not enough. Parallel essays are not enough. Regenerating code until it “morphs” without a harness is amnesia with a Git log. Race the designs against reality. Freeze the world so you cannot cheat. Keep the scoreboard.