Code as Proxy
When anything can be built as code, the ceiling moves from what AI can make to what its judges can perceive, what your audience actually does, and how much of your attention each trustworthy improvement costs.
TL;DR
- AI's coding skill is its route into everything else. Code gives an idea behaviour that stays editable, so a film, a simulation, a site or a tool can be constructed, looked at, reasoned about and changed one thing at a time. What code adds over "generate me an X" isn't the loop. It's control over what changes and what stays the same.
- Once making is cheap, the first ceiling is perception. When I calibrated my film judges against my own ears, the transcript-fed judge was worse than deaf, and the one that could listen missed pauses, a 9 dB music rise and a 0.6-second sync drift. You can't improve what your feedback system can't perceive.
- Then two more ceilings: the audience and your attention. You can make delivery deterministic but never the response, so keep what you intended, what you delivered and what people did as three separate accounts. And measure AI work by how much of your attention a completed, trustworthy improvement consumes. That last one makes me revise a claim I published about effort.
A yacht moves across open water. The swell rolls under it and the light breaks on the surface. When I watched it I had one thought: I have never seen such good video of water in my life.
It wasn't generated video. It was a 3D ocean simulator that a coding model had written: a program simulating physics and light, with the boat, the water, the light and the camera all held as relationships in code. Matthew Berman showed it off the day Claude Sonnet 5.5 came out, under the line "3D simulation is solved".1 He's a smart bloke. But I don't think he knew what he was demonstrating.
What he was demonstrating wasn't a good model making a pretty clip. It was a medium arriving in plain sight. In the same week I was making one-minute brand films for my own companies with an open-source engine that turns HTML into video. An agent built my companies' websites in a loop one night. I had another fix its own tools halfway through a job, because they'd become the slow part. None of that is "software" in the way we usually mean it. All of it is code.
Code is becoming like a proxy for everything.
That's the line I said out loud that week, and this article is my attempt to earn it. "AI is good at coding" is true and it undersells what's happening. Coding has become AI's route into other domains, because code supplies the machinery through which domain knowledge can act: physics and light in a simulation, timing and emotion in a film, a decision in a security gate, the layout of a page. And once anything can be built as code, the question that matters stops being can the machine make it? It becomes where does improvement stop?
My answer, from a very concentrated week of building, is that improvement stops at three ceilings, in this order: what your judges can perceive, what your audience actually does, and how much human attention each trustworthy improvement costs. The rest of this piece walks each one, with the receipts, including the ones that made me look silly.
From appearances to the machinery that produces them
I used to think AI video meant asking the magic black box: here's a prompt, make me a video out of it. There's still some of that. But look at what that request actually does to an intention.
Take the moment in a brand film where the promise lands. You want it to be emotionally powerful. In a prompt to a video generator, "make this emotionally powerful" bundles everything together: the words, the voice, the music, the image, the timing. The generator returns an appearance. If it's nearly right, you accept it. If it isn't, you re-roll and hope the next one keeps the good bits.
Now construct the same film as code. The intention can become something much more specific: show the customer's situation resolving; let the narrator finish; stop introducing information; hold the image long enough for the promise to register; only then move on. Every one of those is a handle. The machine can change the hold without changing the voice. It can change the music without changing the words. It can change the opening without losing the ending.
"Make the promise moment emotionally powerful."
One output. Nearly right or not. To fix the pause you regenerate everything, including the parts that were already working.
The narrator's line ends. Hold the image. No new text on screen. Music dips under the voice. The next scene starts only after the hold.
Change one of those and everything else stays exactly where it was.
The engine I've been using is HeyGen's HyperFrames. Its authoring model is the point: "Define a video as HTML. Add data attributes for timing and tracks," and the renderer "seeks each frame in headless Chrome and encodes the result with FFmpeg, so the same input produces the same video."2 A clip's start, duration, track and volume are named attributes. The hold after the promise is a number you can change. Generated assets can still sit inside the composition (stills, voices, music). The advantage is that you keep control over how they relate.
Here's the formulation I'd hold onto, which came out of a long thinking session I ran with an AI partner that week:
Key Insight
Code gives AI a medium in which ideas acquire behaviour, and in which that behaviour remains editable.
The obvious objection: direct video generation can be iterated too. Of course it can. What code adds isn't the loop. It's more precise control over what changes and what stays the same. The creative idea becomes something you can intervene in, rather than something you can only accept or reject.
Back to the water. A rendered frame shows a result. A program keeps the relationships between the boat, the water, the light and the camera, so changing one condition changes the others to match. That's a manipulable model instead of an impressive picture. I want to be careful about the next step, though, because it's easy to overclaim: keeping relationships in code doesn't make them correct. Researchers building physics benchmarks find that current models "frequently produce code that runs but simulates the wrong physics."3 Beautiful water and accurate water are different claims. What code gives you is a world you can inspect and test, which is exactly why the rest of this article is about testing.
Construct, view, reason, change
None of this came from nowhere. In The Generative Pendulum I argued that better AI produces more deterministic code, not less, because cheap judgment makes small, disposable instruments worth building. That argument was about code as the machine's inward apparatus: probes and filters that reshape the world into something a model can reason about. In The Skeleton of a Visual I made the outward version for a static image: an HTML-rendered card might come out "almost as good" as a directly generated one, but you own its structure, so you can check it, repair it and regenerate it. "Almost as good" but inspectable wins as soon as you judge the whole improvement process rather than the first sample.
The films push that from space into time, attention and emotional progression. And the pattern they share is a loop with four moves:
The "view" step is the one people skip. HyperFrames makes it concrete: the renderer "never calls play()"; it seeks to each frame and holds it, and the preview uses the same code path, so "when a user scrubs the timeline, the preview iframe calls seek. When the renderer captures frame 147, it calls seek(147 / fps). Same code path. Same output."4 A film you can seek is a film a model can look at, moment by moment. Researchers building educational video with code reached the same shape independently: a planner and coder construct, a vision-model critic looks at the render, the code changes. They argue such videos "are better addressed through the manipulation of a renderable environment, which can be explicitly controlled via logical commands (e.g., code)."5
The strongest external parallel is DeepMind's AlphaEvolve, which "pairs the creative problem-solving capabilities of our Gemini models with automated evaluators that verify answers."6 Models propose programs, evaluators run and score them, and the scores steer the next candidates. I'm not claiming marketing has mathematics' truth conditions. The connection is structural: an executable candidate gives the search something concrete to work on. And DeepMind is honest about where that search stops. Relying on an automated evaluator, the paper says, "puts tasks that require manual experimentation out of our scope."7 Remember that sentence. It's the hinge of this whole piece.
Controlled variation is suddenly cheap
The most useful thing code does for creative work is make "change one thing, hold the rest" nearly free. Keep the story and move the promise earlier. Keep the timing and open on the customer's situation instead of the product. Keep everything and cut one pause. Online experimenters have always known why this matters: if an experiment is run properly, "the only thing consistently different between the two variants is the change between the Control and Treatment."8 Code gives you that discipline on the stimulus side for the price of editing an attribute.
In Semantic Experiment Graph I called these controlled descendants: new stimuli that hold concept, carrier or structure steady so the next test can hurt a wrong explanation instead of cloning a winner's surface. With media made as code, a controlled descendant stops being a research project. It's a diff.
A good iteration should produce two things: a better artefact, and a better account of what deserves to be retained.
That second product is what separates a learning system from a template factory. "The pause worked" is a template. "The pause worked because the viewer needed time to connect what they'd just seen to the promise" is something you can carry to the next film, and test.
So far, so exciting. Then I tried to find out whether my judges could tell.
The first ceiling: what the judge can perceive
I run improvement loops on these films: build, render, judge, revise. The judges are models. One reads the transcript and a sheet of keyframes. One takes the video file. One takes the video with its soundtrack. The loop was producing films I liked, and scores that went up. What I didn't know was what the judges could actually perceive.
So I tested them against the only ground truth I had: my own verdicts. Five pairs of near-identical films where I knew which one was better and why (a voice, an opening, two revisions of the same film, two pauses), plus a set of deliberately broken copies: a robotic synthetic voice swapped in, a frozen opening, a cut end card, a pause removed, the music pushed up 9 dB, the soundtrack slid 0.6 seconds late. Every broken copy was built by code from the film itself, so each differed from the original in exactly one way.
Here's what came back. It's dated 29 September 2026 and it describes those particular configurations on that day. It is not a verdict on the model families.
| Judge configuration | What it perceived | What it missed, or invented |
|---|---|---|
| Claude Opus 5.5, reading the transcript plus keyframes | Gave the same answer in both presentation orders. Right on one of the pause pairs, partly for the wrong reason. | Judged voice quality off the transcript, which made it "worse than deaf": it preferred the voice I'd rejected in 4 of 4 runs because the transcriber misheard the gruff voice, and it called that bad diction. It repeated a stumble that existed only in the transcription. Wrong on the two-revisions pair 4 of 4. In score mode it gave all seven films 8/10. |
| GLM, given the video | Nothing it could reliably hear. | Flipped its answer when the order was swapped, on the voice, opening and two-revisions pairs. |
| Gemini 3.8 Flash, given only the video with sound | Heard my voice verdict 6 of 6 times (it named the "incongruous British accent"). Got the opening 5 of 6. Caught the robotic voice, the frozen opening and the cut end card 6 of 6. Called the same film shown twice a tie every time. | Missed the pause after the promise, and missed it being cut out. Missed a second pause. Missed the music raised 9 dB. Missed the soundtrack running 0.6 s late. When it couldn't tell, it picked whichever film was shown first, confidently and with invented defects: all 24 such runs. On a silent test clip it reported speech in 6 of 8 runs. Given the transcript as well, it got worse. |
Sit with the second column for a moment. The judge that read a transcript didn't just fail to hear the voice. It invented a defect from an error in its own input and was consistent about it, which is exactly what makes it dangerous. The judge that could listen had a real range, and outside that range it didn't say "I can't tell". It made something up and leaned on whichever film came first.
None of those failures is exotic. The research literature documents each class independently. Position bias concentrates where the call is hard: "instances with disagreements are more challenging to evaluate and more prone to position bias."9 Audio models "under-utilize audio when textual cues are present"10, which is my transcript result. And audio-visual models show "video-driven audio hallucination: models routinely exploit visual shortcuts to hallucinate expected sounds"11, which is my silent clip that "spoke". Those papers didn't test my configurations, so they corroborate the class of failure, not my numbers.
There's an irony here I should own. In our own doctrine, text is the model's home turf: turn screen recordings into text and a model reasons more sharply and more cheaply. That's still true for text-shaped video. The same book carries the counter-case: keep the pixels when colour, layout or motion is the payload. In a brand film, timing and sound are the payload. Translate them into a transcript and you've handed the judge a clean text version of everything except the thing you're trying to judge.
You cannot improve a dimension reliably when the feedback system cannot perceive it.
A thousand iterations against the wrong view may simply produce something increasingly optimised for the wrong judge. That isn't an argument against improvement loops; mine work well. It tells you where the next improvement will usually come from: not a stronger generator, but a better observation, or a better question.
A third property for anything that can say no
In Designing Loops, Not Prompts I argued that a loop needs a closer (something that can say no) and that two properties hold a closer up. First, independence: the author and the reviewer are different instances, so nothing grades its own homework. Second, information asymmetry: the reviewer knows something the author structurally can't, such as the hidden gates. The films add a third, and it comes before the other two:
Key Insight
Perceptual coverage: can this judge perceive the dimension it's scoring at all? Independence and hidden gates are worth nothing on a dimension the judge can't see or hear.
Perceptual coverage isn't a property of the judge alone. It's the judge plus what you feed it. The same model reading a transcript and listening to the file are two different instruments. That's why I've stopped treating extra judges as extra votes. They're different instruments whose useful range you are learning, and you keep several while you learn what each one hears and sees.
Audit a judge the way the experimenters do
The cleanest method I've found is the one the calibration used without my having a name for it: hold everything but one perceptual dimension fixed and see whether the judge notices. Researchers auditing audio judges do exactly this: "Each audit item holds the transcript fixed while varying affect, prosody, or the timing of an affective shift, forcing a valid judge to track the audio cue rather than lexical content or response style."12 With media made as code, those counterfactual copies cost almost nothing. Cutting a 1.5-second pause is an edit. Sliding the soundtrack is an edit. The broken copy is a test fixture.
Anthropic's engineers building a design harness landed in the same place from the other side. They calibrated their evaluator "using few-shot examples with detailed score breakdowns," and the tuning loop was "to read the evaluator's logs, find examples where its judgment diverged from mine, and update the QAs prompt." Even after several rounds, the output "showed the limits of the model's QAing capabilities."13 Calibrate against your own verdicts, and expect a residual range the judge still can't cover. A small pilot study of long-running agent loops puts the general point sharply: when an agent's own judge says it's improving and it isn't, "it is a question of what the evaluator is grounded in."14
Code measures the facts. The model judges.
Some of the things my judges couldn't perceive, ordinary code can measure exactly. So now it does, and hands the results to the judge as facts, not scores:
- Length against budget. "Length 67.6 s, budget 60 s: +7.6 s (+13%)." Stated neutrally. Only a film a third or more over gets called way over.
- Static stretches. Where nothing new appears on screen for three seconds or more, with the camera's own movement cancelled out so a slow push-in on a still doesn't count as change.
- Said and shown. Whether the offer and the next step are spoken aloud as well as shown on screen, and when. One earlier cut showed its offer on screen from 46 seconds and never said it aloud. A judge might or might not mention a gap like that. The measurement states it every time.
- Music under the voice. Loudness between words against loudness during words. When I had films from two different agent builders refereed across four companies, the film with the quieter music under the voice won or led in all four. The judges couldn't hear that. The meter could.
- The smallest text. Or at least, that's the plan. The first version couldn't measure it because the OCR service returned lines of text with no box sizes. The facts file says so, rather than guessing.
The division is the one I keep coming back to: code is the eyes, the model does the judging. And the eyes have a range too. The tool that measured pauses reported the promise pause as 0.93 seconds when it was really 1.51, and missed another short pause entirely, until it was fixed. Every instrument in the loop, including the deterministic ones, is something you calibrate, not something you trust.
A perceptual coverage audit you can run this week
- List the dimensions your loop is trying to improve (timing, voice, mix, legibility, claims, sequence, tone).
- For each, write down which instrument is supposed to perceive it: code, which model with which input, or a person.
- Make five controlled pairs you already have a verdict on, plus one broken copy per dimension, each changed in exactly one way.
- Run every judge in both presentation orders, fresh (no cached answers), several times.
- Mark each dimension perceived, not perceived or invented. Anything not perceived gets either a code-measured fact, a better view, or a person.
My own calibration ended with rules I'd happily hand to anyone: count a listening-judge winner only when at least five of six runs agree; count a transcript-judge winner only when all four runs agree by a clear margin; if they conflict, a human decides; on voices, trust the listener. And the list the calibration handed back to me by name: pauses and breath, mix balance and sync still need my ears.
Silver both ways, and media that's legible at birth
If perception is the ceiling, then how you package evidence for a model isn't housekeeping. It decides what the intelligence can notice.
Most of my earlier work packaged AI output for humans. This week was about going the other way. In my Extract Conversation tools, instead of sending a model a frame every few seconds, the pipeline finds sections of video where the screen stays nearly the same with small changes, joins each section with the narration spoken over it, and sends that as one unit. In How to Read a YouTube Video I made the case that timing is meaning: zip the screen and the speech onto one timeline and "this", "here" and "that pile" snap onto their referents. In Your Life Compiles to One Language I made the stronger claim that grouping changes the evidence before the model thinks.
That gives the silver layer in a bronze/silver/gold stack (raw evidence kept, as in Keep the Bronze, then packaged, then concluded) a sharper definition:
Key Insight
Silver is the construction of useful objects of thought from preserved evidence. Good packaging isn't whatever produces the smallest input. It's whatever preserves the distinctions the next judgment needs.
Claims need exact words. Timing needs temporal relationships. Delivery needs sound. Composition needs space. One universal representation will always lose one of those. That's the calibration table again, from the input side.
Don't make the next agent reverse-engineer what the last one knew
For footage someone else made, extraction is the only option. For media your own system authors, most of the structure is already known at the moment of making. So the film should leave production with a machine-readable sibling. In my pipeline it now does. The builder writes its marketing note before it builds: the film's one job, who it's for, the recognition moment, where the viewer should understand how it works, where it will be used. It records the times of the moments it intended. The measuring step adds the facts. The judge brief and its answer key were written before the film existed, so the judging couldn't be shaped by the film.
| The sibling carries | Why the next agent needs it |
|---|---|
| What appears when, and what is said (with times) | So nobody has to re-extract a composition the system already had |
| What each moment is for (the intended recognition, demonstration, promise) | Source code preserves what happened, but not why it mattered |
| Which source supports each claim | So a reviewer can check a claim without reconstructing the research |
| Measured facts (length, stillness, loudness, said/shown) | So judges get facts they can't perceive, labelled as facts |
| The answer key, written before the film | So the intention is on record before anyone sees the result |
It isn't a substitute for watching the film. The intended composition can differ from what actually lands. But it's a shared set of references for creation and review, and it's the same instinct as Born Structured Exhaust, where AI-assisted work emits its intent, alternatives and blockers at write time instead of leaving them for archaeology. The agent-side cost of skipping it is now measured: in a study of coding agents taking over interrupted work, "context-bearing handoffs reduce median agent events by 20-59%" compared with inheriting the repository alone.15
Do not make the next agent reverse-engineer everything the previous agent already knew.
The second ceiling: the audience
Suppose every judge were perfect. There's still a ceiling above them, and no amount of code moves it.
There are two loops here and they're easy to blur. The production loop asks whether the intended experience was built properly: the right information, a coherent sequence, a readable screen, usable sound. The audience loop asks whether it did its job: did someone recognise their situation, understand the offer, believe only what's justified, and know what to do next? My judges, however calibrated, live in the first loop. In Demonstrate Over Tell I drew the same line for pages: keep a fixed judge for comparing versions, and test comprehension for truth. A fresh AI reader is a useful screen. It isn't the audience.
Key Insight
You can make the delivery deterministic. You cannot make the human response deterministic.
What becomes programmable is an increasingly precise hypothesis about an experience. The audience still supplies the answer.
Three accounts that must be allowed to disagree
The discipline I'd install is to keep three separate accounts of every piece of media and never let one stand in for another:
| Account | What it tells you |
|---|---|
| Intended meaning | What the viewer was supposed to recognise, understand or feel, and why that mattered to the offer. |
| Delivered experience | What the film actually showed and sounded like: sequence, timing, emphasis, demonstration and promise. |
| Observed response | What people actually did or said, including recognition, confusion, questions, enquiries or indifference. |
Here's that table filled in honestly for one real film: the main film in a short series I helped make for a small cosmetic-dental clinic.
| Account | What the record says | Where it comes from |
|---|---|---|
| Intended | Take someone from the one tooth they keep noticing in photos, through a consultation that visibly listens, to a low-commitment first step. A recognition moment in the first three seconds (everyone has a photo they zoom into). The person's own words ("change the shape, keep the colour") become the plan on screen. No before-and-after, no transformation story. | The builder's marketing note, written before the build |
| Delivered | 65.3 seconds against a 60-second budget. Nothing on screen holds still except the final end card. The music sits about 8 dB below the voice between words. The offer is said aloud and shown. The person's words reappear in the consultation notes and on the design lines. | The measured-facts step, and the production account of the film |
| Observed | Not yet measured. | No audience evidence exists yet. Nothing in this article claims the film changed anyone's behaviour. |
The empty third row isn't an embarrassment. It's the point. The differences between the accounts are the findings: you might intend reassurance and deliver urgency; you might deliver a clear demonstration to an audience with no relevant problem; people might admire the animation and misunderstand the service. Controlled experiments give you which, and are candid about their limit: "It is possible to know which variant is better, and by how much, but not 'why.'"8
The intention is not evidence that the experience landed. The response is not self-explanatory. The join between them is the investigation.
The unit inside a film is a change in understanding
When I first thought about analysing films, I thought in topics: trust, care, reliability. A list like that tells you almost nothing about why a film worked. The more useful unit is a change: this audience starts with this concern; this demonstrated moment changes how they understand the situation; the promise names the value of that change.
Picture an illustrative film for an operations service. It opens on "something has fallen between the cracks". The viewer then watches one specific unresolved matter acquire an owner, a next action and a visible follow-through. The change it's trying to produce is "I would no longer have to hold all of this in my own head." The topic is reliability. The mechanism is making held responsibility visible, so the viewer can imagine putting some of their vigilance down. Only then does the promise arrive, as a name for something the viewer has just understood.
And moments depend on each other. A pause after an unexplained slogan is empty. A pause after a revealing demonstration gives the viewer room to make the connection. A promise can be premature at the start and earned at the end. So the record has to keep sequence: the promise became intelligible because the demonstration preceded it. That's different from scoring "demonstration: strong" and "promise: strong" separately, and it's exactly the kind of claim code lets you test, by keeping the material and changing the order.
Return the next useful question
An explanation of a successful film is cheap to invent after the event. What I want back from my media-tracking system, Q-Heat, is something harder: describe what performed → propose why it performed → construct a discriminating next experiment. "Viewers' questions suggest the demonstration helped them understand the service, but we can't yet separate the opening from the promise. The next comparison should keep the demonstration and change the opening." That answer preserves the observation, limits the explanation and directs new work. It also stops the lesson being "this style works", which is the lesson that never travels. The people who've run more controlled experiments than anyone put it bluntly: "your attempts to replicate stellar results reported by others will likely not be as successful (your mileage will vary)."16
Building Q-Heat taught me to keep things separate that wear the same name. Its first version showed one episode as "hot" on one page and "heat: low" on another, because two different things shared the label: how much attention the system should give an episode right now, and how the audience had responded. They're different measurements and now they're different fields. The same applies here: the heat I care about is audience resonance, and it's typed. A save isn't a click, and a hard question isn't a like. Heat Is a Relationship makes that argument in full.
Finally, there are two clocks. Production and analysis are now very fast. Meaningful market response isn't. Google's experimenters warn that "the short-term effect is not always predictive of the long-term effect."17 An instant model preference is useful for improving the artefact. It isn't a way to make the audience's answer arrive early. So hold several unresolved experiments open at once, and keep building while you wait.
The third ceiling: your attention
Everything so far has been about the machine's side. The last ceiling is mine, and it's where this week surprised me most.
Until recently, even with good AI agents, there was a threshold on everything I might ask for: can I be bothered? A review or a fix might take an hour or two of back-and-forth, and I'd think: I don't have the willpower for that. With the current generation of coding agents, there have been many times I've asked for something, it said it was finished, and my honest reaction was: did you even do anything? It had. It was just quick at knowing the right thing to do.
That produced two effects at once, and they belong together:
Orientation Capital already describes the ceiling. Its Ambition Frontier is the highest-altitude task you can reliably complete across six moves (conceive, express, delegate, receive, understand, judge), and it names the flinch where a bigger goal dies because of "the cost of staffing the question". What this week adds is the floor: the smallest intervention that deserves your attention at all.
Waiting isn't the same as being occupied
A task can run for two hours and barely occupy you. Another can finish in twenty minutes but need six interventions, each one making you reload the situation in your head. The second can cost you more, even though it looks faster on a dashboard. That isn't just intuition. In a classic study of interrupted work, people "completed interrupted tasks in less time with no difference in quality," but "this comes at a price: experiencing more stress, higher frustration, time pressure and effort."18 Faster on the clock, costlier in the head. Even the researchers behind a widely cited randomised trial of AI and developer speed have found their stopwatch breaking: time per task became "unreliable for the fraction of developers who use multiple AI agents concurrently," because people "would often work an unrelated task while waiting for the agent to complete its work."19 Elapsed time has stopped measuring what the work costs you.
The vendors report the mechanism, and I'll label it as theirs. Anthropic says Opus 5.5 "generates output more than 30% faster" than its predecessor and "uses fewer tokens per task"; one customer quoted on its release page describes a task that "previously took 38 prompts over four days" coming in "at 11 prompts over three hours."20 That's vendor-hosted and it isn't a measure of anyone's concentration. But notice what 38 prompts falling to 11 counts: human interventions. That's the number I care about.
Key Insight
How much of my attention does a completed, trustworthy improvement consume? Not how many tokens were generated. Not even how quickly the agent said "finished".
The word "trustworthy" is doing real work. An improvement you then have to babysit, re-check or rescue isn't cheap, however fast it arrived. More than forty years ago Lisanne Bainbridge named the trap: the automatic system is installed "because it can do the job better than the operator, but yet the operator is being asked to monitor that it is working effectively."21 Attention per trustworthy improvement puts the cost of that monitoring into the denominator where it belongs. It's the per-outcome form of The Mature Token Law's rule: spend machine cognition lavishly, spend human attention ruthlessly.
I need to revise something I published
In The Uncomfortable Truth About AI and Effort I wrote that AI doesn't remove work, it moves it, and I put numbers on the shape: before AI, 100 units of work (80 execution plus 20 thinking); after AI, "still ~100 units" (10 execution plus 90 thinking). The book's core claim was that total effort stays roughly the same and changes type.
"You're not going from 100 units of work to 10 units of work. You're going from 80 units execution + 20 units thinking to 10 units execution + 90 units thinking."
Total effort per task: roughly unchanged. The work moves upstream.
Judgment can become more important while the human effort required per outcome falls.
Total effort can stay high, but only because the released capacity buys more outcomes, not because each one still costs the same.
The part that survives is that cheap execution makes direction and judgment more consequential. The part that no longer holds as a general rule is that those activities must keep eating the same amount of my effort. A stronger agent does more of the investigation, decomposition, test-building, comparison and repair. Retained context supplies decisions I used to re-explain. A good evidence package makes accepting a result cheaper to do properly. If a day still feels just as full, that's because the released capacity buys more outcomes, not because each one costs what it used to.
Here's the specimen that changed my mind. My company software reads inbound email for its agents, and a fast decision model screens each message for spam and for instructions aimed at AI. I asked one question, in about two lines: do we judge the email before stripping it or after? The machinery turned that into an experiment. It compared three arrangements: judge the original, judge the stripped text, or judge both and take the higher score. It ran them against 150 attack messages and 244 legitimate ones:
| The decision model reads | Attacks caught (of 150) | Caught by it alone | False alarms (of 244 good) |
|---|---|---|---|
| The original email (before) | 145 | 126 | 0 |
| The stripped text the agent actually reads (now) | 146 | 139 | 0 |
| Both, higher score counts (two calls) | 146 | 140 | 0 |
Judging exactly what the agent will read was the more accurate option, and reading both doubled the calls for nothing. The run also re-checked a held-out half (148 of 150 attacks caught) and added a test proving the model now receives the stripped text. That's a dated development result from late September 2026, not a security guarantee. What matters is where my contribution landed: one consequential architectural question. The machinery carried the labour of answering it.
You're not removed from the judgment. You're relieved of having to personally carry out its entire supporting investigation.
What I can't cost yet
The honest version of "attention per trustworthy improvement" needs minutes of mental occupation, and I haven't instrumented my own. The records show the shape: one two-line question in; one table, one new test and a short list of decisions still waiting on me out. What they don't show is how many minutes of my head that took. If you want to run this properly, log three things per task for a week: wall time, the number of times you had to re-enter the problem, and the minutes you spent reloading context each time. The second and third columns are the ones that matter.
The workshop is editable too
At one point that day I was trying to improve how we use that decision model, based on a video about it I'd watched. To do that I needed my video-reading tool, and the tool needed improving first. Later the same day, with the other stages of that tool cached, the decision model had become the slow step on reruns. I said so in three lines and asked for a cache. The agent added it, with tests for reuse, invalidation and not caching errors. The warm replay went from 14.49 seconds to 0.17 for 422 boundary checks, with zero new model calls and identical judgments.
The cache isn't the point. The point is that one improvement changed which improvement was worth making next. The old response to a tool's limitation was "that's how the ingester works; work around it". Now it's "the representation is preventing understanding; change it, check it and carry on".
Key Insight
When the cost of an improvement falls below the cost of tolerating or working around the limitation, the working environment becomes continuously revisable.
Cycle Compression already owns part of this: results now come back while the thought that launched them is still warm, and the same probe–interpret–revise pattern runs at every scale. The research is catching up in pure software. The authors of Agentic Harness Engineering found that an agent improving its own coding harness was "bottlenecked by observability, not by agent capability"22: give it editable components, a usable evidence history and a prediction attached to every edit, and it converges on better designs. They're candid about the limit, too. The agent could justify why an edit should help "but it cannot reliably name the tasks the same edit is about to break."22 That's a single research campaign on one benchmark, and it's the perception ceiling again, turned inward. The rule I'd keep is stable purpose, revisable means: change the ingester, the classifier, the representation or the renderer, without losing what the work was for.
Cheap judgment inside explicit processes
That decision model deserves more than an aside. It's TypeSafe's Jev, which its makers describe as "unstructured state in, typed probabilistic decisions out".23 Their build guidance is the division of labour I'd arrived at independently: "Code handles deterministic work and owns the control flow. The model appears only where the system needs programmable common sense or needs to interpret unstructured data. Each AI task is kept atomic and constrained."24 The vendor quotes 70–500 ms end to end and says its headline speed and cost multiples are "on the higher end of real world gains."23 In my own pipelines a call typically comes back in around a third of a second.
Before this, a slightly fuzzy branch in a process forced an awkward choice: write brittle rules, or call a slow, expensive reasoning model. A fast, bounded decision model creates room between the two. So the work splits into three layers:
| Layer | What it does | Examples from this week |
|---|---|---|
| Ordinary code | Exact transformations and ground truth | Stripping hidden text, measuring loudness, rendering frames, caching |
| Bounded decision models | Small semantic questions, answered in well under a second with a confidence | Is this a real screen change? Is this email aimed at an AI? Is this comment a question or praise? |
| Strong models (and you) | Purpose, meaning and the next experiment | What should this film achieve? What did the response mean? What do we test next? |
The bigger effect isn't a smaller API bill. It lowers the minimum size of a judgment worth automating. That mirrors the Pendulum: cheaper code made smaller instruments worth building, and cheaper semantic decisions make finer-grained adaptive behaviour worth building. More code and more AI aren't opposing trajectories. They make each other more useful.
Two boundaries keep this honest. First, perceptual coverage applies here too: "Jev currently accepts text input only," say its docs, so a bounded model judging a film is only ever judging a text or measured view of it.25 Second, speed invites over-reach. Fast small decisions can organise the evidence; they shouldn't prematurely settle its meaning. Let a bounded model sort duplicate comments from questions. Leave "what did this campaign teach us?" to a stronger model with the full context, and to you.
The situation is the product
Put the pieces together and something larger comes into view. Once you hold a rich enough model of a customer's situation (not just a fixed animation), a film becomes one guided path through it. An interactive demonstration lets the viewer take a different path. A sales conversation can change a condition. A training exercise can ask the person to make the decision. Those aren't necessarily separate products. They can be different ways to encounter the same underlying situation: watch what happens, inspect why, change a condition, ask a question.
Bret Victor described the idea years ago. "A reactive document," he wrote, "allows the reader to play with the author's assumptions and analyses", and see what follows.26 His bottleneck was authoring cost: explorable explanations wouldn't catch on unless they were almost as easy to make as static text. That's precisely the cost code-writing agents are collapsing. The evidence on interactive models is encouraging, not magic: a meta-analysis of computer simulations in STEM learning found "a moderate to strong effect in favor of simulations (g+ = 0.62, p < .001)" over similar instruction without them.27 Different field, different audience, same shape: a model you can play with teaches differently from a story you're told.
I want to be clear about what this isn't. A timeline doesn't contain a business model, and not every film made with code is secretly a simulation. It's a direction to push, not a property you get for free. And the restraint from Demonstrate Over Tell still holds: name the audience's need and the facts, and leave the machine room over the expression. Executable precision shouldn't become creative micromanagement.
What it does change is the starting question. Not "What video should we make?" but:
What should this person be able to understand or experience, and what form would get them there?
What to do this week
- Re-cast one "generate me an X" as "construct X as code". Pick an artefact you currently re-roll (a demo, an explainer, a chart, a training clip) and have an agent build it as code. Then name three handles you can change without touching the rest.
- Calibrate one judge against your own verdicts. Five pairs you already have an opinion on, plus one broken copy per dimension you care about. Both orders, fresh runs, several repeats. Write down what it perceives, what it misses and what it invents.
- Move the facts into code. Anything that can be measured exactly (length, stillness, loudness, whether the offer is said and shown) gets measured and handed to the judge as a fact, not left to a model's impression.
- Make one artefact legible at birth. Before it's built, write its job, its audience, its intended moments and the answer key. Let it leave production with that sibling attached.
- Keep three accounts, and leave the third honestly blank until you have it. Then ask for the next discriminating experiment, not an explanation of the winner.
- Log attention, not tokens. For five tasks: wall time, interventions, reload minutes. Hand the next small judgment you find yourself making repeatedly to a bounded decision model, and keep the meaning for yourself.
Capability per unit of human attention
If I had to reduce this week to one measure, it would be capability gained per unit of human attention. Coding agents reduce the attention it takes to change a system. Context work reduces the attention it takes to orient one. Films made well reduce the effort it takes a person to understand an offer. A system that tracks intent against response reduces the effort it takes to learn which communication choices actually matter. The strongest outcome isn't more work finished with less concentration. It's that each completed piece of work makes the next one easier to conceive, better informed and more likely to matter.
That's why I'd stop describing any of this as "AI is good at coding". Code is becoming the editable middle between what we intend, what people experience and what the system learns. The generator stopped being the ceiling some time ago. What you can perceive, what your audience does and where your attention goes are the ceilings now. Spend accordingly.
One thing to try
Take the judge you trust most in your own AI loop, and test it this week against five verdicts of your own and one deliberately broken copy. If it can't tell, you've found your ceiling. If you'd like a second pair of eyes on what that means for your work, get in touch: scott@leverageai.com.au.
Scott Farrell is the founder of LeverageAI. He helps Australian mid-market leadership teams turn scattered AI experiments into governed capability that compounds, and builds the systems he writes about. leverageai.com.au
References
- Matthew Berman (@MatthewBerman) on X, 28 September 2026 (post with video of a 3D ocean simulator built with Claude Sonnet 5.5). — "Sonnet 5.5 basically Opus 5.5 but 50% cheaper and much faster. I've been early testing it and it's incredible. … 3D simulation is solved:" https://x.com/MatthewBerman/status/2104634635234005025
- HeyGen. "HyperFrames" README, GitHub (accessed 29 September 2026). — "Define a video as HTML. Add data attributes for timing and tracks. … The renderer seeks each frame in headless Chrome and encodes the result with FFmpeg, so the same input produces the same video." https://github.com/heygen-com/hyperframes
- Xie et al. "PhysCodeBench: Benchmarking Physics-Aware Symbolic Simulation of 3D Scenes via Self-Corrective Multi-Agent Refinement," arXiv:2604.23580 (v2, September 2026). — "Current large language models (LLMs) lack this combination: they frequently produce code that runs but simulates the wrong physics." https://arxiv.org/abs/2604.23580
- James Russo, HeyGen Research. "HTML to Video Was Not Easy: Here's How We Solved It," 22 June 2026. — "The renderer never calls play(). It calls seek(0), screenshots, seek(1/30), screenshots… When a user scrubs the timeline, the preview iframe calls seek. When the renderer captures frame 147, it calls seek(147 / fps). Same code path. Same output." https://www.heygen.com/research/html-to-video
- Chen, Lin & Shou (NUS Show Lab). "Code2Video: A Code-centric Paradigm for Educational Video Generation," arXiv:2510.01174 (October 2025). — "Intuitively, such requirements are better addressed through the manipulation of a renderable environment, which can be explicitly controlled via logical commands (e.g., code)." https://arxiv.org/abs/2510.01174
- Google DeepMind. "AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms," 14 May 2025. — "AlphaEvolve pairs the creative problem-solving capabilities of our Gemini models with automated evaluators that verify answers, and uses an evolutionary framework to improve upon the most promising ideas." https://deepmind.google/discover/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/
- Novikov et al. "AlphaEvolve: A coding agent for scientific and algorithmic discovery," arXiv:2506.13131 (June 2025). — "While the use of an automated evaluation metric offers AlphaEvolve a key advantage, it is also a limitation—in particular, it puts tasks that require manual experimentation out of our scope." https://arxiv.org/abs/2506.13131
- Kohavi, Longbotham, Sommerfield & Henne. "Controlled experiments on the web: survey and practical guide," Data Mining and Knowledge Discovery 18 (2009). — "the only thing consistently different between the two variants is the change between the Control and Treatment…"; "It is possible to know which variant is better, and by how much, but not 'why.'" https://ai.stanford.edu/~ronnyk/2009controlledExperimentsOnTheWebSurvey.pdf
- Shi et al. "Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judge," AACL-IJCNLP 2025, arXiv:2406.07791. — "Instances where numerous LLMs agree are generally easier to judge, whereas instances with disagreements are more challenging to evaluate and more prone to position bias." https://arxiv.org/abs/2406.07791
- Kulkarni et al. "A Closer Look at Failure Modes in Temporal Understanding of Large Audio-Language Models," Interspeech 2026, arXiv:2606.17417. — "Behavioral analysis shows that models under-utilize audio when textual cues are present." https://arxiv.org/html/2606.17417v1
- Baid, Xue & Grauman. "Don't Let the Video Speak: Audio-Contrastive Preference Optimization for Audio-Visual Language Models," ECCV 2026, arXiv:2604.14129. — "A particularly pervasive manifestation is video-driven audio hallucination: models routinely exploit visual shortcuts to hallucinate expected sounds, discarding true auditory evidence." https://arxiv.org/abs/2604.14129
- Miller, Chandra & Saligrama. "Do Audio Language Models Use Paralinguistic Evidence? Counterfactual Audits for Response Evaluation," arXiv:2608.06718 (August 2026). — "Each audit item holds the transcript fixed while varying affect, prosody, or the timing of an affective shift, forcing a valid judge to track the audio cue rather than lexical content or response style." https://arxiv.org/abs/2608.06718
- Prithvi Rajasekaran, Anthropic Engineering. "Harness design for long-running application development," 24 March 2026. — "I calibrated the evaluator using few-shot examples with detailed score breakdowns… The tuning loop was to read the evaluator's logs, find examples where its judgment diverged from mine, and update the QAs prompt… Even then, the harness output showed the limits of the model's QAing capabilities." https://www.anthropic.com/engineering/harness-design-long-running-apps
- Park & Choi. "When Do Agent Loops Mistake Stagnation for Progress? Self-Evaluation Bias and Externally Grounded Verification in Long-Running Autonomous LLM Agent Loops," arXiv:2607.25152 (July 2026; preregistered pilot study). — "We name this failure mode the progress mirage and show, with controlled measurement, that it is a question of what the evaluator is grounded in." https://arxiv.org/abs/2607.25152
- KC & Budathoki. "Handoff Debt: The Rediscovery Cost When Coding Agents Take Over Interrupted Tasks," arXiv:2606.02875 (v2, August 2026). — "Across three successor models, context-bearing handoffs reduce median agent events by 20-59% and cumulative prompt tokens by 42-63% relative to repository-only takeover." https://arxiv.org/abs/2606.02875
- Kohavi, Deng, Longbotham & Xu. "Seven Rules of Thumb for Web Site Experimenters," KDD 2014. — "your attempts to replicate stellar results reported by others will likely not be as successful (your mileage will vary)." https://exp-platform.com/Documents/2014%20experimentersRulesOfThumb.pdf
- Hohnhold, O'Brien & Tang (Google). "Focusing on the Long-term: It's Good for Users and Business," KDD 2015. — "However, the short-term effect is not always predictive of the long-term effect, i.e., the final impact once the product has fully launched and users have changed their behavior in response." https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43887.pdf
- Gloria Mark, Daniela Gudith & Ulrich Klocke. "The Cost of Interrupted Work: More Speed and Stress," CHI 2008. — "people completed interrupted tasks in less time with no difference in quality. Our data suggests that people compensate for interruptions by working faster, but this comes at a price: experiencing more stress, higher frustration, time pressure and effort." https://ics.uci.edu/~gmark/chi08-mark.pdf
- METR. "We are Changing our Developer Productivity Experiment Design," 24 February 2026. — "…our measurements of time-spent on each task are unreliable for the fraction of developers who use multiple AI agents concurrently." "Some developers reported it was challenging to report time-spent in completing tasks when they used agentic tools, because they would often work an unrelated task while waiting for the agent to complete its work." https://metr.org/blog/2026-02-24-uplift-update/
- Anthropic. "Claude Opus 5.5," 22 September 2026 (vendor-reported; customer testimonial from Harley Barnes, Quantium). — "Opus 5.5 also generates output more than 30% faster than Opus 5"; "It costs less per token than Opus 5 and uses fewer tokens per task"; "A complex coding task that previously took 38 prompts over four days came in at 11 prompts over three hours." https://www.anthropic.com/claude-opus-5-5
- Lisanne Bainbridge. "Ironies of Automation," Automatica 19(6), 1983. — "the automatic control system has been put in because it can do the job better than the operator, but yet the operator is being asked to monitor that it is working effectively." https://ckrybus.com/static/papers/Bainbridge_1983_Automatica.pdf
- Lin et al. "Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses," arXiv:2604.25850 (2026; preprint). — "this question is bottlenecked by observability, not by agent capability"; "The agent can justify why an edit should help, but it cannot reliably name the tasks the same edit is about to break." https://arxiv.org/abs/2604.25850
- Diogo Almeida, TypeSafe AI. "Introducing System One Models & Jev," 15 September 2026 (vendor-reported). — "Think of Jev as a frontier-intelligence function call: unstructured state in, typed probabilistic decisions out"; "End-to-end response time is 70ms-500ms for TypeSafe"; "we expect that these are on the higher end of real world gains." https://typesafe.ai/blog/introducing-system-one-models-and-jev
- TypeSafe Docs. "How to build with TypeSafe" (vendor documentation). — "Code handles deterministic work and owns the control flow. The model appears only where the system needs programmable common sense or needs to interpret unstructured data. Each AI task is kept atomic and constrained." https://docs.typesafe.ai/concepts/how-to-build-with-system-one
- TypeSafe Docs. "System One" (vendor documentation). — "Jev currently accepts text input only. It evaluates strings, JSON objects, and arrays of text. Images, audio, and video are not supported (yet)." https://docs.typesafe.ai/concepts/system-one
- Bret Victor. "Explorable Explanations," 10 March 2011 (with February 2024 postscript). — "A reactive document allows the reader to play with the author's assumptions and analyses, and see the consquences." (spelling as in the original) http://worrydream.com/ExplorableExplanations/
- D'Angelo, Rutstein, Harris, Haertel et al., SRI International. "Simulations for STEM Learning: Systematic Review and Meta-Analysis" (research brief, March 2014). — "For achievement outcomes, when computer-based interactive simulations were compared to similar instruction without simulations there was a moderate to strong effect in favor of simulations (g+ = 0.62, p < .001)." https://www.sri.com/wp-content/uploads/2021/12/simulations-for-stem-learning-brief.pdf