Field Note · Build Retrospective

Text Is the Model's Home Turf: A Field Note on the Pendulum Between Code and Judgment

We built an AI pipeline that turns long-form articles into social media. The interesting part was never any single component. It was the year we spent swinging the pendulum — between deterministic code and the model, between image and text — until each part was finally doing the kind of work it was actually good at.

By Scott Farrell · LeverageAI

Where we started: the image is the thing

It began with a small delight. We rendered a pull-quote from an article as a clean, branded image card — a screenshot of styled HTML — and the reaction was immediate: woah, that's cool, beautiful images straight out of the article. The image was the product. Everything else was scaffolding around making more of them.

That framing — the image is the thing — quietly shaped the next year of decisions. Most of the journey is the story of unlearning it. What follows is a candid retrospective: the order we figured things out in, the moves we got wrong first, and the meta-lesson underneath them — when to trust deterministic code, and when to trust the model.

Act I — Ask the model: "is this a good image?"

The first instinct was the obvious one. We had images; we wanted the good ones. So we asked a multimodal LLM to look at each card and score it: is this a good image for social media? One to ten. Keep the high scorers.

It worked well enough to be encouraging and badly enough to be instructive. The model would look at a branded card and rate it — but it was rating a picture, and a picture of text is a strange thing to ask a language model to judge. We didn't see the problem yet. We just saw numbers that were mostly sensible, and moved on.

The tweet came next, almost as an afterthought: alongside the image, generate a short post to carry it. An aside. A caption. That aside turned out to be the real product. We just didn't know it for several more moves.

Act II — The meaning jump: give the model the why

The captions were flat at first, because the model only had the quote. It would paraphrase the card — dead weight, since the card already shows the words. The leap came from changing what context we fed it: instead of the quote alone, we gave it the article's outline and the chapter the quote came from. Suddenly the model could see the quote as one move inside a larger argument. It stopped echoing and started surfacing the idea underneath — the consequence, the second-order effect, the uncomfortable implication the quote only hinted at.

That was the first big jump, and it set a pattern we'd repeat: quality came from giving the model the right representation to reason over, not from a cleverer instruction. Hold onto that sentence. It is the whole article in disguise.

Act III — Generate-and-judge: score by the post it could become

Then a reframe that, in hindsight, dissolved the original problem. We stopped asking "is this quote good?" and started asking the model to do the task and judge its own work: draft the tweet you would actually post, then rate how well that tweet — alongside the card — would land.

The score became a byproduct of attempting the work. This is "generate-and-judge," and it beats one-shot rating because the model can't anchor on a number and rationalise backward; it commits to a real artifact first, reasons about it, and scores last. We enforced exactly that order in the output schema — draft_tweet first, final_rating last — so the model's "thinking" genuinely precedes its verdict. That ordering isn't a quirk of our setup; it's the documented practice for LLM-as-a-judge, which is most reliable when the explanation is produced before the score rather than after it.¹

It also rhymes with a result we lean on whenever someone reaches for a bigger model instead of a better loop: in Andrew Ng's agentic-workflow work, GPT-3.5 scored about 48% on the HumanEval coding benchmark in one shot, but roughly 95% when wrapped in an iterate-and-reflect loop — surpassing GPT-4 alone.² Structure beats raw capability. Making the model do the task and check itself is the structure.

Something subtle had happened: the scorer and the writer became the same act. The "assessment" was now a real post, and we read its self-rating off the side. The text — the thing we'd treated as an aside — was now doing the heavy lifting.

Act IV — The image was the wrong thing to show the model

Here's the realization that turned the whole pipeline inside out.

We were still sending the rendered card image to the model so it could judge the "combo." But a language model reads overlapping, clipped, or mangled text just fine — it parses the characters and never notices that, visually, the card is broken. So the model was blind to render defects. Worse, in head-to-head tests the image added almost nothing to the ranking (rank correlation was ~identical with and without it), and when present it actually inflated junk — it would rescue a citation card or a half-rendered box that a human eye would reject at a glance.

The conclusion was uncomfortable but clean: for judging a card, the image is the wrong representation. Text is the model's home turf. The image's only real job was to tell us whether the render was any good — and a language model is precisely the wrong tool for that, because it's too forgiving of visual garbage.

So we did the thing that felt backwards and was right: we stopped sending the image to the scorer at all. Scoring went text-only. The image became a render artifact to be validated separately, by something that actually cares about pixels.

The proof arrived later, vividly. A card had its attribution line overlapping the quote (a stray negative margin from the source HTML). The vision model read "— Diana Hu, Y Combinator" perfectly and never flagged it. A deterministic OCR pass read the card and its word-coverage against the known text dropped to 63% — caught. The "dumber" tool was the more accurate one, because it was the right tool.

The meta-theme: the pendulum between deterministic and AI

This is the through-line of the entire project, and it's worth stating plainly because we swung the pendulum both directions, more than once:

• AI → deterministic. Judging whether a card rendered correctly started as an LLM vision task and moved to deterministic checks: a DOM probe (is it styled? does anything overflow?) plus OCR-coverage against the ground-truth text. Reliability and ground truth beat judgment here.
• deterministic → AI. Deciding which passages are interesting started as fourteen hand-written CSS-heuristic extractors and moved to an LLM. Judgment beats rules here, and the rules had become an endless game of whack-a-mole (every new article's styling needed a new heuristic: coloured left-bars, grey callout boxes, reference-chapter exclusions, dedup…).

The principle that emerged: use AI for nuance and judgment; use deterministic code for reliability and ground truth. The skill isn't picking a side once — it's knowing, for each sub-problem, which kind of problem it actually is, and being willing to move a component the other way when the evidence says so.

We've written before about why the deterministic side has to be structural, not pleading. In "AI Doesn't Fear Death," the rule is "can't beats shouldn't": don't ask the model nicely to behave, build a gate it cannot get past. The OCR-coverage check is exactly that — it doesn't ask the model whether the card is fine, it measures the card against text we already hold. As that piece puts it: prompts are manners; architecture is physics.

A deterministic check is more trustworthy when it compares against something known. OCR alone is noisy. OCR compared to the source text we already have is robust — we're not asking "what does this say?", we're asking "is the text we expect actually present, and is there foreign text that shouldn't be?"

That reframing turns a flaky tool into a reliable gate.

Image → text, wherever we can

The deeper pattern under "don't send the image" is this: modern AI can do images, but it is not as nuanced with images as it is with text. Its reasoning is sharpest over text representations. There's a concrete reason for that, and it isn't mysticism. Language models saw trillions of tokens of text and code in training and only a thin sliver of anything else; the model's fluency tracks what it has seen most. We've made this argument at length in "Why Code Execution Beats MCP" — the code corpus dwarfs the tool-schema corpus by something like fifty-million-to-one, so the model is reliable in code and shaky in formal schemas. The same logic explains why it's reliable over text and forgiving over pixels. The best interface for a language model is the one closest to its training distribution — and that's text.

So a recurring, high-leverage move was to convert image-shaped problems into text-shaped problems:

• Judge the card → judge the quote text + the tweet about it (text).
• Validate the render → OCR the card to text and compare to the source text (text vs text, deterministic).
• Even the OCR choice followed this logic. We put the heavy lifting on Apple's Vision framework running locally on an M4 — a deterministic, Neural-Engine OCR that reads clean rendered text (including italic serif that lighter OCR engines miss entirely) and hands back text, which is the representation everything downstream actually wants. This isn't an idiosyncratic choice; the wider field has landed in the same place — specialised OCR for accurate transcription, general vision-language models for semantic understanding, used as complements, not substitutes.³

Every time we turned a pixel question into a text question, the system got both cheaper and more accurate. That's not a coincidence — it's the model meeting us on its own ground.

The render-quality detour: fix it at the source

Validation taught us to detect bad renders. But detection invites a better question: why are they bad? The failures were almost all the source HTML's responsive styling misfiring when we screenshotted a single card out of a full-page layout:

• An attribution with a negative top margin that looks fine in the flowing page but overlaps the quote on an isolated card.
• A callout set to float right at ~45% width that, at our wide screenshot viewport, floated out of its container and got clipped — the screenshot clipping the wrong piece.

The first fixes were CSS overrides (kill the negative margin, kill floats). The real fix was upstream and almost embarrassingly simple: render the card at a narrow viewport (~700px), below the framework's responsive breakpoints. At that width the page renders in its single-column layout — none of the multi-column/float machinery fires, nothing escapes, and the screenshot clips exactly the right region. We stopped fighting the symptom and removed the cause.

And the validation gate itself got a second axis, because "is all the text there?" isn't the whole question. "Is there extra text too?" matters just as much — a mis-clipped card can contain the right words plus a neighbour's content. So the check became bidirectional: recall (is the quote present?) and precision (is anything foreign present?), both measured against the known card text. Two cheap, deterministic numbers that together describe "did this render correctly."

Act V — Let the AI pick the quotes (and the combinations)

With scoring text-only and rendering validated deterministically, the last bastion of brittle heuristics was selection itself — the fourteen extractors deciding what counts as a quotable passage. They were the original sin: deterministic code trying to find "interesting," which is exactly the judgment a model should own. Rules can match patterns; they can't recognise what's worth saying — so the heuristics were always going to lose to a model.

So we moved selection to the model. But the crucial design choice was what to show it. Not the rendered images (we'd learned that lesson). Not raw HTML (Tailwind-class noise drowns the signal). Not markdown (it loses the visual signal — a coloured callout box becomes plain text, but the box is part of why a passage is interesting). The answer was a denoised semantic DOM:

• strip the noise (scripts, styles, utility-class clutter, nav/footer),
• give every content scope a stable #id,
• annotate emphasis as a hint (blockquote, coloured box, left-bar, grey-bg),
• and send the whole chapter — drop nothing.

This is the idea we kept coming back to: a "text vision" of the document. Not a picture for the model to look at, but a clean, structured map for the model to read — the way a text model actually perceives a chapter at its best. The denoiser's only jobs are to clean and to label scopes; it makes no judgment about what's interesting. (We measured this: the denoised block list preserves 99.4–100% of the chapter's words. It curates nothing.) The model does all the discrimination.

If that move sounds familiar, it should. It is context engineering applied to a single document: context isn't capacity, it's attention. When we de-cluttered an agent's tool list from 47 tools to 8, first-try task completion jumped from ~60% to ~85% on the same model — not because it had more room, but because its attention stopped diffusing across noise. The denoised DOM does the same thing to a chapter: strip the Tailwind static so the model's attention lands on the content. And the framing — give the model a map before it starts — is the same control-surface idea we use in the Cognition Supply Chain: a map is not documentation; it's the thing that tells the model what exists and what matters.

Two more principles made it sing:

1. North-star prompting, not a checklist. We stopped enumerating "types" of good quotes and gave the model a single goal — "anything interesting enough that we could build a compelling tweet or post around it" — plus permission to favour recall, combine blocks into one card (a heading + a box, two related boxes), and overlap freely (the same line alone, and again inside a larger arc). Less prescription, more judgment. A checklist is a heuristic wearing a prompt's clothing; the north star is the judgment you actually want.
2. The model returns references, not text. It selects by #id. We then extract the text and HTML from the DOM by those ids. This keeps the scored text, the rendered card, and the validation gate referencing the exact same bytes — they can't drift, and the model never has to transcribe (the one thing it's worse at than judging). A selection is one or more whole scopes — never a fragment of a paragraph, because you can't screenshot half a paragraph. Judgment lives in the model; the renderable truth lives in the DOM.

The results were striking. On a single chapter the model returned ~20 thoughtful selections — singles, combinations, overlaps, with-or-without the heading — each justified against the north star in language that was genuinely social-media-literate ("the secretary line is standalone tweet gold," "if either agent dies the state survives — the one-sentence proof of the second axis"). And the regression test we cared most about: against the existing high-rated quotes, the new selector independently rediscovered 98% of them — while adding all the combinations and framings the heuristics could never produce. The single "miss" was a transitional setup sentence that arguably shouldn't be a card. The model wasn't just keeping up with the rules; it was exercising better taste than them.

We also learned the context could be lean: outline + the chapter's blocks was as good as outline + the entire article, at half the tokens. The chapter is already in front of the model; the outline gives it the frame; the rest was redundant.

The economics, because they shaped the design

None of this is free, and cost pressure repeatedly improved the architecture:

• Scoring on a premium model quietly consumed our plan quota, which forced a real cost/quality study: a benchmark harness (with a persistent, keyed result store so we never re-pay for a test we've run) comparing models on rank-correlation, cache behaviour, and tweet quality.
• That study taught us the cost lives in input tokens and caching, not in the model's cleverness — a large, constant context prefix that's reused across hundreds of quotes should be cached, and the cards' images are a rounding error next to it.
• It also pushed work onto the right machines: plan-funded routing for the heavy judgment, a free text-only model for the recall-favouring selection pass, a local Mac for OCR. The cheapest configuration was usually also the most accurate one, because cheapness came from using each tool on its home ground.

Cost wasn't a constraint we suffered; it was a design force that kept pointing us toward "use the right representation on the right substrate." It's the same instinct we describe in "The Index Is the Data": pre-computed, cached structure is capital; recomputing on every call is an operating expense that never stops. Cache the constant prefix once, and every subsequent quote rides for almost nothing.

The recursive twist

Step back and the whole thing has a pleasing recursion. The source articles are themselves written with heavy AI assistance — drafted, researched, structured by models. And now the pipeline repurposes them aggressively, with AI, into other media — pull-quotes, cards, tweets, LinkedIn posts. AI-authored content, AI-repurposed, with deterministic rails where ground truth matters.

There's a fitting resonance in the test article we kept using — "Designing Loops, Not Prompts." Its thesis is that the unit of AI work has moved from the prompt to the loop: a small system that finds the work, hands it out, checks it, records what was done, and decides the next thing — while you watch instead of type. That is almost exactly what this pipeline became. We stopped writing prompts to extract quotes and started designing a loop that selects, renders, validates, scores, and posts — with the human holding the north star and the deterministic rails, and the AI holding the judgment.

That piece makes a sharper point we'd earned the hard way: the part that makes a loop trustworthy isn't the number of agents or the tokens burned — it's the verify phase, an adversarial closer that checks each output against something real. A loop with five agents and a real test gate beats a fifty-five-agent tournament that's only scored against itself. The token-maxing is the theatre; the apparatus is the asset. Our bidirectional OCR-vs-source gate is that verify phase. We built the thing the article was about, to repurpose the article itself.

The other recursion: AI built this, too

The product uses AI. But so did the building of it — and that turns out to be the deeper recursion. The AI in this project wore four hats, not one:

1. Analysis (LLM over text). Scoring quotes, drafting the tweets and LinkedIn posts, selecting passages, judging which framings land — the model reasoning over text, which is its sweet spot. This is the part everyone pictures when they say "AI in the loop."
2. Writing the deterministic code. The denoiser, the render check, the narrow-viewport fix, the Apple Vision service, the CSV plumbing, the auto-poster's selection math — the AI wrote the non-AI parts too. The boring, reliable rails that the judgment runs on were themselves AI-authored. The system isn't "AI where it's smart, hand-written where it's safe"; it's AI on both sides of that line, used differently on each.
3. Recommendations — i.e. design. This is the one people underweight. Several of the bigger architectural moves came from the AI as a suggestion, not from a human spec. The "don't let the model judge the image — OCR it and compare to the known text" reframe was an AI recommendation. So was a lot of the "convert the pixel problem to a text problem" instinct. Other big moves came from the human: "move quote selection to an LLM that breaks a chapter into interesting passages" was a human call. And some of the most important decisions were the human holding a line against the AI's natural drift — most of all "send the whole chapter to the model; don't pre-select what's interesting in code" — which kept the design honest when the AI kept wanting to be helpful by filtering first. Design, it turns out, is a conversation: the AI proposes, the human disposes, and either can originate the idea that changes everything.
4. Testing and verification. Smoke tests before integration; wiring a new component in and running it end-to-end; checking the new approach back against the last run (the coverage test that proved LLM selection rediscovered 98% of the known-good quotes before we trusted it); benchmark harnesses with persistent result stores; head-to-head comparisons of a deterministic check versus an LLM decision to decide which tool owned which job; retries and back-offs around flaky model calls; A/B-ing the context to see what actually mattered. The AI didn't just produce the system — it produced the evidence that each choice was right, and was willing to be proven wrong by it.

The honest summary of the collaboration: analysis, code, and recommendations — which together are just "design" — plus the testing that keeps design accountable. The human's irreplaceable contributions were taste (what "interesting" and "lands" actually mean), direction (the north star, the lines held), and judgment about the judgment (deciding when to trust the model and when to trust the deterministic gate). That boundary is the one we describe as the cognitive exoskeleton: the AI saturates the pre-work; the human owns the judgment call. The advanced move isn't obeying the model — it's interrogating it, and knowing which of its suggestions to keep.

So the recursion goes one level deeper than "AI writes the articles and AI repurposes them." It's: AI helped design and build the loop — including the parts that aren't AI — that repurposes the AI-written articles, with a human holding the few lines that mattered. The unit of engineering really has moved from the prompt to the loop; this is what it looks like when the loop helps build itself.

And then one more turn — the one you're standing inside as you read this. A human/AI team wrote this article, about AI writing recursive code and the loops that run it. Its own production is one of those loops: part of the build is an AI agent that researches our back-catalogue agentically — reading an AI-built wiki of our past articles and running its own internet searches — then drafts from what it finds while a human holds the through-line and the few lines that matter. The essay about the recursion is itself a product of the recursion. It really is loops nearly all the way down — and the one thing that doesn't recurse is the judgment about which loop to trust.

And it won't stop at the writing. The moment this piece ships, it feeds the very machinery it describes. It gets indexed twice — into a RAG store (deterministic chunking, then AI-synthesised summaries and augmentation) and into the agentic wiki an AI just searched to research this very article. Then it enters the quote pipeline — passages selected, cards rendered, validated, scored, and posted to social — which is, of course, the exact software this article is about. The loop we built to repurpose our writing is about to repurpose the piece that describes it.

Principles, distilled

A working list — the things we'd tell ourselves at the start:

1. Use AI for judgment, deterministic code for ground truth. And be willing to move a component either direction when the evidence says so. The pendulum is the method.
2. Text is the model's home turf. Convert image-shaped problems into text-shaped problems; you'll get cheaper and more accurate at the same time.
3. The image isn't the product. It's a render artifact. The text — the post — is the product. Validate the artifact deterministically; judge the product with the model.
4. Validate against what you know. A check that compares output to ground truth (OCR vs source text, both directions: present and not-foreign) beats a check that judges in a vacuum.
5. Prevent at the source over detect-and-patch. The narrow-viewport render fix removed a whole class of failures the validator was catching.
6. Show the model a map, not a picture. A denoised, id-tagged, emphasis-annotated DOM is "text vision" — how a language model best perceives a document. Clean and label; don't curate.
7. Generate-and-judge. Make the model do the task and rate its own work; the score is a byproduct of real effort, and the ordering (work first, verdict last) matters.
8. North-star prompts over checklists. Give the goal and the permissions (combine, overlap, recall-favour); trust the judgment. Prescription is the heuristic trap in prose form.
9. Reference, don't transcribe. Let the model select by id; extract the bytes from the DOM. Judgment in the model, truth in the source — and they never drift.
10. Cost is a design force. Cheapness usually coincides with correctness, because both come from using the right tool on its home ground (cache the constant prefix; free model for the cheap pass; local OCR for pixels).
11. AI is a design collaborator, not just a runtime component. It does the analysis, writes the deterministic code, and proposes architecture — which together is design — then tests its own choices (smoke tests, regression against the last run, deterministic-vs-LLM bake-offs, back-offs on flaky calls). The human's irreplaceable part is taste, direction, and the few lines worth holding ("send the whole chapter; don't pre-select"). Either side can originate the idea that changes everything; keep the conversation honest with evidence.

The interesting part was never any single component. It was the evolution — the repeated, evidence-driven swing between code and model, between image and text, until each part of the system was finally doing the kind of work it was actually good at.