A LeverageAI Field Guide

The Agent’s Retina: Perceptual Engineering

How agents forge disposable eyes — SQL, grep, regex probes — to compress unreadable reality into a textual sensorium tuned between blinding and starving.

An agent’s hidden core problem is engineering how the world appears to itself. It forges disposable deterministic instruments to make the next cognitive step possible — and the freakish capability lives in the oscillation where AI narrows the question and determinism narrows the world.

Epistemic tool forging. The resolution axis. “Eyeball it” as a real instruction primitive. Text as the cognitive membrane.

The argument in three lines

  • Forge eyes. Disposable SQL/grep/regex probes compress unreadable stores into patterns the model can hold.
  • Tune resolution. Too much world blinds; too little starves. Useful perception is middle-band evidence.
  • Eyeball when ready. “Review” re-triggers probes. “Eyeball it” demands semantic judgment.

Scott Farrell · LeverageAI

01
Part I · How Agents See

The Hidden Core Problem

An agent does not “look at” a thirteen-gigabyte database. It engineers a view — and that craft is the hidden core of agent capability.

TL;DR

  • Raw reality at machine scale is cognitively useless to a model. Perception has to be constructed.
  • The chain is world → probe → representation → context → cognition — not “open the database and see.”
  • Treat representation construction as first-class agent work. That is perceptual engineering.

If you have ever sat beside a coding agent for a few minutes while it digs through Postgres, you have seen something product demos almost never name. It throws a shell script you would not have written. It invents SQL you cannot fully parse. It stacks a grep, a regex, a file hunt. It enumerates. It retries. After a handful of runs, it surfaces a shape of the data that no one on the team would have scoped as a three-week discovery project. You are not watching “a chatbot with a database plugin.” You are watching an intelligence manufacture a way to see.

That distinction is the load-bearing claim of this ebook. An agent’s hidden core problem is not merely choosing the next action. It is engineering how the world appears to itself.

How do I make the relevant part of reality legible to myself?

Why raw reality fails as input

Give a frontier model access to a multi-gigabyte message store and ask it what is “in there.” Without intermediate work, the honest answer is: nothing useful. The world is still sitting in tables, pages, and indexes. The model’s working memory is a context window — a bounded attention budget, not a magic mirror. Industry context-engineering guidance puts this bluntly: models draw on a limited attention budget when parsing large volumes of tokens, so the craft is curating the right set of tokens, not dumping everything that exists.1

So the agent does not “look at” the database. It forges a probe. The probe returns a representation. The representation enters context. Only then does cognition get a purchase. The chain is almost boring once you write it down — and almost always ignored when people talk about “giving the agent tools.”

WORLD
  ↓
SENSOR / PROBE
  ↓
REPRESENTATION
  ↓
CONTEXT
  ↓
COGNITION

Perception is a pipeline the agent builds under pressure — not a free gift of multimodality.

What the incumbent story gets wrong

The market story still leans on three substitutes for this craft. First: more tools on a menu. Second: bigger context windows stuffed with raw dumps. Third: better multimodal vision as if pixels alone solved legibility. Each can help. None of them is the core motion you watch when an agent turns an unreadable store into a dozen patterns it can reason about.

Fixed tool menus are especially seductive. They look like progress because the spreadsheet of integrations gets longer. But benchmarks of agentic tool use keep reminding us that a larger action space is not free — performance can degrade as the haystack of tools grows.2 A purpose-built SQL filter forged for this question often beats browsing a catalogue of generic connectors. The instrument is temporary. The legibility is the point.

What this ebook owns (and refuses)

We are writing a theory of agent perception: the sensorium, the resolution axis between too much world and too little, the forge-the-eye move, and the instruction primitive “eyeball it.” We are not writing the sibling doctrines that keep bleeding into the same conversations. Capability synthesis for action — building drivers that change the world as a product — is a different ownership. Compile-time intermediate representations and ingestion-time IR belong elsewhere. Full modality-conversion pipelines (speech plus OCR plus timeline merge) have their own field guide. Here the question is narrower and sharper: how does an agent make reality legible at runtime, and how do you instruct it to look?

That question has a name. Perceptual engineering is the runtime construction of representations that make the next cognitive step possible. The chapters that follow unpack the mechanism (epistemic tool forging), the oscillation that makes it freakishly strong (AI narrows the question; determinism narrows the world), a full Postgres archaeology trace, the resolution axis, the eyeball primitive, and the textual membrane everything has to cross.

Key takeaways

  • Raw machine-scale reality is not a model input; a constructed view is.
  • Representation construction is first-class agent activity, not a side effect of tool use.
  • If you only optimise models, menus, and context size, you will miss the craft that actually produces insight.
02
Part I · How Agents See

Epistemic Tool Forging

The agent does not merely have tools. It forges a disposable instrument for the question it has right now — then throws the instrument away when the question changes.

TL;DR

  • Epistemic tool forging: manufacture a probe that makes an otherwise unreadable world cognitively accessible.
  • Compression is the point: multi-gigabyte mess → structured samples → a handful of patterns the model can hold.
  • Code is cheap enough to be pure cognition apparatus — not a product that needs a roadmap.

Here is the cleanest name we have found for the motion inside the rabbit hole: epistemic tool forging. The agent faces a question it cannot answer by staring at raw state. So it builds something that measures the world in a way that serves this question. SQL. A regex. A Python reducer. A temporary table. A sample harness. The instrument is not sacred. It is an eye grown for one look.

Grew an eye. Looked. Didn’t like the view. Grew a different eye.

The forge loop

Write the loop without romance and it looks like automated scientific method aimed at computers and data:

QUESTION
   ↓
I can't perceive the answer directly
   ↓
FORGE PROBE
   ↓
MEASURE WORLD
   ↓
INTERPRET
   ↓
NEW QUESTION
   ↓
FORGE NEXT PROBE

That is not “tool use” in the sense of selecting from a fixed palette the product team shipped. It is closer to a field biologist improvising a trap for a species they have not seen yet. The first trap is wrong. The second is less wrong. By the fifth measurement cycle, the organism of the data starts to show its shape.

Thirteen gigabytes is not a thought

Take the lived shape from a real archaeology session. A large database sits on disk — on the order of thirteen gigabytes of stored history. As an object of cognition, that number is almost a joke. The model cannot attend to thirteen gigabytes. So it forges a query. Suddenly it can see a few thousand rows. Still messy. It manufactures a filter: strip newsletters, group by sender, count subjects, sample bodies. Now it sees on the order of a dozen patterns. Then the language model does the only thing language models are truly for: it recognises. “Oh. That’s the structure.”

Notice what the deterministic code did. It did not “solve the problem.” It reshaped the world into a representation the intelligence can reason about. Messy reality in; compressed signal out; cognition on the signal; new question; new eye. That is epistemic tool forging. Chapter 4 walks the same trace as a full narrative. Here the point is the mechanism itself.

Disposable instruments, not product software

Software culture still wants every script to become a maintained asset. Epistemic probes reverse that instinct. The SQL does not need a product manager. The one-off Python does not need a deprecation policy. The regex that killed newsletter contamination can die when the contamination is gone. Code became cheap enough — generate, run, learn, delete — that instruments can be pure cognition apparatus.

This sits next to a broader belief we have argued elsewhere: code is not always the durable asset. Sometimes the durable thing is the understanding, the kernel, the next question. Here the claim is sharper and local: sometimes the software is not the output at all. Sometimes the software is the eye you needed for ten minutes.

Industry is edging toward the same shape from a different door. Guidance on code execution for agents emphasises processing and filtering data in an execution environment before results enter the model — so intermediate mess never spends the attention budget.3 They are solving token economics. You can read the same pattern as perceptual engineering: determinism narrows the world until the retina is usable.

Re-forging is not failure

Operators sometimes flinch when the first query is wrong — as if the agent wasted a step. Wrong. The wrong view is evidence. “Mostly newsletters” is a successful measurement even when it is not the desired end state. The next forge is smarter because the last look existed. Epistemic tool forging is a loop, not a single perfect instrument chosen in advance.

That is why watching an agent for five SQL runs feels like magic if you expect waterfall design, and like ordinary scientific method if you expect contact with reality. Nobody could have written the final pipeline on day zero without the intermediate wrong eyes. The intermediate wrong eyes are the design process.

Key takeaways

  • Forge an instrument for the question you have now; do not wait for a permanent tool catalogue.
  • Compression into legible patterns is the job of the probe; recognition is the job of the model.
  • Disposable cognition apparatus is a feature. Re-forging after a bad view is progress.
03
Part I · How Agents See

AI Narrows the Question. Determinism Narrows the World.

The freakish capability is not AI or deterministic code. It is the tight oscillation between them — and the workflow is the trace that oscillation leaves behind.

TL;DR

  • Refuse the binary. Capability lives in the ping-pong.
  • AI narrows the question. Determinism narrows the world. That is the sharpest line in the doctrine.
  • From outside it looks like a pipeline. From inside the agent is just thinking. The workflow precipitates from contact with reality.

Most arguments about agents still frame a false choice. Should we use AI? Or should we use deterministic software? That question is how you build mediocre systems. The sessions that look like magic sit somewhere else entirely.

The ping-pong

Watch the micro-breath inside a good archaeology run:

AI          → decides what needs measuring
DETERMINISM → measures it exactly
AI          → interprets what the measurement means
DETERMINISM → reshapes / filters / enumerates
AI          → recognises an emerging pattern
DETERMINISM → tests the pattern at scale
AI          → changes the hypothesis

Neither side is enough. Pure LLM without contact produces plausible essays about your database. Pure deterministic output produces row counts and result sets that sit there like untranslated tablets. Together you get something that sounds like intelligence because it is intelligence in contact with reality: sampled thousands of rows, found them dominated by automated senders, excluded them, clustered the human remainder, and opened a new question about chronology and provenance.

AI narrows the question. Determinism narrows the world.

That sentence is the spine of this chapter. The model is not reliable as a bulk counter. Deterministic code is not reliable as a meaning-maker. Split the jobs on purpose. Push toward deterministic tools when you need exact narrowing. Push toward the model when you need a better question. Hand-in-glove, inhale and exhale, over and over until the next cognitive step is obvious.

It does not experience itself as a pipeline

Here is the part that rewires how you design systems. From the outside, an observer can draw boxes:

SQL → AI → SQL → AI → regex → AI → Python → AI

and congratulate themselves on a hybrid neuro-symbolic architecture. Meanwhile the agent is effectively thinking: too many newsletters; filter them; better; what is that; count those; oh. It does not experience itself as a component in an AI pipeline. It is just thinking.

Historically we designed systems the other way around. An architect anticipated the topology. LLM step one, SQL step two, LLM step three — fixed boxes, fixed edges. In the agent case, the topology emerges from the cognition. Thought, measurement, thought, different measurement, thought, transform, look again. The pipeline is an afterimage. If you force a full topology before contact with reality, you often invent the wrong boxes.

Industry is already half-describing this

You do not need our vocabulary to see the shape in public engineering notes. Anthropic describes agents as systems that dynamically direct their own tool usage across many turns on open-ended problems — not single-shot completions.4 Their work on code execution with tool protocols stresses filtering and transforming data outside the model so only the reduced product spends context.3 That is deterministic narrowing of the world, in production language.

What those write-ups under-name — and what operators feel in their bones — is the reciprocal half: the model’s job is not only to call tools but to re-ask better questions after each return. Without that half, you get eternal SELECT loops. Without the deterministic half, you get eloquent hallucination over a thin sample. The freakish zone is the oscillation.

Design implication

If the workflow precipitates from contact with reality, your job as operator shifts. You still set goals, done-tests, and boundaries. You do not pretend you can pre-draw every SQL step of an archaeology you have not begun. You build an environment where forging is allowed, where probes are cheap, where wrong views are cheap, and where you can interrupt with a better instruction primitive when the agent gets stuck measuring forever (Chapter 6).

We have written about the deterministic–AI pendulum as pipeline design advice — put judgment on the model substrate and ground truth on deterministic rails. This chapter is the same pendulum at the resolution of seconds: not a planned two-box architecture, but a living respiration that leaves a pipeline-shaped fossil behind.

Key takeaways

  • Capability is the oscillation: question-narrowing and world-narrowing in alternation.
  • The agent is thinking, not role-playing a box in your diagram.
  • Prefer environments that let workflows precipitate over cosplay topologies designed before contact.
04
Part II · One Full Trace

Postgres Archaeology: Growing Eyes in the Dark

One end-to-end forge → probe → interpret → re-forge trace — the flagship proof that perceptual engineering is a practice, not a slogan.

TL;DR

  • A multi-gigabyte store is opaque until the agent grows eyes for it.
  • Wrong views (newsletter flood) are successful measurements that teach the next forge.
  • Nobody pre-designed the final sequence. The sequence is the fossil of thinking under contact.

This chapter is the flagship. The earlier chapters named the mechanism; this one walks a real shape of session you can almost feel if you have ever watched a coding agent with shell and database access. The operator in the source material is cheerfully blunt about SQL skill: not a DBA, not a query artist. The agent does not care. It is not performing SQL for a human audience. It is manufacturing optics.

Setup

There is a large Postgres-backed store — on the order of thirteen gigabytes of historical material in the lived case — and a human goal that is not “migrate everything elegantly.” The goal is closer to archaeology: what is actually in here that matters? Which threads look like real staff discussion rather than automated noise? What can be recovered as provenance for later work? Traditional project thinking reaches for discovery phases and specialist migration. The agent reaches for a probe.

The trace

1. First eye: too much mess

The first SQL is broad. It returns a flood of rows — thousands, messy, mixed. As cognition, this is almost as bad as the raw thirteen gigabytes. The representation entered context, but the resolution is wrong. The model is not stupid for failing to “just know.” It is correctly blinded by a bad retina.

2. Second eye: structure by sender

So it re-forges. Group by sender. Enumerate classes. Count. Suddenly a pattern appears that no dashboard was commissioned to show: automated newsletter and marketing identities dominate the volume. That is not the desired end state. It is a successful look. The view was wrong for the human goal and exactly right as measurement.

3. Third eye: exclude the automated class

New probe: filters that push newsletters and similar automated sources out of the result. The remaining set is smaller. Human-looking senders start to dominate. The agent is still not “done.” It has merely moved from blinding toward a middle band where semantic texture might exist.

4. Fourth eye: sample bodies, not only counts

Counts and sender lists can still starve cognition. The agent samples message bodies, trims them, and sometimes invents a narrow projection — date, sender, subject, first few hundred characters. Now the model can read. This is the hinge where deterministic narrowing hands the baton to semantic judgment. You can almost hear the operator’s later instruction vocabulary forming: do not just measure forever — eyeball the sample.

5. Fifth eye: chronology and identity clusters

With a human-looking subset on the table, the next questions become chronological and social. Which identities cluster? What conversations look like project provenance rather than noise? After on the order of five SQL-led cycles in the lived telling, the agent is holding something like a dozen patterns — a compressed sensorium — and can speak intelligently about what the store contains. That is the payoff. Not a pre-built “mail intelligence product.” A temporary stack of eyes.

~13 GB database
      ↓  forge SQL probe
thousands of messy rows
      ↓  group / enumerate senders
newsletter-dominated view
      ↓  exclude automated classes
human-looking subset
      ↓  sample bodies + trim
representative evidence
      ↓  cluster / chronology
~12 patterns the model can reason over
      ↓  new question → re-forge

Forge → probe → interpret → re-forge. The numbers are case texture from practice, not industry statistics.

Ugly intermediates still serve cognition

In adjacent work on the same sprawling recovery, the agent sometimes chooses paths a tasteful engineer would reject — extracting into intermediate files large enough to make a human wince, then transforming again. The aesthetic offence is real. The cognitive service can still be real. If the world loop closes and the next representation is better, the intermediate was apparatus, not the product. Perceptual engineering optimises for legibility under a goal, not for a portfolio of elegant ETL diagrams.

What you could not have shipped on day zero

This is the part that should make product thinkers uncomfortable in a useful way. There was no way to write, in advance, the exact software that would have produced that archaeology in one shot — not without already knowing that newsletters dominated, which senders mattered, and which body samples carried signal. The knowledge was downstream of the wrong eyes. Traditional automation assumes a connector and a known schema of meaning. The agent assumes a goal and the right to forge.

It is not that somebody built a brilliant mail tool. It is that an agent repeatedly grew the organs it needed until the store became thinkable.

Keep the scope honest. This chapter is about perception — instruments that make reality legible. Extreme stories about building readers for dead formats can tempt you into a different doctrine: runtime capability synthesis for action. That ownership lives elsewhere. Here the NSF-grade extremes are only a reminder that when the eye does not exist, the agent may build one. The Postgres session is enough to prove the ordinary case: SQL, grep, regex, sample, interpret, repeat.

Industry descriptions of agents already assume multi-turn tool direction on open-ended problems rather than single-shot answers.4 The archaeology trace is that pattern made concrete: each turn is not a chat reply — it is a forged measurement, an interpretation, and a better question. Code-execution guidance that keeps intermediate mess outside the model is the industrial twin of the same motion.3

Key takeaways

  • Trace one full loop out loud; you will start seeing it in every good agent session.
  • Wrong views are fuel. Newsletter dominance was a successful measurement.
  • The final sequence was not designed as a product pipeline. It precipitated from contact.
05
Part III · Operating the Retina

The Resolution Axis: Between Blinding and Starving

The same probe technology can ruin cognition in two opposite ways: drowning the model in world, or feeding it a single number with no texture.

TL;DR

  • Too much world blinds: hundred-thousand-line dumps erase patterns in noise.
  • Too little world starves: a lone COUNT(*) has no semantic texture.
  • Useful perception lives in the middle: representative evidence at the right resolution.

Once you accept that agents forge eyes, a second craft appears: eyes have resolution, and resolution can be wrong even when the tool choice is right. SQL can be the correct family of instrument and still produce a useless sensorium. The failure modes sit at opposite ends of one axis.

TOO MUCH WORLD  ←────── useful perception ──────→ TOO LITTLE WORLD
100k raw rows          representative evidence       one COUNT(*)
context pollution      semantic texture + structure  no texture
pattern disappears     next question becomes clear   nothing to understand

The resolution axis — definitive placement. Later chapters assume this diagram.

Blinding: when the sensor floods the mind

A poor probe can return a hundred thousand lines. Technically, the agent “got the data.” Cognitively, it just poisoned its own attention. Patterns that would have been obvious in a sample disappear into volume. The text becomes too verbose, too long, too mishmashy — the operator complaint from the source material is also a precise engineering diagnosis.

This is not only folk wisdom. Context-engineering practice treats the window as an attention budget, not a free warehouse.1 Research on long contexts shows performance is not a free gift of length: relevant information placed in the middle of a long context is easy to under-use — the familiar U-shaped degradation.5 Dumping more rows can feel thorough and still be a perceptual error.

Tool-description bloat is the same disease at a different layer: pack enough tool schemas into the prompt and you crowd out the project context the model needed to think.6 Too much world is not only a data problem. It is any representation that spends the retina on the wrong pixels.

Starving: when the aggregate erases the thing

The opposite failure looks professional. COUNT(*) = 4827. Clean. Exact. Almost content-free. The model now knows a cardinality and still cannot answer whether those rows are newsletters, incidents, or gold. Over-compression removes the semantic texture judgment needs. Pure deterministic systems will happily starve forever: another aggregate, another join, another schema check — never the thirty examples that would make a human say “ah.”

Starvation is why “just give me the metrics” is sometimes the wrong operator instinct. Metrics are eyes too. They are eyes with a very particular blur. Useful for some questions. Lethal for the questions that require recognising what a thing is.

The middle band

Useful perception is engineered into a middle band: not the whole table, not a single integer. In the Postgres archaeology, the working retina often looked like a projection — date, sender, subject, first few hundred characters of body — after automated classes were excluded. Representative evidence. Enough structure to navigate. Enough texture to judge. Not so much volume that the pattern dissolves.

Not too broad. Not too narrow. Not too verbose. Not too lossy. That is a cognition problem. The SQL is only the optics.

The same axis appears outside databases. On a codebase, a repository map that exposes signatures and structure without full file bodies is a middle-band retina — enough to navigate a tree larger than the context window, not so much that every line competes for attention.7 On a UI, a denoised semantic DOM is the same idea: structure without pixel soup. Wherever you work, ask the same question after every probe: am I blinding or starving?

How to tune

After each measurement, force a resolution check before the next forge:

  • If blinded: sample, project fewer columns, filter classes, aggregate then sample the buckets, page deliberately.
  • If starved: request examples, widen the projection, show bodies not only counts, retrieve outliers as well as means.
  • If middle: stop tuning optics for a moment and eyeball (next chapter). More probes are not always progress.

Resolution is a first-class design variable. Teams that only debate which model to buy will keep losing to teams that debate what the agent is allowed to see.

Resolution is not the same as “more detail later”

Software people sometimes hear “resolution” and think of progressive disclosure as a UI nicety — start coarse, drill down when clicked. Agent perception is harsher. A wrong resolution is not merely incomplete; it can actively prevent the next thought. A blinded context does not politely wait for a finer pass. It spends the attention budget on noise and returns a confident wrong story. A starved context returns numbers that feel rigorous while the category of the thing remains unknown.

That is why the middle band is engineered, not hoped for. You choose columns. You choose sample size. You choose which classes to exclude before sampling. You choose whether the next step is another probe or an eyeball. Each choice is a retina setting. Operators who never name those settings will keep blaming the model when the real failure was optics.

Key takeaways

  • Blinding and starving are opposite failures of the same axis.
  • Representative evidence is manufactured, not discovered by accident.
  • After every probe, name which way you missed — then re-forge deliberately.
06
Part III · Operating the Retina

Eyeball It: Instruction Primitives That Change Behaviour

“Review,” “validate,” and “analyse” often re-trigger the forge. “Eyeball it” means something else: stop measuring and apply semantic judgment to the evidence already in hand.

TL;DR

  • Eyeball is a distinct instruction primitive: consume evidence perceptually.
  • Natural-language near-synonyms are operationally opposite modes.
  • Build a shared vocabulary with your agent or you will fight invisible mode errors.

This chapter exists because a working operator invented a word when ordinary English failed. Watching the deterministic–AI ping-pong, he noticed a failure mode: when he asked the agent to review a result, it often forged another probe. Another grep. Another count. Another filter. The loop that had been a strength became a trap. So he started saying something else: eyeball it.

Stop manufacturing measurements for a moment. Put the evidence into your own context and apply semantic judgment.

What the words actually do

In agent sessions, vocabulary is not decoration. It is mode control. Different verbs reliably elicit different behaviours — even when a thesaurus claims they are synonyms.

Primitive Operator intent Typical agent behaviour Mode
Measure Get exact facts SQL, COUNT, schema, filters Deterministic probe
Validate Check against a rule Tests, asserts, regex gates Deterministic probe
Review Often ambiguous Frequently re-forges greps and filters Probe-prone
Analyse Often ambiguous May spawn more instruments Probe-prone
Eyeball Semantic judgment on evidence in hand Read samples; name what it is Perceptual consumption

Instruction-primitive table — definitive placement for this ebook.

Two modes, side by side

Deterministic mode

  • How many?
  • Which rows?
  • Which pattern matches?
  • Does X equal Y?
  • Can I eliminate newsletters?

Eyeball mode

  • Read this.
  • Look at the examples.
  • What does this feel like?
  • What is actually in here?
  • Is this the thing we are looking for?

Pure deterministic systems are terrible at knowing when to switch. They can run SELECT for eternity. The model is the component that can say: forget the aggregate for a moment — show me thirty examples — ah, I see it. That return to semantic consumption is the cognition re-entering the loop. If your instructions keep saying “review” and “validate,” you may be accidentally forbidding that return.

When to say which word

  • Measure when you need ground truth cardinality, existence, or exact match — and you know the question is quantitative.
  • Validate when a rule or invariant already exists and the job is pass/fail against it.
  • Eyeball when the probes have produced a middle-band sample and the missing step is recognition, category, or taste.
  • Avoid bare “review/analyse” unless you immediately disambiguate: “review by eyeballing twenty rows” or “review by validating against schema X.”

This is operator skill, not model mysticism. You are teaching a shared dialect. The same way teams invent words for deploy states and incident severities, agent operators need words for perceptual modes. “Eyeball it” is one that has already paid rent in the wild. Steal it.

Notice the placement in the larger doctrine. Chapters 2–4 celebrate forging. Chapter 5 teaches resolution. This chapter is the brake pedal: forging is not always the next move. Sometimes the eye is good enough and the failure is continuing to manufacture measurements instead of thinking.

A worked failure, then the fix

Imagine the agent has already produced a middle-band sample: thirty human-looking messages after newsletter exclusion, trimmed bodies, sane senders. You say “review this and tell me if we found the staff discussion we care about.” A probe-prone agent often hears a quality-assurance brief. It greps for keywords. It counts. It writes another filter. It never sits with the thirty examples long enough to notice the thread that is actually a hiring conversation, not project provenance. You spent the attention budget on a second instrument when the missing step was recognition.

Now say instead: “Eyeball these thirty samples. Do not run another query first. What is actually in here — and is it the thing we are looking for?” You have forced perceptual consumption. Multi-turn tool agents are excellent at directing tools across open-ended work;4 your job is occasionally to tell them to stop directing tools and use the evidence already paid for.

Teach the dialect once per project — in the system prompt, in the goal preamble, or as a standing rule: when I say eyeball, you read; when I say measure, you probe; when I say validate, you assert against a named rule. Shared vocabulary is cheap. Mode confusion is expensive.

Key takeaways

  • Eyeball = consume evidence perceptually; do not auto-reach for another probe.
  • Review/validate/analyse are probe-prone unless you disambiguate.
  • A small shared vocabulary prevents invisible mode errors mid-session.
07
Part III · Operating the Retina

Text as Cognitive Membrane

Multimodality can get the world through the door. Text is where the model takes its coat off and starts thinking.

TL;DR

  • For coding agents, the world must become context — and that context is overwhelmingly textual.
  • The agent writes text (code) to see different text (probe results). That loop is the sensorium.
  • Prefer text as the default cognitive home; use other modalities as sensors when structure itself is the payload.

It is immediately obvious, once said out loud, that AI sees a world of text. What is less obvious until you watch a long shell session is the recursive trick: the model emits text that becomes a program; the program returns text; the model thinks again. Sensing is real. Crossing into thought still happens on a textual membrane.

The membrane

Reality on a machine is enormous: multi-gigabyte databases, multi-terabyte pools, process tables, network state, filesystem trees, screen buffers. The model cannot hold the world. It can hold a bounded representation. So every useful contact looks like this:

WORLD
  ↓  (probe / sensor)
REPRESENTATION
  ↓  (must fit attention)
CONTEXT
  ↓
COGNITION
  ↓  (emit code / commands)
WORLD again

For shell-heavy work, the representation is overwhelmingly textual: ls, rg, find, ps, journalctl, psql, jq, git diff, strings, hexdump, Python print. Even a screenshot is usually a temporary sensor hit — a way to learn that a boot screen says INACCESSIBLE_BOOT_DEVICE — after which the working state becomes textual hypotheses about storage controllers and VM configuration.

Multimodality gets the world through the door. Text is where the model takes its coat off and starts thinking.

Default text without absolutism

Operators who live in terminals sometimes overstate the case into “only ever put text into AI.” The sharper doctrine is almost as strong and much harder to knock down: text should be the default cognitive representation. Other modalities are sensors used when the non-textual structure itself carries the meaning — colour scales on a chart, motion in a UI transition, composition in a film still.

We have already published the field note that text is the model’s home turf and that the deterministic–AI pendulum places judgment and ground truth on different substrates. The skeleton-of-a-visual move is the same membrane doctrine on a rendered page: denoise the DOM into structure the model can read without drowning in pixels. Full modality-conversion pipelines — speech plus OCR plus timeline merge for video — are a sibling craft with their own boundary conditions. This chapter does not rebuild those pipelines. It names why, once the world has crossed the door, you usually want to live in text.

That question is perceptual engineering again. The membrane is the constraint. The forge is the method. The resolution axis is the tuning. The eyeball primitive is the moment you stop treating text as something to be re-filtered and start treating it as something to be understood.

Why the membrane makes forging inevitable

If the only world the model can think in is a bounded textual context, then every large system becomes a forging problem. You will never “just load the database.” You will always invent projections, samples, skeletons, maps. Context-engineering guidance that prefers just-in-time loading via tools — paths, queries, links as handles rather than full preloads — is industrial language for the same constraint.1

Staged cognition helps too. A supply chain that routes, explores, and only then judges is already admitting that perception is not a single multimodal gulp. Exploration is where eyes are forged. Judgment is where eyeballing earns its keep.

What crosses the membrane well

Not every representation that is “text” is equally thinkable. Verbose dumps are text and still blind. A single integer is text and still starves. The membrane constraint does not excuse bad resolution — it makes resolution mandatory. Prefer structured, id-tagged, sample-rich text: tables with the right columns, logs with the right window, skeletons with the right hierarchy. Prefer handles that can be re-fetched (paths, queries, primary keys) over pasting the entire world once and hoping.

That is also why “just use vision on the whole dashboard” is a weak default for long work. A screenshot may be a perfect sensor hit for a one-line state. It is a terrible place to live for three hours of reasoning. Extract the state into textual working memory, then forge the next eye against systems that answer in text. The coat comes off at the membrane, not on the pixel sofa.

Key takeaways

  • The cognitive membrane for coding agents is textual; plan for that constraint.
  • Use multimodal inputs as sensors; prefer text as the working state for long thought.
  • Forcing everything through text is why disposable probes are not optional extras — they are how the world enters the mind.
08
Part III · Operating the Retina

Practice: Make Reality Legible

Doctrine becomes useful when it changes what you type into the agent and what you refuse to pre-design.

TL;DR

  • Operate a short loop: question → forge → resolution check → eyeball when ready → next question.
  • Use a phrase bank that forces the right mode.
  • Kill the anti-patterns: dump-then-pray, mega-menus as eyes, pipeline cosplay, eternal review-greps.

If you only remember the magazine diagrams, you will enjoy this ebook and change nothing. This chapter is the operating layer: what to do on Monday when the agent is mid-scramble in your data.

The perceptual loop (operator checklist)

  1. Name the question the eye must serve. Not “explore the DB.” Something answerable: which classes of messages dominate? where is human discussion?
  2. Forge the smallest probe that could confirm or wreck your current hypothesis.
  3. Read the result on the resolution axis. Blinding or starving? (Chapter 5)
  4. Re-forge until you have representative evidence — structure plus texture.
  5. Eyeball when instruments are good enough and meaning is the missing step. (Chapter 6)
  6. Let the next question emerge. Do not force the rest of a pre-drawn eight-box pipeline.
  7. Dispose of instruments that have served their cognitive purpose. Apparatus, not heirloom.

Phrase bank

Steal these lines. They are mode control, not flavour text.

“Eyeball the sample — don’t grep again.”

“Narrower view: exclude automated senders, show thirty bodies trimmed to 300 characters.”

“Too much world — compress to patterns, not a full dump.”

“Too little world — I need texture, not only counts.”

“What would I need to see to know the next step? Forge that eye.”

Anti-patterns

What success feels like

You will know the craft is working when the agent’s language changes. It starts naming classes, not only row counts. The next question becomes obvious to both of you. Wrong views arrive quickly and cheaply instead of as three-week surprises. Occasionally your own world-model updates: something you thought required a specialist turns out to be one persistent goal and a night of forged eyes. That perturbation is a feature of contact with reality. It is also how tomorrow’s intents get bolder.

Where this sits next to “better models”

People still budget as if the only lever is model IQ. Model quality matters — better hypotheses, better recognition of contamination, better next questions. But a strong model staring at a blinded context still fails. Industry work on code execution for agents is effectively an admission of that: filter and transform outside the model so only a reduced product spends the attention budget.3 That is perceptual engineering under a different name. Pair it with the eyeball brake and you have the full inhale–exhale: determinism narrows the world; AI narrows the question; sometimes AI must simply look.

Keep the ownership clean when you apply this. Forging eyes is not the same as building productised action tools that change the world as a service. It is not a compile-time IR project. It is not a modality-conversion pipeline. It is runtime representation construction so the next cognitive step becomes possible. Stay on that craft and the rest of the stack stays clear.

A five-minute drill

Next time you open a large store with an agent, run this drill once on purpose. (1) Ask for a first probe without designing the whole pipeline. (2) Force a resolution check out loud: blinding or starving? (3) Re-forge once toward the middle band. (4) Say eyeball — no further tools until a semantic read. (5) Only then allow the next forge. You will feel the difference between a session that thrashes instruments and a session that thinks.

Close the thesis

An agent’s hidden core problem is engineering how the world appears to itself. It forges disposable deterministic instruments to compress unreadable reality into a textual sensorium tuned between too much world and too little. The freakish capability is the oscillation: AI narrows the question; determinism narrows the world. The workflow you admire from the outside is usually the trace left behind by that thinking — not a topology anyone fully designed first.

Your job is not to become a better SQL artist than the model. Your job is to treat representation construction as first-class work, to tune resolution on purpose, to say eyeball it when measurement has done enough, and to refuse the comforting lie that more tools, more tokens, or more multimodal pixels will substitute for a well-forged eye.

Make reality legible at the right resolution. Everything else is commentary.

Key takeaways

  • Run the perceptual loop deliberately; do not outsource mode control to vague verbs.
  • Keep a phrase bank that separates measure, validate, and eyeball.
  • Success is a clearer next question — not a prettier pre-drawn pipeline.
REF
Sources & Evidence

References & Sources

The evidence base behind every claim — primary research, industry analysis, and technical specifications

Research Methodology

This ebook draws on primary research from standards bodies, independent research firms, enterprise technology vendors, and consulting firms. Statistics cited throughout have been cross-referenced against primary sources.

Frameworks and interpretive analysis developed by Scott Farrell / LeverageAI are listed separately below — these represent the practitioner lens through which external research is interpreted, and are not cited inline to avoid self-promotional appearance.

Primary Research & Standards Bodies

Anthropic Engineering — Effective Context Engineering for AI Agents [1]

Context engineering curates optimal tokens; LLMs have an attention budget

https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents

arXiv:2508.16260 — MCPVerse: A Real-World Benchmark for Agentic Tool Use [2]

Agent performance can degrade as action space grows

https://arxiv.org/abs/2508.16260

Anthropic Engineering — Code execution with MCP: building more efficient AI agents [3]

Code execution filters data before it reaches the model; intermediate results can stay outside context

https://www.anthropic.com/engineering/code-execution-with-mcp

Anthropic — Building Effective Agents [4]

Agents dynamically direct tool usage and operate for many turns on open-ended problems

https://www.anthropic.com/research/building-effective-agents

Liu et al., TACL — Lost in the Middle: How Language Models Use Long Contexts [5]

Accuracy degrades when relevant info sits in the middle of long contexts

https://aclanthology.org/2024.tacl-1.9/

Aider — Building a better repository map with tree-sitter [7]

Structure without full bodies so the model can navigate a large codebase

https://aider.chat/docs/repomap.html

LeverageAI / Scott Farrell — Practitioner Frameworks

The interpretive frameworks, architectural patterns, and practitioner analysis in this ebook were developed through enterprise AI transformation consulting. The articles below are the underlying thinking behind those frameworks. They are listed here for transparency and further exploration — not cited inline, as this is the author's own analytical voice.

Scott Farrell — Text Is the Model's Home Turf: A Field Note on the Pendulum Between Code and Judgment

Deterministic–AI pendulum; text as home turf

https://leverageai.com.au/text-is-the-models-home-turf-a-field-note-on-the-pendulum-between-code-and-judgment/

Scott Farrell — The Skeleton of a Visual

Denoised semantic DOM as structure-without-pixels

https://leverageai.com.au/the-skeleton-of-a-visual-judging-and-generating-images-through-their-structure-not-their-pixels/

Scott Farrell — How to Read a YouTube Video

Modality conversion pipelines; when text conversion is destructive

https://leverageai.com.au/how-to-read-a-youtube-video/

Scott Farrell — The Cognition Supply Chain: From Search to Compounding Agentic Cognition

Route → Explore → Judge as staged cognition

https://leverageai.com.au/the-cognition-supply-chain-from-search-to-compounding-agentic-cognition/

Industry Analysis & Vendor Research

Jentic — The MCP Tool Trap [6]

Tool descriptions crowd the context window, reducing space for project context

https://jentic.com/blog/the-mcp-tool-trap

About This Reference List

Compiled July 2026. All URLs verified at time of compilation. Regulatory documents and standards specifications are subject to revision — check primary sources for the most current versions.

Some links to academic papers and vendor research may require free registration. Government and standards body publications are freely accessible.

↑ Prev Next ↓