AI Strategy · Context Architecture

The Inference Field: Why a Million-Token Window Is a Residence Upgrade, Not a Capacity Upgrade

The RAM metaphor for the context window is right, and then it becomes wrong. What a long window actually buys is the Residence Dividend — and that is a different thing from the Traversal Dividend it keeps getting confused with.

By Scott Farrell, LeverageAI · July 2026 · ~19 min read

I have an app I wrote that talks to my own knowledge wiki, and I can watch its tool calls scroll past. A few searches, some page lookups, a couple of chapter reads, and the context is over a hundred thousand tokens. On a hard question it gets close to two hundred thousand. It reads a lot.

The first time I really noticed that, my instinct was the one I suspect you would have had: that is a problem, I should compact. Twenty years of engineering habit says a number climbing towards a ceiling is a resource leak with a friendly face.

It took me an embarrassingly long time to notice the actual signal, which was pointing the other way. The answers were not getting vaguer as the context filled. They were getting better — deeper, more surprising, more willing to refuse the obvious suggestion. Whatever all those tokens were doing, they were not sitting there costing me money.

And that is a genuinely confusing thing to observe, because it appears to contradict a discipline I have written and taught myself. So this piece is an attempt to work out what changed, and to name it precisely enough that it can be argued with.

The number went up, and everyone read it as a bucket

The capability change is real and it is dated public fact. OpenAI's GPT‑5.6 family publishes a context window of 1,050,000 tokens.¹ Anthropic publishes one million tokens for Claude Opus 5, Sonnet 5 and the Opus 4.6–4.8 line, and — importantly — states that "for every model with a 1M-token context window, 1M is the default: you don't need a beta header, and long-context requests are billed at standard pricing."² Google publishes "1M / 64k" for the Gemini 3 family.³

It is not universal — smaller models in the same families still ship at 200k³ — but the frontier has clearly entered the million-token class, and it did so without ceremony. What was a gated beta in February is the default now. Here is how it felt from inside the work:

All the frontier models are at a million tokens now. Previously we had a couple of hundred K, and before that 32 and 64K.

Google's own documentation offers the most useful sense of scale: a million tokens is roughly "50,000 lines of code… 8 average length English novels" or "transcripts of over 200 average length podcast episodes."⁴

Now watch what almost everybody does with that number. They read it as capacity. More room. More retrieved chunks per query. More documents you can stuff in before you have to start summarising. The window is a bucket, the bucket got bigger, fill it.

We already have a name for that error in our own work — capacity thinking — and it is the frame this piece exists to break. Because capacity is the least interesting thing that changed.

This article is the sequel to a specific loose end. The Wiki Playbook ended its Part V by naming the Traversal Dividend — the way a compiled knowledge graph becomes worth more as models get better at walking it — and then closed with a deliberate deferral: a longer walk has to be kept somewhere while it is being made, and how much of a walk can stay simultaneously operative is a live constraint in its own right. A different argument, for a different day.

This is that day.

The RAM metaphor is right — and then it becomes wrong

Let me concede the incumbent model completely, because it is a good one and it is partly mine.

At the resource-management layer, the context window behaves exactly like RAM. The durable wiki is storage. The context is the working set. Tool calls page more material in. Irrelevant material causes thrashing. Completed work eventually has to be evicted. That analogy has a formal academic statement — MemGPT proposed "virtual context management… drawing inspiration from hierarchical memory systems in traditional operating systems"⁵ — and the vendors reach for the same register, describing the window as "a 'working memory' for the model."²

It is also the frame behind every piece of context discipline worth having. Attention is a budget, not a bucket: as Anthropic puts it, "LLMs have an 'attention budget' that they draw on when parsing large volumes of context," and "every new token introduced depletes this budget by some amount."⁶ None of that is wrong. I am not about to tell you to stop caring.

But the analogy breaks at one specific seam, and the whole argument lives there.

In an ordinary program, a value sitting in RAM does nothing until some instruction explicitly addresses it. The control logic lives elsewhere. The bytes are inert until read. That is what makes memory memory: it is storage, and storage waits.

A language model has no separate control logic. There is no instruction pointer somewhere else deciding which parts of the window to consult. Every token in the window participates in the attention computation for every token generated. Material in context is not waiting to be fetched — it is already eligible. Not equally weighted, not perfectly recalled, but eligible to alter:

• what the model notices;
• which analogy becomes available to it;
• which path it follows;
• what it treats as important;
• which proposal it never makes;
• and which tool it calls next.

We have argued this before under the name epistemic conditioning: what an agent reads does not sit politely beside its reasoning; it changes the reasoning. What is new is not the mechanism. What is new is how much of it there is room for.

And you can see the disanalogy in the vendors' own prompting guidance, which is the part I find genuinely convincing. Anthropic tells you to put long documents near the top of the prompt and the query at the end, because "queries at the end can improve response quality by up to 30 percent in tests."⁷ Google independently says the same: "in most cases, especially if the total context is long, the model's performance will be better if you put your query / question at the end."⁴

Think about what that means. Where a thing sits in the window changes what it does. That is not how RAM behaves. RAM does not care about address order. This behaves like a field with structure — a room where position, adjacency and salience matter.

The field is not just "the prompt." Everything in the request is in it — "the system prompt, every message… (including tool results, images, and documents), and your tool definitions."² Which in a working agent means the field holds a strange and very specific collection at any moment: the governing posture and instructions; the current intent and definition of done; whichever semantic modules got loaded; tool results and observations; intermediate hypotheses; rejected branches; source excerpts; conversation state; and the current working conclusions.

Hold on to "rejected branches." It is the item that will decide the last section of this article.

None of that is a storage inventory. It is closer to the process image of a temporary mind — and unlike an ordinary process, every part of it is expressed in one medium, language, and interpreted by one runtime.

Which is why the storage word was doing quiet damage all along. If the window is storage, the discipline is eviction and the question is how much fits. If the window is a field, the discipline is composition and the question is what should be co-present. A lookup is an event; residence is a condition — and a condition shapes every decision made inside it, including the thousand small ones nobody would ever write a lookup for. I made that argument at length in #include the Wiki, and again in Executable Worldview: once the relevant sub-world shares a working context with the task, every subsequent token is conditioned on it.

What I got wrong in those pieces was the scale. I argued residence was affordable because you only ever needed to keep a handful of pages resident — a small share of a modern window, and shrinking. That was true, and it under-claimed the thing. At a million tokens you are not keeping the highlights. You can keep the entire walk.

Two contexts, the same size, nothing alike

Here is the claim that makes this more than "long context is good."

Two contexts can both be 200,000 tokens and be cognitively nothing alike. Not because one has better documents in it. Because of how the tokens got admitted.

Consider the two shapes side by side.

The old move, when an agent underperformed, was to stuff more retrieval in and let the model sort it out. It is a reasonable instinct and it produces a specific kind of context: a big pile, sorted by resemblance, with the actual work of finding the structure deferred to the model at the last moment.

The redundancy in that loop is structural rather than a tuning failure you can re-rank your way out of. The top-k nearest chunks to a query are, by construction, also near each other — that is what "nearest" means — so a second, slightly reworded query lands in the same neighbourhood and hands back much of the same pile. Meanwhile the model is being asked to hold two facts it is genuinely bad at holding across a long context: what have I already seen, and what is still out there unseen. A walk never asks it to. Visited territory is simply in the transcript; unvisited territory is enumerated on each page as named, typed edges. The bookkeeping is on the page, not in the head.

The academic comparisons point the same direction without settling it. An evaluation of long context versus retrieval found that "summarization-based retrieval performs comparably to LC, while chunk-based retrieval lags behind"⁸ — which is a finding about how material was prepared, not how much of it there was.

Retrieval spends context on candidates. A traversal spends context on consequences. One is a haystack. The other is a travelled argument.

That is the whole distinction, and it has a consequence worth stating plainly: a walk-built context contains not just information but a causal history of inquiry. Two hundred thousand tokens accumulated through a disciplined walk are not two hundred thousand arbitrary tokens. They are the serialised trace of a system progressively constructing its own understanding — and every step in that trace is still in the room, still conditioning the next one.

This is the point where an argument like this usually overreaches, so: I have no measurement, and as far as I can find, nobody does. What I have is a mechanism and an observation. The falsifier is at the end of the next section.

The Residence Dividend

Now the distinction this article exists to draw, because if you have read the previous piece you are about to conflate two things that are genuinely different.

The Traversal Dividend is published doctrine and I am not going to re-argue it. Its claim: a compiled knowledge graph becomes more valuable as models get better at agentic tool use, because a more capable agent decomposes queries better, selects edges better, holds parent intent longer, backtracks better, and therefore reaches more of the graph. Better driver, more of the road network becomes usable. It is denominated in usable graph radius — how far an agent travels before it loses the intent, pollutes its context, or settles prematurely. That case is made in full in The Wiki Playbook, Part V.

And it has a gap in it, which the deferral at the end of that chapter was quietly admitting.

Reaching is not keeping. An agent that traverses ten neighbourhoods and can only hold three of them together when it comes time to synthesise has spent its whole traversal and banked a third of it. The other seven were reached, read, and then squeezed out by the eighth. Traversal buys you territory. Something else has to keep the territory in the room.

Three effects, three different things, and they are easy to collapse into one another:

If you take one thing from this piece: Traversal is about reach. Residence is about retention under inference. A better model with a small window has a large radius and a leaky room. A weaker model with a huge window has a tidy room and nothing interesting in it. They are different multipliers on the same asset, and a weakness in either one caps the result.

Why residence compounds rather than merely accumulates

Here is where it stops being a capacity claim, which matters, because "more stays loaded" is exactly the frame I have been attacking.

The value is not in retaining isolated thoughts. It is in making them co-present — which is a different act from recall. Recall is fetching one cold item back through a narrow doorway, one at a time. Co-presence is having many prior thoughts available to bear on one live problem simultaneously, so they can meet each other.

And the meetings grow much faster than the material. Seventeen items co-present on one problem permit a hundred and thirty-six possible pairings between them; the eighteenth item does not add one unit, it adds seventeen new possible collisions. That is arithmetic — a fact about pairs, n(n−1)/2, not a measured statistic, and emphatically not a claim that the model examines every pair. But it explains the shape of what I keep observing: a larger, well-structured active field producing nonlinear gains rather than just storing more facts. That argument is developed properly in The Clasp.

Which forces the qualification that keeps this honest, and I want it in the middle of the section rather than buried at the end:

A million-token pile produces combinatorial noise. A million-token task world — with maps, typed edges, source relationships and held intent — produces guided combinatorial possibility.

The structure is doing the work, not the size. Partial activation is a graph property: edges act as a page table, and a pile of markdown without relationships has nothing to page in along. That is the limit of this whole argument, and I would rather state it than have it found. Without a compiled substrate underneath, a bigger window really is just a bigger pile, and everything in the degradation literature applies to you at full strength.

A contradiction I owe my own previous article

The previous article published a memory hierarchy that put the context window firmly in the depreciating column: weights depreciate as they commoditise; the context window and the KV cache evaporate — paid again every call, gone at session end; only the compiled layer appreciates. A reader who noticed that table is entitled to ask what I think I am doing now.

The reconciliation is precise, and it does not require retracting anything. The window still evaporates. It is rented, per call, per session, and nothing about a bigger one changes that. What has changed is how much of your durable asset the rented room can hold while it is alive. The Residence Dividend is not a claim that the window became an asset. It is a dividend paid to the compiled layer by a bigger window — the same graph, unchanged, activated more completely.

Which is the same structure as the Traversal Dividend, one layer over. The graph did not change. What changed is how much of it can be operative at once.

The honest boundary, stated before it is asked for

There is no measured magnitude for the Residence Dividend, and I am not going to invent one. This is a mechanism argument plus behavioural evidence from one operator's own system, reported as such. The nearest external number is Anthropic's finding on its own research evaluations that "token usage by itself explains 80% of the variance" in performance⁹ — which is suggestive and is not the same claim. It says how much the system spent, not how long material stayed loaded, and variance explained is not causation.

The experiment that would settle it is cheap and specifiable, which is the only thing that makes the claim worth publishing. Take a hard question and a fixed model. Run the full walk. Then, immediately before synthesis, evict everything except a good summary of what was found — same findings, same conclusions, none of the residence — and blind-score both answers. If the summarised arm scores the same, residence is not doing the work I think it is doing, and the better explanation is simply that the walk found better material. Until someone runs that, this is a mechanism argument and it should be read as one.

An exoskeleton wearing an exoskeleton

There is one more turn, and it is the part that changed how I think about reasoning effort.

The model is not merely reasoning over a fixed context. It is using the current context to decide what context to acquire next. It reads enough of the map to choose an edge; the page at the end of that edge changes its understanding; that changed understanding produces a better search; the search reveals a convergence; the convergence justifies reading a source chapter; the chapter changes the eventual thesis.

The thinking process builds the substrate it will think with next.

Which is why extra reasoning effort has changed in value. In a fixed text chat, "think harder" mostly means searching the model's own possibility space more carefully — useful for logic and planning, useless for recovering distinctions that were never in the packet. Against a tool-connected graph, "think harder" can mean construct more probes, travel more edges, descend to source more often, and keep the resulting distinctions co-present long enough to combine them. The reasoning budget can now purchase more relevant world, not merely more rearrangements of the original prompt.

That is no longer an inference about lab priorities; it is documented vendor intent. Anthropic's effort parameter explicitly governs tool calls, not just internal deliberation — "lower effort would mean Claude makes fewer tool calls" — and its highest setting is described as "the recommended starting point for coding and agentic work, and for exploratory tasks such as repeated tool calling, detailed web search, and knowledge-base search," with "token budgets in the millions" for long-horizon work.¹⁰ OpenAI ships a setting whose entire purpose is to keep prior reasoning — not merely prior text — available across turns.¹¹ The residence idea is already in the API surface.

Holding all of that together across a long walk is a coherence problem, not a storage problem, which is why the parent intent has to be the unit of work rather than the last question typed. That is the argument of The Intent Compiler, and it is the selection pressure that keeps a growing field a travelled argument rather than accumulated sediment.

Which brings me to the joke I noticed a while back and never named properly:

And here's the funny thing — I'm using the AI as a cognitive exoskeleton already.

I hook up a wiki compiled from my own IP, on the subject of how to use AI, and talk to the model about that same subject. So I am wearing the AI as an exoskeleton for my thinking. And the AI is wearing my compiled worldview as an exoskeleton for its thinking. Then the conversation produces new judgment, which gets written back into the worldview, which conditions the next conversation.

I want to be straight about what is and is not new here. Neither half is new. "Cognitive exoskeleton" — the human wearing the AI — is an established pattern. "The agent becomes a runtime instance of the compiled worldview" is Executable Worldview. The write-back loop is a whole chapter of The Wiki Playbook. What was missing was the name for the two of them facing each other, and the reason it matters is that the loop only closes if the second exoskeleton is durable and legible — which is the case for putting the learned state in language rather than in weights, made in The Promise of AI Learning, Kept.

The felt version, which is the one I would defend in a pub: my past cognition has become a callable organ inside my present cognitive system.

One boundary, because a piece like this has an obligation to draw it. Activation improves cognition; it does not mint authority. A well-conditioned agent is more effective and more dangerous, and the accountable human is a row in the architecture, not a decoration on it. Nothing here says the system decides better. It says the system arrives at the decision point carrying more of what you already knew.

The honest limit: trim on deterioration, not on the number

Now the part where I have to argue against my own enthusiasm, because there is a real ceiling and the research on it is good.

Long context is not uniformly useful. The canonical finding is the U-curve: performance "is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models."¹² Chroma's study of eighteen frontier models found that "models do not use their context uniformly; instead, their performance grows increasingly unreliable as input length grows."¹³ And the vendors say it about their own products: "more context isn't automatically better. As token count grows, accuracy and recall degrade, a phenomenon known as context rot."² Google warns that multi-needle retrieval is materially worse than single-needle.⁴

I take all of that seriously. And then I read the sub-finding that I think is the most important result in the whole literature for anyone building this way. Chroma found that "even a single distractor reduces performance relative to the baseline… and adding four distractors compounds this degradation further," and that models score significantly higher on focused prompts than on full prompts containing irrelevant context.¹³

The damage is coming from irrelevant material sharing the field — not from token count as such. Which is not a refutation of the argument here. It is the mechanism that makes the argument matter. If distractors are the problem, then how a token got admitted is the variable, and the discipline is composition, not size.

The strongest objection to this piece is Anthropic's own guidance that good context engineering means "finding the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome."⁶ On its face that is the opposite instinct. I do not think it is, and the reconciliation is in the sentence itself: the optimand is high-signal, and "smallest set that maximises the outcome" is a constrained optimisation, not a minimisation. A walk-assembled working set is a candidate for that set. A similarity-stuffed pile never was.

So: trimming will eventually matter. My own position on my own system is unglamorous and I will state it as it is — possibly at some stage I am going to have to do some context trimming, but for now it all fits in the one million pretty easily, and the answers are getting more coherent rather than more forgetful.

What has to change is the trigger. A token count measures the container. Here is what to watch instead.

Note that call count is not on that list, and should not be. A better agent might initially make more calls because it recognises deeper investigation is available; a more mature one with a better map might make fewer, because it navigates through typed edges instead of guessing search strings. The metric is useful territory activated per unit of attention, not traffic.

And when trimming does become necessary, the default operation is wrong.

Do not trim the history. Compile the state.

"Summarise the oldest messages" keeps the prose and destroys the trace. Remember what was in that field: rejected branches, preserved contradictions, known absences, the specific reason a promising path was abandoned. A summary keeps the conclusions and throws away the reasoning that earned them — which is precisely the material that stops the agent walking back down a road it already ruled out. Keep three distinct things instead:

• The full walk log — immutable trace and provenance. Cheap to store, impossible to reconstruct.
• The task-world state — intent, accepted findings, rejected paths, contradictions, absences, and source pointers.
• The active inference field — only what still needs to participate in the next stage of thought.

That boundary — persist what is expensive, reusable and integrative; recompile what is volatile, lens-specific and task-local — is worked through properly elsewhere, and I would rather point at it than rebuild it here. It is worth noting that the vendors have converged on the same shape from the other direction: Anthropic's own guidance for long-running agents concedes that compaction "isn't sufficient" on its own and describes carrying forward an explicit progress artefact so a fresh window can understand the state of the work.¹⁴

There is one more reason not to panic about the number, and it is on the pricing sheet. Cache reads cost a tenth of standard input pricing, so keeping material resident across turns is now roughly ten times cheaper than re-reading it fresh.¹⁵ That is a real change — the accounting flip I argued for in Context Arbitrage, showing up in the vendors' own price lists. But it comes with the sentence that should be printed above every compaction routine: "cached prompt prefixes still occupy the context window: prompt caching changes what you pay for those tokens, not whether they count."²

Which is the entire discipline in one line. The cost that matters stopped being the invoice and became the attention. The million-token window delays the moment at which you need a compaction architecture. It does not abolish it.

The room, and who is in it

Set the layers out and each one has exactly one job.

The wiki is potential cognition. The task world is selected cognition. The context is activated cognition. The model is cognition in motion.

The long window enlarges the room. The compiled substrate decides who should be invited. The model conducts the conversation. You decide what the conversation is for — and you remain the one accountable for what gets done about it.

None of the four is sufficient alone, and that is the part the spec sheets cannot tell you. A great model with no substrate is a brilliant stranger. A great substrate with a poor navigator is inaccessible intelligence. Great traversal with a small window means the discoveries fall out of the room before they can combine. A huge window with no structure is expensive confusion. All of it with weak intent is sophisticated wandering.

Which is why the honest summary of the last two years is not that the models got smarter, though they did. It is that we finally stopped asking the model to be the whole system. We gave it a better thinking substrate, and then we got a room big enough to hold the substrate while it thinks.

No wonder the output of this stuff is just insanely better.