Ingest Is a Query: The Self-Hosting Wiki

Hand your ingestion engine the same toolbelt as your search, and writing collapses into reading. The wiki starts reading itself to write itself — and compounding stops being a slogan on a slide and becomes a mechanism you can watch.

By Scott Farrell, LeverageAI

Here's a wall that almost every retrieval system is built around, so quietly that nobody names it: writing and reading are two different machines.

In a standard RAG pipeline, ingestion is the dumb half. You chunk a document, embed the chunks, store the vectors, and move on. Nothing at that stage ever asks what the document means or how it relates to the thousand documents already in the store. All the thinking is deferred to query time, when someone finally asks a question and the system scrambles to re-derive, on the spot, relationships that were never built. Ingestion writes; query reads; and the two never share a thought.

That wall has a cost you've probably felt without diagnosing: your corpus grows but never gets smarter. Double the documents and you double the noise, not the intelligence. It accumulates. It doesn't compound. And it can't — because the only moment the system could decide how a new thing relates to everything it already knows is at ingest, and at ingest you built it to think about nothing at all.

So tear the wall down. It's cheaper than it sounds, because you already built the expensive part.

The move: give the ingester the query toolbelt

Your query engine already has a toolbelt. To answer a question, an agent walks the wiki: it pulls the map to orient, searches for candidate pages, opens the promising ones, follows their edges to the neighbours. Map, search, get, neighbours — a handful of navigation tools, exposed over the same interface your users hit.

The move is one sentence: hand that exact toolbelt to the ingest engine, and let it use the tools while it's ingesting. In my own dev wiki, the ingester carries the identical navigation surface I give the query agent — the same tools, over the same interface. So every package I ingest arrives holding the means to explore how it fits into the whole scheme, before it commits to a single edge.

What that does is quietly radical. Ingestion stops being classification-in-isolation and becomes a query. When a package comes in, the agent doesn't stare at it alone and guess a category. It walks the existing corpus to decide how this thing fits — using the same navigation it will later hand a user. Writing collapses into reading. The ingester is a reader while it files, so it files a claim exactly where a reader would later look for it. That writer/reader symmetry isn't a slogan; it's a filing discipline, and I've written about it as one before. What I want to sit with here is what the discipline causes once you run it at scale.

Watch one package land

Make it concrete. A package arrives — say a whole project folder with the conversation that built it, bundled as one unit. (What makes a good ingest unit is its own argument, and I'll take the bundle as given.) In the dumb pipeline, that package gets chunked and embedded and its relationships are left for some future query to reconstruct. In the query-powered pipeline, this happens instead:

Look at where the edge came from. In a bolt-on linking pass, an edge is computed — two things are near in vector space, so we draw a line. Here the edge is discovered, because the agent literally travelled to the neighbouring node and formed a view about the relationship. Same word, "edge," completely different provenance. And provenance is the whole game: an edge that was travelled-to is one a reader can trust and a maintainer can audit, because it records a judgment, not a coincidence. This is the same "following a named link is home turf for a language model" property that makes wiki retrieval work in the first place — except now that home-turf navigation is doing the authoring. The wiki reads itself to write itself.

An edge isn't computed by similarity or bolted on in a separate pass. It's discovered — because the agent travelled to the neighbour and formed a view. The wiki reads itself to write itself.

"Isn't that ruinously expensive per package?"

It's the first objection, and it's fair: a wandering, context-aware, corpus-walking ingest sounds like you're paying frontier prices on every package that lands. You're not — if you split the work along a time seam. The expensive exploration all rides a cheap, read-only scout that walks the corpus and freezes what it found; a single frontier senior then inherits that transcript and emits one governed commit. The scout does the wandering; you pay for exactly one judgment. It's the same two-speed pattern I use everywhere, and it means you get powerful, wandering, situated ingestion without paying frontier prices for the wandering.

There's a second thing riding the back of the search, and it's worth naming because it's the same instinct pointed a different way. I wanted RAG in the loop — sometimes the structural walk misses a needle that only similarity can find — but I refused to put it in the toolbelt, because that would give it a vote, a second navigator with out-of-distribution instincts muddying the agent walking the map. So I demoted it from oracle to prior. In the back end of the search, a RAG lookup runs on the same question and returns its top few hits by count and score as a bare hint — "you might want to include these" — never the chunks, never an answer. A pointer, not a verdict. The reason that integration level is exactly right is the asymmetry: a bad hint costs one wasted glance, while a good hint rescues a needle the structural layer would never have ranked. Hand the model the chunks and a confident-but-wrong one poisons the context; hand it a bare pointer and the worst case is nothing. That demotion — every fuzzy signal to advisory prior, exactly one model deciding — is a doctrine of its own that I've laid out separately; here it's enough that the nudge runs at ingest too.

The consequence: a wiki that hosts itself

Now the part that makes this more than a tidy refactor. Once ingestion is a query against the corpus, the corpus is participating in building itself. The wiki is self-hosting — self-hosting the way a compiler is one, when it's mature enough to compile in its own language.

Follow that through and the physics change. Early packages ingest half-blind: there's barely any corpus to situate against, so they land with thin, tentative edges. Later packages ingest sighted, because by then there's a whole world to connect to. The ingest of package number five hundred is a genuinely smarter act than the ingest of package number five — not because the model got better, but because the map it was walking got richer. Every package you add makes the next one smarter to ingest. That's not a metaphor and it's not a growth-hack aspiration on a slide. It's structural: it's what "ingestion is a query" mechanically forces.

This is also where the dumb pipeline's ceiling shows. A chunk-and-embed store can only ever accumulate — each document lands identically ignorant of the rest, forever, because ingest never reads. A query-powered ingest appreciates, because each document lands into — and learns from — everything filed before it.

Which means ingest order matters

Here's the consequence people find uncomfortable, and it's the tell that the compounding is real. If ingestion is a query against the current corpus, then what's already there changes how the next thing lands — so order matters, by construction.

Take the same package and ingest it twice, in two different worlds. Ingest it when the wiki holds three pages and it can only connect to those three; it files a couple of weak edges and sits half-orphaned, correctly describing a neighbourhood that doesn't exist yet. Ingest that identical package when the wiki holds three hundred pages and it lands into a dense neighbourhood: it finds the five real relationships, situates itself precisely, and pulls its weight from day one. Same package. Same model. Different result — and the only variable was how much world was already there to read.

Engineers reach for order-independence as a virtue — reprocess anything, anytime, get the same answer. But a system where order doesn't matter is a system where earlier work taught the later work nothing. Order-dependence isn't the bug here; it's the signature of a corpus that's actually compounding. And it has a clean operational upshot: a later re-grounding pass will always place an early page better than its original ingest could, because it re-runs the query against a richer world. You don't fight the order-dependence; you exploit it — you let the compacting agent revisit the early, half-blind pages once there's enough corpus to see them properly. The maintenance pass isn't cleanup; it's the early pages finally getting the ingest they couldn't have had the first time.

A system where ingest order doesn't matter is a system where the earlier work taught the later work nothing. Order-dependence is what compounding looks like from the inside.

One honest open fork: the back-edge question

I'll end on a real design decision I haven't settled, because it's exactly the kind of fork this move forces and I'd rather pose it straight than pretend it's solved.

The ingest-time toolbelt lets a new package see that it connects to a page that was ingested long ago. Good. But when the new evidence reveals the relationship was mutual — that the old page should now point back at the new one — does ingest get to reach back and edit the old page's edges? Two honest answers, and they're a genuine fork:

Which one you want isn't obvious, and I don't think there's a universal right answer — it's a real trade between freshness and containment. It's the same fork the "bias, don't decide" principle would have an opinion about: reaching back is a write with authority, and authority is exactly the thing you want to keep scarce and governed. My instinct leans toward letting ingest propose the back-edge and letting the separate compacting pass be the one that commits it — keeping ingest additive and reconciliation deliberate — but I hold that loosely, and the other branch is defensible for any corpus where staleness is the bigger sin. I'm flagging it as an open fork, not resolving it, because the interesting systems are the ones where you've noticed the fork exists.

Where this leaves your pipeline

If you run a retrieval layer and it plateaus — grows without getting smarter — the diagnosis is almost always the same: you left all the intelligence at query time and let ingestion stay dumb. The fix isn't a bigger model or a better embedding. It's moving one boundary. You already built a toolbelt to read the corpus; hand that same toolbelt to the thing that writes it, and let it walk the wiki while it files. Ingestion becomes a query, the edges start coming from travel instead of arithmetic, and the corpus starts hosting its own construction.

Then watch for the two signatures that tell you it's working: a package that landed thin and half-orphaned early gets placed properly on a later pass, and ingest order starts to matter. Neither of those is a defect to engineer away. They're the sound of a corpus that finally learns from itself — the wiki reading itself to write itself, one package at a time.

Scott Farrell writes on AI engineering, knowledge architecture, and the economics of building compounding intellectual assets at LeverageAI. If your knowledge base grows every week and somehow never gets any smarter, the problem probably isn't retrieval — it's that nothing reads while it writes.