AI Architecture · LeverageAI

RAG Was Built for Chatbots — Agents Need a Wiki

RAG isn’t failing you. It was engineered for the one-shot chatbot turn. Your agents have a different job — they must traverse, write, hold state, and compound — and on every one of those axes a wiki-graph is native and RAG is a mismatch.

Scott Farrell · LeverageAI · A field guide for AI architects & platform engineers

I dropped my car at Tesla for a failing heated seat. The service rep frowned at his screen: the ticket said driver-occupancy sensor, not heated seat. He looked at it, looked at it again, and said, “yeah, I think the AI got that one wrong.” The triage AI had read my complaint and pre-ordered a plausible part. Here’s the thing — the occupancy sensor genuinely is a common upstream cause of heated-seat faults, so the AI wasn’t even being stupid. It just had no memory of the thousands of nearly identical cases that would have told it to stage the heating element too. It had retrieval. It did not have institutional memory. And that gap — not the model, not the prompt — is the thing most teams are about to slam into as they move from chatbots to agents.

For two years, “give the AI memory” has meant one thing: stand up a vector store and retrieve. Retrieval-augmented generation became the default substrate for AI memory so quickly that we stopped asking what it was actually designed to do. So let’s ask. And the honest answer reframes the entire debate — not as RAG versus graphs, but as a tool quietly being run outside the job it was built for.

RAG solved a different problem

Go back to the founding paper. When Lewis and colleagues introduced retrieval-augmented generation in 2020, they framed it precisely: a way to combine “pre-trained parametric and non-parametric memory for language generation,” grounding a single generation in a dense vector index so models “generate more specific, diverse and factual language” on knowledge-intensive tasks.¹ Read that as an engineer. The unit of work is one generation. The shape is retrieve → generate → done. There is no notion of traversing relationships across steps, writing anything back, holding state between turns, or getting smarter over successive runs. There didn’t need to be — because the job was the chatbot turn.

RAG is genuinely excellent at that job. Fast, cheap, deterministic; one embedding call and a similarity search and you have grounded context in milliseconds. If your workload is single-shot question answering — “what does our refund policy say?” — RAG is the right tool and this article is not arguing otherwise.

But the workload is moving. Anthropic draws the line cleanly: agents are systems where “LLMs dynamically direct their own processes and tool usage,” used for “open-ended problems where it’s difficult or impossible to predict the required number of steps” — and where “the LLM will potentially operate for many turns.”² That isn’t the chatbot turn; it’s the opposite of it. And it’s already here: LangChain’s platform telemetry shows the average number of steps per trace more than doubling in a single year — from 2.8 to 7.7 — as tool-using traces went from a rounding error to a fifth of all activity.³ Their survey of 1,300+ practitioners found about 51% already running agents in production, with performance quality — not cost, not safety — the number-one concern.⁴ Gartner expects roughly a third of enterprise software to embed agentic AI by 2028, up from less than 1% in 2024 — while also predicting that over 40% of agentic projects will be cancelled by end of 2027, citing inadequate foundations among the causes.⁵

The four things an agent needs from its memory

Strip away the vendor noise and an agentic workload makes four demands on its memory that a single chatbot turn never did. On each one, a wiki-graph — a maintained map of atomic claims and typed edges rather than a pile of text chunks — is native, and RAG was simply never built to deliver. This is the whole argument in one frame:

None of these is a quirk of one product; the agent-memory literature documents each as a real gap and reaches for a graph or a stateful store to fix it.

Traverse. The vendor of RAG tooling concedes the point itself: Microsoft’s GraphRAG team writes that “baseline RAG struggles to connect the dots… when answering a question requires traversing disparate pieces of information through their shared attributes.”⁶ Academically, HippoRAG shows graph-structured memory beating vector RAG on multi-hop question answering “by up to 20%,” and doing the multi-hop in a single retrieval step instead of re-deriving the relationships query by query.⁷ The relationship an agent needs to walk is precomputed in the graph and rediscovered, expensively, on every RAG query.

Write & compound. Every serious agent-memory system adds a writable, consolidating layer precisely because retrieval alone can’t update or accumulate. Stanford’s Generative Agents store “a complete record of the agent’s experiences… synthesize those memories over time into higher-level reflections, and retrieve them dynamically to plan.”⁸ Reflexion agents “maintain their own reflective text in an episodic memory buffer” and measurably improve across trials.⁹ Mem0 reports a writable, optionally graph-structured memory beating full-context baselines with “26% relative improvements” in quality while cutting token cost by over 90%.¹⁰ The reflex objection — “just put it all in a giant context window” — fails on its own terms: “Lost in the Middle” shows accuracy sagging when the relevant fact sits mid-context,¹¹ and Chroma’s testing of 18 frontier models found they all degrade as input grows, “often in surprising and non-uniform ways,” well within their advertised limits.¹²

Hold state. Production agent frameworks already put state in an external, durable store and resume from it — LangGraph’s persistence layer exists so an application can “keep useful information beyond a single graph run.”¹³ MemGPT made the analogy explicit, giving the agent an operating-system-style memory hierarchy outside the context window.¹⁴ And on compounding, a16z says it most bluntly: “retrieval is not learning. A system that can look up any fact has not been forced to find structure.”¹⁵ A read-only index cannot compound, because nothing about the act of using it makes the next use better.

Notice what just happened. Four independent research threads, each chasing a different agentic failure, all converge on the same prescription: give the agent something traversable, writable, stateful, and compounding. That something is a graph you maintain — a wiki.

Same loop, two substrates

Abstractions are cheap, so run my Tesla repair as an actual agent loop, twice, and watch the four needs bite.

On RAG. The ticket arrives. The agent embeds “heated seat not working” and searches the service history. It gets back chunks — some manuals, some closed tickets that mention heated seats — and generates a guess. It cannot walk from heated-seat complaint to occupancy-sensor fault to 2023 seat-module change, because no edge exists; each of those is a separate chunk, related only if the model happens to infer it. When the proposal fails a customer-trust check, the agent re-searches the same corpus and gets the same chunks. Nothing is written back. The thousandth identical heated-seat case is handled exactly like the first. The agent is fast, and it is amnesiac.

On a wiki. The agent lands on [[Heated Seat Complaints]] and traverses a typed edge — possible-upstream-cause → [[Driver Occupancy Sensor]] — then to [[Model 3 Seat Module]] and [[First Visit Resolution]]. The relationships were compiled from thousands of closed cases before this ticket arrived. It holds its candidate proposals as state across repair attempts. And when the case closes, the outcome — which part was actually used, whether there was a return visit — writes back. The next ticket starts from a better map.

Same loop. Same model. Two substrates, and only one of them is doing the job the agent actually has. The difference isn’t intelligence; it’s where the relationships live. Which is why the deepest line from this whole investigation is about knowing versus searching:

A naked LLM doesn’t understand the domain. RAG can search the domain. A wiki-graph can begin to know it.

A workshop manual tells you how the car is supposed to work. The wiki captures how it actually fails, how customers describe it, how often the first diagnosis is wrong, which model years changed the failure mode, and the senior technician’s tacit rule — “when the customer says X but the code says Y, don’t trust the obvious part.” That is dependency-shaped, compiled operational experience that plain chunk retrieval structurally cannot see — the problem I’ve called the Cognition Supply Chain: the model is rarely the bottleneck; the supply chain feeding it is.

Why the embedding caps your agent

Here is the part that took building both to see. In RAG, the embedding plays the role of intelligence — turn the query into a vector, compare against stored vectors, return the nearest blobs. It is brilliant for speed, and it is thin. So however smart your agent gets, it is held back behind the RAG search: it can only act on the nearest few chunks from a couple of semi-random queries. The wiki is the opposite — it compounds on the agent’s intelligence, because a smarter agent extracts more from the map, the edges and the source anchors, and leaves the map better for the next one.

That difference shows up in three concrete ways. RAG confuses relevance with authority: a second-hand mention can sit closer in embedding space than the canonical source, and the agent can’t tell. Nearest is not canonical. RAG also can’t tell you what it missed — it has returned chunks, not a map — so it has no real stopping condition. And RAG gets more complex, not less, as you try to make it smarter: bolt on rerankers, metadata, freshness, source-hierarchy, synthetic chunks, and re-index every time. Which produces the punchline of the whole exercise:

RAG hides its intelligence inside similarity math you can’t inspect. The wiki externalises intelligence as structure you can read: claims, typed edges, canonical references, the reasons an edge exists. I’ve made the architecture-and-economics case for that structure in full elsewhere — The Index Is the Data — so I won’t re-derive the engine here. The point for this argument is narrower: structure you can read is structure a stronger agent can compound on.

From searcher to navigator

The most convincing evidence I have isn’t a benchmark; it’s a behaviour I watched. I built a reviewer agent that gardens a wiki-graph. Early on, when the graph was sparse, it leaned hard on broad search — it didn’t know the terrain, so it had to hunt. Then I changed the first prompt: instead of just handing it a search tool, I injected the map — the root pages, a short description of each, and the list of available ebooks. From that point its behaviour changed. As the graph connected up over successive runs, the broad-search calls fell away almost entirely. It did a few targeted page reads, anchored back to the source ebooks, and navigated by edges. It had become a navigator instead of a searcher.

That phase transition is the thesis in miniature, and it taught me three things worth keeping. The map matters more than the index — a wiki gives the agent a map before the question; RAG gives it fragments after the question. The semantics live in the descriptions: IDs locate, titles name, descriptions orient — and a tool the agent has but can’t inventory is not really available to it. And falling search is a quality signal, but only the right kind: broad search should decline as the graph matures while source-anchoring stays high; search that disappears because the model got overconfident is a different, worse thing.

RAG mostly lives on the bottom rung and stays there. Every query starts cold, assembles context around the question, and leaves the corpus no better organised than it found it. The wiki climbs — and the climbing is the compounding.

Keep RAG for what it’s best at

This is the part the “graphs win” crowd skips, and it’s the part that makes the rest credible. The wiki is not free and it is not always right. It needs an LLM as its intelligence layer, which makes it slower and more expensive per query than a vector search. It has a higher setup cost. And self-maintaining is not unsupervised: a janitor agent can over-compress two distinct failure modes or calcify bad lore into a confident claim, so consolidations need human-auditable diffs and reverts. A wiki you don’t tend will mislead you with more authority than a pile of chunks ever could.

So this was never “wiki beats RAG.” It’s a question of fit. RAG is a fast recall layer; the wiki is a slow cognition layer. Keep RAG for exactly what it’s brilliant at: single-shot lookups, fuzzy recall of a forgotten mention, freshness, exact figures, and the long tail of documents not worth pre-digesting. The mature architecture isn’t a winner; it’s an assignment of each tool to the job it was built for:

RAG is the index. The wiki is the map. The LLM is the navigator. The DAG is the law. The receipt is the proof.

Even the figures should split along the same seam: keep durable relationships in the graph (“occupancy sensor is a common upstream cause under these conditions”) and route to live source systems for the exact current percentage, because stale numbers are dangerous while relationships are directionally durable. RAG for reflex; the wiki for reasoning; the source of record for the number.

This also reframes what “AI learning” even means. People imagine learning as something baked into a bigger model. But you cannot fine-tune inside a loop on every closed case — you can write to a wiki, instantly, cheaply, inspectably, reversibly. The neighbouring doctrine fills in the rest: the loop is robust because the durable state lives outside the agent (Designing Loops, Not Prompts), and once the knowledge is an external, versioned artefact you can also govern it — replay exactly what the agent knew when it acted (The Model Is Not the Memory). The line that ties it together:

RAG isn’t failing you. It did the job it was built for — the chatbot turn — about as well as that job can be done. Your agents just have a different job. Give them the memory it actually requires.