Agent memory architecture

RAG Demoted to a Sensor

When you already have a wiki and a vector index, the hard problem is not “which one wins.” It is how they compose — without stuffing raw chunks into the main agent’s head.

Scott · LeverageAI · ~8 min read

I almost shipped the wrong interface.

I already had a wiki tool that walks claims and edges. I was about to hang a second tool on the main agent: “run a RAG search.” It would have returned paragraphs — chunks — straight into the conversation context. Classic stack. Tutorial-complete. Wrong.

The fix was a demotion, not a deletion. Put similarity behind the wiki search. Give it to the backend agent that already rummages the graph and the bronze. Do not present the paragraphs. Do not present the chunks. Treat RAG as just another axis.

RAG stops being an architecture and becomes a sensor — one more axis alongside wikilink traversal, bronze grep, and time.

That sentence dissolves a fight most teams are still having. The substrate question — which corpora should be a wiki versus a vector index — is real, and it has its own rule of thumb: query shape × reuse × loss tolerance, settled elsewhere.¹ This piece is the next problem: you already have both. How should they compose?

The hybrid that isn’t a hybrid

Most “wiki + RAG” stacks are two co-equal tools bolted onto one orchestrator. The main agent calls vector search, gets slabs of similar text, and tries to reason while swimming in them. Or it calls the graph, gets a walk, and still pastes chunks “for coverage.”

That is not composition. That is competition for attention.

Original RAG was designed as non-parametric memory for language generation: retrieve, augment the prompt, generate.² That pattern fits chat turns and single-shot Q&A. Agent work is different. Agents need to traverse relationships, hold state across steps, and not re-derive the same join every time someone asks a multi-hop question. That is why agents need a wiki-shaped memory in the first place.³

Vector search still matters. It finds semantically adjacent material when no edge exists yet. What it does not do is know how facts connect.⁴ Similarity is a real capability with a limited warrant. Treat it like a smoke detector, not like the building’s structural steel.

Multi-axis search: every axis is a sensor

The moment that made the demotion obvious was not a vector leaderboard. It was a missed axis in a walk I thought I understood.

We were reconstructing how an agent had found an old internal email that corroborated a claim — a pivot that never made the client proposal. I narrated the graph path. Someone pointed out I had skipped the date axis.

The pivot was foundational. Foundational ideas predate the proposal. The proposal has a date. The client relationship has a start. That is a bounded window. Add the client name (entity), the email corpus (source), and a multi-decade archive collapses to a short set of bronze records — small enough that a grep is decisive in both directions. Presence and absence both become honest.

That temporal bound was not a filter bolted on after retrieval. It was a causal ordering heuristic used as a search plan: ideas precede artifacts. Date, subject, corpus, wikilink, bronze grep, similarity — each axis is a sensor with its own warrant. Stack them and the search becomes reasoning about how evidence was laid down, not a single similarity lottery.

Chunks never touch the main agent

The double tax of query-time discovery is well named: you pay once to re-infer relationships from loose chunks, and again when the bloated payload dilutes attention.⁵ Models do not use long contexts uniformly. Performance can collapse when the useful span sits in the middle of a long stack,⁶ and frontier systems get less reliable as input length grows — even on simple tasks, well inside advertised windows.⁷

So the hard rule is simple:

The backend search agent does the messy work. It walks edges, greps bronze, runs a vector pass on a few terms, and returns a shaped result: wiki references, short summaries, pointers, and — if similarity found something the graph missed — a typed hit at lower rank. The main agent inherits curated signal, not exploration sludge. That matches the broader discipline of keeping structured memory and re-fetchable tool output outside the long-lived decision window.⁸

Industry hybrids often weave graph features into retrieval itself.⁹ Fine — when that is the product you are building. If your primary substrate is already a navigable wiki, the composition move is different: keep the graph as the map, demote raw similarity to a sensor under it, and type every route so the senior agent knows what it is holding.

Type discipline: the typed-result schema

Trust is not “the vector score was 0.82.” Trust is knowing whether a result arrived via an edge someone concluded or a vector that merely rhymed. Same discipline as derived cache pages: guesses are typed and ranked below sources.¹⁰

Here is the minimum shape the backend should return — enough for the main agent to weight, cite, and escalate, not enough to recreate the double tax.

If the main agent cannot tell those rows apart, you do not have a hybrid architecture. You have vibes with two indexes.

Absorption: RAG proposes; the graph keeps

Sensors that only read forever are a tax. The composition closes when a load-bearing similarity hit becomes a candidate edge for the janitor — the same “query is a write in disguise” loop as filing walks back into the map.¹⁰

Worked example (from a real pattern, not a fake dashboard)

Before. Certifications live in a dump-ground store. Pricing relationships between successive certs never existed as a schema column. Every few months someone asks “what certs do I have / what discounts applied?” The backend vector pass (or a lucky bronze rummage) surfaces an email: Lotus Notes 5 exam discounted 50% because of the Notes 4 cert. Useful. Fragile. No edge. Next quarter, same discovery tax.

During. A walk marks the similarity (or grep) hit as load-bearing — the answer depended on it. The shaped result sets candidate_edge: true. Telemetry of the walk goes to the janitor queue: propose cert:LN4 —qualifies-for-discount→ cert:LN5 with support pointers to the email and the cert record.

After. The edge is filed at the appropriate type and rank. The next query resolves primarily via graph walk. Similarity may still fire as a safety net; it is no longer the only way to know. You did not “migrate the RAG corpus.” You absorbed one recurring rhyme into structure.

That is the design property people want when they ask for a “RAG-hit-share declining over time” chart. If the absorption loop is running, the share of answers that depend on similarity should trend down for stable regions of the graph. Measure it if you instrument typed warrants. Do not invent a percentage to look scientific. The architecture earns the curve; the curve does not invent the architecture.

Writable, consolidating memory is the direction agent systems are already being pushed toward in the wider literature — remember outside the window, reduce re-token cost next time.¹¹ Filing edges from sensors is the same instinct with typed provenance attached.

What this is not

• Not a re-litigation of which corpus should be wiki versus RAG. That is the substrate rule.¹
• Not a full treatment of walk telemetry and derived-page filing mechanics. That is File Back the Walk.¹⁰
• Not “RAG is dead.” Demotion keeps the sensor. It forbids the main agent from drowning in it.

Build checklist

1. Expose one search tool to the main agent: multi-axis, backend-owned.
2. Implement sensors with explicit warrants: edge, time, entity, corpus, grep, similarity.
3. Return shaped results only — gists, ids, paths, pointers.
4. Type every hit; rank similarity below edge.
5. Ban raw chunk injection into the main context (hard fail in code review).
6. When a similarity hit is load-bearing, enqueue a candidate edge for the janitor.
7. Log warrant mix per query so you can later observe whether absorption is shrinking the sensor’s share — without faking the number today.

Closing

The wiki-versus-RAG argument was always slightly the wrong altitude once both systems exist. The useful move is demotion: similarity as a recall net under the graph, typed and ranked, never allowed to impersonate a concluded edge, never allowed to dump its guts into the senior agent’s attention budget.

Put RAG behind the wiki search. Keep the chunks offline. Let the janitor promote what keeps proving itself. The graph gets denser. The sensor’s job gets smaller. That is composition with a trust architecture — not two retrieval products arguing in the same context window.