Knowledge graphs · Agentic retrieval

Your Wiki's Redundancy Is Error Correction, Not Bloat

When the same idea appears several times in a knowledge graph, deduplication is the wrong first instinct — most of that repetition is load-bearing structure. Classify any apparent duplicate into four types; only one is safe to merge.

Scott Farrell · LeverageAI · July 2026

About a month into building a knowledge graph on top of an LLM, almost every team hits the same moment. Page count is climbing. Two pages say something that sounds identical. A third restates it for a different audience. Someone opens a pull request titled “dedupe the corpus.” The instinct is storage engineering: if the bytes repeat, compress them. That instinct is wrong for a retrieval graph — and acting on it is one of the fastest ways to destroy the property that makes agent walks robust.

I have been watching agents walk the same body of knowledge under different search parameters, different result depths, slightly different questions. The path rarely repeats. The answer quality barely moves. The pivotal chapters still get read. The same principle shows up as an original essay, then as an update, then as a concept page, then as a later ebook chapter — and a walker that enters through any of those doors can still synthesise something grounded. That is not a pile of wasted copies. It is a substrate with multiple ways in.

How you know the answers are stable across routes is a measurement question. That argument — four quantities, a family of walks, an omission test for model-prior substitution — lives next door in Route-Invariant Grounding. This piece assumes route variation happens and asks a different question: what corpus shape makes that variation survivable? Which repetitions are load-bearing? Which ones are free bytes?

Classify any apparent duplicate into one of four types. Only the fourth is safe to merge.

Why storage instincts mislead you

Storage culture trains teams to hunt redundant, obsolete and trivial material. Classic industry surveys of enterprise data estates have long reported large shares of stored content as known ROT or “dark” — value unknown to anyone.¹ That framing is useful when you are paying for disks full of abandoned exports. It is a poor frame when you are building a graph an agent will navigate.

In a retrieval graph, the expensive resource is not a few kilobytes of repeated prose. It is reachability under vocabulary shift. An agent that arrives with the user’s words, not your house terms, needs a door that matches those words. If you collapsed every restatement into one preferred phrasing, you optimised for a single entrance and called it cleanliness. You sold a free gigabyte for a route you will miss the next time the question is asked in a dialect you did not author.

The canon already knows half of this story. Chronological stacking, contradiction-as-edge, and a janitor that can combine, fade, convert claims to edges and spin off pages are load-bearing maintenance doctrine — not optional polish.

What the canon has not yet done clearly enough is defend agreement stated twice. We already preserve disagreement as structure. The counter-intuitive half is protecting agreement that looks like duplication but is actually an entrance, a developmental step, or a cross-domain restatement. Without that positive account, every tidy-up pass will treat types one through three as free meat for the combine job — and the graph will get smaller in the wrong places.

The four types — and what “preserve entrances, collapse authority” means for each

Here is the taxonomy. The rest of the article does the work: what each type looks like in a real graph, why merge is or is not safe, and what the remedy means concretely — not as a slogan.

The first three are analogous to error-correcting structure. You can lose one page, or approach the idea from an unusual angle, and still recover the meaning. Deduplicating them removes routes, not waste. The fourth is the only type where the merge genuinely reduces confusion without selling reachability.

Type 1 — Multiple semantic entrances

What it looks like. You have a claim that matters: say, that a compiled claims-and-edges layer is a transformation of sources, not a cache of them. That claim appears on a framework page written for architects. It appears again on a concept page aimed at operators who arrived through a different search phrase. It appears in a short “why this exists” blurb on a boot-profile hub. The sentences are close. A similarity score would light up. A storage-minded engineer sees three copies.

What the graph actually holds is three doors. One door matches “compile vs cache.” Another matches “index is the data.” A third matches “why not just RAG the emails.” A walker that never saw your preferred title still lands in the territory. That is the point of multiple semantic entrances: the same principle in different contexts and vocabulary, reachable from more questions.

Why merge is unsafe. If you combine the three pages into one “canonical” page with the house phrasing, you did not only delete bytes. You deleted the vocabulary surface that matched the odd question. The next agent that arrives with the user’s words — not yours — has one less place to enter. You may still have edges, but you have narrowed the semantic basin that made route variation survivable. The multi-entrance behaviour that looks like “redundant knowledge depth” from the outside is exactly what this type buys.

What “preserve entrances, collapse authority” means concretely for type 1:

• Preserve the entrance pages as first-class navigational objects. Do not force every walker through one preferred title.
• Collapse authority by attaching a single current claim record (or a clearly marked current claim block) that all three entrances point to with a typed edge — for example, compresses toward a capstone claim, or updates toward the present wording of the proposition.
• Do not average the prose into one paragraph that is less specific than any entrance. Authority collapse is about which claim is current, not about sanding every surface into a single tone.
• Lint for orphans, not for near-synonyms. An entrance with no path to the current claim is a maintenance bug. An entrance with different vocabulary and a clear edge to the current claim is a feature.

A practical check before you “dedupe” a type-1 cluster: ask which user questions each page actually answers as a first hit. If the questions differ by domain language or job role, you are looking at entrances, not clones. Keep the doors. Point them at one authority.

Type 2 — Progressive development

What it looks like. An original article introduces an idea when it is still half-formed. Six months later an update extends it, corrects a boundary, or applies it under pressure. A year later a capstone ebook chapter compresses the region. The update often carries enough of the earlier shape that a present-doctrine question does not need the full original. The original still holds origin detail the update never restated. Both pages “say the same thing” if you only skim the thesis sentence. They do not occupy the same temporal role.

This is the situation that produces the practical retrieval question: do you read the update first and go back, or does the update already carry enough? The honest answer is: it depends which question you asked. Progressive development is not a filing mess. It is history that still has work to do.

Why merge is unsafe. Flattening original and update into one “current page” destroys two things at once. First, it destroys the origin surface — the page that should answer “where did this start?” and “what did we believe before the correction?” Second, it invites the worst janitor failure mode: a merge that reads more confidently than either source, silently erasing a real distinction. That failure already has a name in the canon — hallucinated consolidation — and a set of mitigations (chronological legibility, reviewable diffs, lint). This piece will not re-teach those mechanics; it will only say that progressive development is exactly the material those mitigations are meant to protect from naive combine jobs.

What “preserve entrances, collapse authority” means concretely for type 2:

• Keep both ends of the chain visible — original and later work — with edges that name the relationship (develops, updates, supersedes).
• Mark currentness on the claim, not by deleting history. Chronological stacking already does cheap temporal decay: newer claims sit where a top-down reader sees them as current; older claims remain as history rather than ambushing search as if they were still policy.
• Route by question. Present-doctrine walks should prefer the update or capstone. Origin walks should land on the original. Change-history walks need both. That is not indecision; it is question-relative canonicality, developed fully later in this piece.
• Never “merge for neatness” when the only difference is date and corrective force. That difference is the entire point of progressive development.

A practical check: if removing the earlier page would make “how did this idea change?” unanswerable from the graph, you are not looking at accidental duplication. You are looking at a developmental chain. Edge it. Do not flatten it.

Type 3 — Cross-domain restatement

What it looks like. The same principle is re-expressed in software architecture language, then in strategy language, then in publishing operations, then in governance. The core proposition is recognisable. The surrounding graph is not. In software it sits next to test harnesses and regeneration loops. In strategy it sits next to decision rights and review cadence. In publishing it sits next to proof surfaces and interrupt budgets. In governance it sits next to authority boundaries and audit trails. A similarity model still says “near duplicate.” The neighbourhoods disagree.

This is where storage instincts and retrieval instincts collide hardest. Surface text looks mergeable. Graph role does not. The principle is not duplicated if the surrounding relationships differ — because those relationships are what the walker will use next. Edges are the load-bearing asset of a compiled layer; pages are not isolated essays floating in a folder.

Semantic Refraction names a related discipline from the publishing side: a rich source resolved into meaning-complete, provenance-linked directions of meaning, rather than shredded into lesser promotional scraps. Averages destroy edges; addresses create them. Cross-domain restatement is the same grain problem facing the other way. You are not looking at “the strategy page is a lesser clone of the software page.” You are looking at the principle acquiring different legitimate joins in different domains. Collapse those joins into one page and you did not tidy — you destroyed domain-specific edges.

What “preserve entrances, collapse authority” means concretely for type 3:

• Keep domain pages when their neighbour sets differ in operationally meaningful ways (different applications, different failure modes, different owners).
• Collapse authority at the abstract claim — one current statement of the principle — and hang domain pages under applies (or an equivalent application edge), not under “duplicate of.”
• Refuse marketing atomization as your mental model. Channel coverage logistics (cut a pillar into twelve social posts) is not the same job as relational enrichment across domains. One ships surfaces; the other creates joins.
• Test by neighbourhood, not by sentence. List the five most important edges out of each candidate page. If the edge sets diverge, you are in type 3.

A practical check: would a walker who only saw the software restatement be missing a governance constraint that lives only on the governance page? If yes, merging is not compression — it is information loss dressed as hygiene.

Type 4 — Accidental duplication

What it looks like. Two pages say substantially the same thing. They occupy the same graph role: same kinds of neighbours, same claim status, same temporal position, no additional route. Neither is an entrance for a distinct vocabulary. Neither is an earlier or later developmental step. Neither carries a domain neighbourhood the other lacks. They exist because two ingest passes, two authors, or two copy-forwards produced the same unit twice. This is the only type that is safe to merge.

Why merge is safe here — and only here. You are not removing a door. You are not erasing a developmental stage. You are not collapsing domain-specific joins. You are removing a second copy of the same navigational object. The combine job in the janitor’s inventory is for this case. So is a careful convert-to-edge when the “page” is really a relationship that should never have been prose. The knowledge-graveyard failure mode is real: a system that only appends and never integrates becomes less useful as it grows. Type 4 is the legitimate fuel for that integration. Types 1–3 are not.

How to recognise type 4 without fooling yourself:

• Same claim status (both marked current, or both marked historical in the same way).
• Same temporal role (neither supersedes the other; neither is an origin with a later correction).
• Same neighbourhood shape (edges point to the same cluster of concepts, people, projects, sources — not merely similar words).
• No distinct entrance vocabulary that real queries use (if users or agents actually arrive through different phrasings, reclassify as type 1).
• No domain-specific applications that only one page carries (else type 3).

What “preserve entrances, collapse authority” means for type 4 is almost inverted: you do collapse into one page, and you preserve nothing that was only a second copy. Still leave a short provenance note if the duplicate came from a real second source event — but do not keep two live authorities. Two current pages for one claim with one graph role is how agents get double-weighting and humans get “which one is real?” fatigue.

Typed edges: tell the walker why each version exists

Authority without edges is a slogan. The operational half of “preserve entrances, collapse authority” is a small inventory of relationship types that make the graph honest about why two pages both exist. Walk each one — do not treat the list as decoration.

develops is the edge from an earlier formulation to later work that extends it without pretending the earlier page never happened. Use it when the later page adds mechanism, scope, or application while still depending on the original’s introduction. A walker on a change-history question follows develops chains; a walker on a present-doctrine question may stop at the later node once it has enough shape.

updates is the same lineage with a newer statement of the claim. Prefer it when the correction is local — wording, boundary, status — rather than a broad extension. Updates are how progressive development stays readable without rewriting history into a single confident paragraph.

supersedes is stronger: the newer claim is current; the older remains history, labelled as history. Supersession preserves dissent and change without making every historical version look equally current. The temporal machinery of validity windows and as-at walks is owned elsewhere in the canon; here the maintenance principle is enough: keep the older page, mark the relationship, stop dressing 2019 as 2026.

applies is the cross-domain hinge. The abstract claim is one place; the software, strategy, publishing, or governance restatement hangs off it as an application, not as a rival copy. When a janitor sees two pages that “say the same thing,” the first structural question is whether one applies the other in a different neighbourhood. If yes, you are in type 3.

compresses points from a region of pages toward a denser synthesis — often a capstone chapter or a high-density attractor. Compression is not deletion. It is a declaration that a walker with a present-doctrine question may load the compressed form first, while still being able to descend for origin detail. Capstone ebooks often play this role: high-density, source-oriented attractors that consolidate a region without requiring every walker to rediscover the full publication history.

originated-in points back to introduction — the first appearance, the conversation that minted the idea, the essay that first named it. Origin questions need this edge. Present-doctrine questions may ignore it. Without it, every later compression pretends to be the birth of the idea, and institutional memory loses the ability to answer “where did this start?”

None of these edges says “these two pages are the same page twice.” That sentence is almost always a type-4 claim, and type 4 should merge rather than edge-dance forever. The edges exist so that types 1–3 stop looking like type 4 under a careless similarity score.

Worked example: one claim, three questions, different correct pages

Abstract principles are cheap. Walk a single claim through the machinery.

Suppose your graph holds a principle we will call, for this walk, P: “A compiled claims-and-edges layer is a transformation of sources, not a cache of them — the edges are added structure that was never present in the raw material.”

In the graph, P exists as:

• Page O — the original essay that introduced the compile-not-cache distinction, with early examples and a few claims later corrected.
• Page U — an update that tightens the claim, adds the janitor and lint context, and carries enough of O’s shape that a current question often need not open O.
• Page C — a capstone chapter that compresses the region, dense and source-oriented.
• Page S — a software-domain restatement: regeneration loops, characterisation tests, “spec as asset.”
• Page G — a governance-domain restatement: reviewable diffs, self-maintaining ≠ unsupervised, authority boundaries.
• Page X — a near-identical copy of U created by a second ingest of the same source, same neighbours, no new vocabulary, no new domain.

Now three questions arrive.

Question A — origin: “Where did the compile-not-cache idea start, and what did the first version get wrong?” The correct load-bearing region is O, with edges into U for the corrections. Landing only on C is a compression that answers the wrong job. Merging O into U earlier would have made this question unanswerable without archaeology outside the graph.

Question B — present doctrine: “What do we currently hold about the compiled layer versus a cache?” The correct authority is U or C (whichever your graph marks current for that claim), with applies edges available into S and G if the asker needs domain colour. O is optional colour, not required. This is the case where progressive development legitimately lets the later page carry enough of the earlier shape.

Question C — change history: “How did our view of consolidation risk evolve?” The walker needs O and U (and likely the hallucinated-consolidation material) as a chain, not a single smoothed paragraph. Supersession and development edges are the path. A “deduped” single page that averages O and U into confident present tense has lied about the institution’s learning.

And page X? That is type 4. Merge it into U. Keep a one-line provenance if you care that a second ingest occurred. Do not keep two current authorities for the same claim with the same role.

Notice what the example did not do. It did not declare one page “THE canonical document” for all three questions. It declared a current claim for present doctrine, preserved entrances and history for the other question shapes, and reserved merge for the true clone. That is the whole doctrine in miniature.

Canonicality belongs to the claim-and-question pair

The sharpest consequence of the taxonomy is what it does to the word “canonical.”

Most teams import a document-management meaning: there is one true file, and everything else is a shadow. That meaning is already too blunt for ordinary drives full of version chains — which is why canonicality has to be treated as a synthesis product: a concluded judgment about which of several versions is current, recorded once with reasons and receipts, not re-litigated by every search. For agentic graphs the blunt meaning fails a second way. Even after you have concluded which claim is current for present doctrine, the origin question and the change-history question still need other pages.

So canonicality is partly a property of the claim-and-question pair, not simply of one document forever:

This is not relativism. The present-doctrine claim can still be firm, owned, dated, and reviewable. You are not saying “every page is equally true.” You are saying “truth for action” and “truth for history” and “truth for domain application” are different retrieval jobs that share a graph. A single canonical document cannot serve all three without lying to at least one of them.

Boot profiles make the same point from another angle: in a graph you can enter at any page and still reach what the task needs, because edges guarantee reachability. The entry point sets where you start and what you see first; it does not have to be the only authority that exists. Multiple entrances are not a failure of single-source discipline. They are how a field, rather than a monolith, works.

The redundancy-ablation experiment (a design you can run)

Honesty fence: the protocol below is proposed, not executed as a reported study in the source material for this piece. Do not read the following steps as results. They are a design you can run on your own cluster.

If the taxonomy is right, you should be able to put it at risk. Here is a falsifying experiment specified to the point a team can execute it next week.

1. Pick one apparently repetitive cluster. Choose three to six pages that a similarity search or a tired human would call “the same idea.” Prefer a cluster you were already planning to merge — that is where the bias is strongest.

2. Classify first, without merging. Force a written type label (1–4) for every pair in the cluster, with one sentence of evidence per label (vocabulary entrance, temporal role, neighbourhood difference, or true clone). If the team cannot agree, that disagreement is already data: your merge rule was under-specified.

3. Build a “routing-only” variant of the cluster. For every pair labelled type 1–3, replace the duplicated prose with: (a) a short entrance stub if the page was primarily a door, (b) explicit typed edges (develops, updates, supersedes, applies, compresses, originated-in), and (c) a single current claim block for present doctrine. For every pair labelled type 4, actually merge. Do not silently rewrite distinct ideas into one confident paragraph — that is the hallucinated-consolidation failure mode, not the experiment.

4. Pin a question family. Write six to twelve questions that should hit this region: at least two origin questions, two present-doctrine questions, two application or domain questions, and two adversarial rewordings that avoid your house vocabulary. Freeze the question text before you look at either graph variant’s answers.

5. Replay under both graphs. Same model family if you can, same tool surface, same source-descent rule. Capture which pages opened, which claims were cited, and the answer text. You are not yet scoring “vibes.” You are scoring whether entrances still work and whether authority is clear.

6. Interpret along three lines — not one score.

• Meaning: Did answers lose qualifications, corrections, or domain constraints that only lived in a removed restatement?
• Vocabulary coverage: Did reworded questions that previously entered through a type-1 door now miss the territory?
• Mere bytes: Did type-4 merges (and only those) reduce confusion with no loss on the questions above?

7. Report the awkward outcome too. Sometimes the redundancy really was bytes. That is a win for the combine job — and a validation of type 4. Sometimes “cleaning” type 1 or 2 pages quietly breaks origin questions or vocabulary-naive walks. That is the result that should stop the next reckless dedupe PR. A protocol that only celebrates successful merges is not a test; it is a cleanup ritual with telemetry cosplay.

What this experiment does not claim: a universal percentage of “good” versus “bad” redundancy across all corpora. Your graph’s mix will differ. The point is local falsifiability: for this cluster, did the repetition carry meaning, coverage, or only bulk?

How this sits beside route-invariant grounding: if you already measure evidence invariance across varied walks, ablation asks a corpus-shape question those walks cannot answer alone — whether the multi-entrance structure you are measuring is still there after you “tidy.” Link the measurement piece for the invariance argument; run ablation when you are about to delete structure.

The janitor’s forbidden zone — and what this is not

The dual-agent picture already has a builder that adds structure and a janitor that subtracts, reconciles, and turns claims into edges so the map compounds instead of becoming a graveyard. The janitor’s jobs are few and nameable: combine, fade, convert-to-edge, spin-off. The lint pass turns self-maintenance into a cheap review signal rather than silent self-certification. Self-maintaining is not unsupervised.

This article’s job is not to re-explain those jobs. It is to draw a fence the combine job must not cross without a type-4 warrant. Types 1–3 are the forbidden zone for aggressive merge. They are also the zone where convert-to-edge and spin-off are often the right tools — because the fix is structural honesty, not prose deletion. Fade belongs to temporal decay of claims that no longer earn attention, not to entrances that still match real questions.

Two neighbouring concepts are easy to confuse with this argument. Name them so they do not swallow it.

Knowledge graveyard is the failure of append-only capture: volume rises, integration does not, usefulness falls. Guarding type-1 entrances is not an argument for never compacting. It is an argument for compacting with a classification rule. The graveyard and the over-zealous dedupe are opposite ditches on the same road.

Copied-lineage collapse is a different discipline: when you treat repetition as confirmation, you must first collapse near-identical copies into independent lineages so amplifiers cannot farm the truth lane with volume. That is about evidence counting, not about deleting navigational entrances from a knowledge substrate. Ten clones of one rumour are not ten independent proofs. Three meaning-complete entrances to one doctrine in three vocabularies are not “ten clones.” Use both rules; do not swap them.

External maintenance culture is useful and incomplete in the same breath. Periodic lint for contradictions, stale claims superseded by newer sources, and orphan pages with no inbound links is a healthy health-check triad.² Lint is not a licence to merge every near-paraphrase. Orphans are a graph defect. Near-paraphrases with different entrances are often the error-correcting structure this piece defends.

What you should do on Monday

Take the last three “duplicate” tickets or janitor suggestions in your queue. For each pair, write the type label before anyone is allowed to merge. If the label is 1, 2, or 3, write the PR: preserve the entrance or the developmental page, collapse authority with a typed edge and a current claim. If the label is 4, merge without guilt — and record that you did, so the next person can see that combine was earned.

Then pick one cluster that still looks suspicious and schedule the ablation design above as a real experiment, not a metaphor. Pin the questions first. Build the routing-only variant second. Replay third. Interpret meaning, vocabulary coverage, and mere bytes separately. If the redundancy was only bytes, celebrate the merge. If it was coverage, put the entrance back and update your house rule: similarity is a hint, graph role is the warrant.

Finally, stop asking “which single document is canonical for everything?” Ask instead: canonical for which claim under which question? Origin, present doctrine, and change history are allowed to load different pages of an honest graph. That is not mess. That is a substrate mature enough to be approached from more than one road.

Most repetition in a mature knowledge graph is error-correcting structure, not bloat. Preserve entrances. Collapse authority. Merge only the fourth type.

The takeaway is operational. After this, you can classify any apparent duplicate into one of four types — and you know that only the fourth is safe to merge.