Semantic Decompilation
Recover the Design Hidden Inside Prose and Code
After Reading This Ebook, You Will:
- ✓ Treat structure as a symbol table and claims as functions with typed edges
- ✓ Run forward, reverse, and cross engineering as a verification loop
- ✓ Decompile one source with retained scope IDs and reconstruct claims, roles, and edges
- ✓ File a doctrine/implementation mismatch and correct it with evidence
TL;DR
- • A summary compresses. Semantic decompilation recovers design.
- • Code owns cuts and exact recovery; AI owns judgment of role and significance.
- • Claims are functions; edges are calls; prompts and transcripts are debug symbols.
- • Forward / reverse / cross turns mismatch into epistemic lint — the product of the round trip.
- • Build working drawings first; promote meaning, not prose; keep the source map.
Summary Is Not Design Recovery
You can compress a rich argument into a blurrier one and still know almost nothing about what each part does.
Someone hands you an expensive source. A field-guide ebook. A long conversation that became doctrine. A repository that grew while the team was busy shipping. The instinct is almost automatic: summarise it, or shove it into a retrieval store and call the problem solved. Either way, the move feels like understanding. Usually it is compression under another name.
A summary asks what the source says, then spends fewer words saying something nearby. That can be useful for orientation. It is a terrible substitute for recovering design. Design is the load-bearing internals: which claims are premises, which are mechanisms, which are exceptions; what depends on what; what the implementation was supposed to embody; what becomes visible only when one unit is placed against another body of work. Compression throws most of that away on purpose. Then we act surprised when the graph cannot answer a precise question, or when the shipping code no longer matches the essay we published last quarter.
A summary asks what the source says. Semantic decompilation asks what the source is made of, what each part does, and what becomes visible when that part is related to another world.
The average that pretends to be a relation
When a source carries many ideas and you ask how it relates to everything you already know, you are forced toward a general answer. Three themes. A resemblance to two frameworks. An overall significance score. That is not relational precision. It is an average. Averages destroy edges.
I used to treat cut-ups of a long piece as lesser descendants of a pillar. That metaphor was wrong for a different reason than I first thought. The issue is not only that pieces can be valuable in their own right. The issue is that without addresses, you cannot judge each idea against a different axis: what it contradicts, what it extends, who it is for, why it matters now. A fifty-idea ebook forces a blurry join. Fifty addressable units get fifty precise ones. Semantic Refraction named that grain problem on the publishing side — why closed units can form joins the undifferentiated whole was too coarse to hold. This book does not re-derive that thesis. It takes the engineering step that follows: how you recover design from compiled prose and from the prompts and code that implement the doctrine.
Two failed substitutes for understanding
What people call “ingest”
- Summary ingest — replace the source with a shorter document. Fast. Lossy. No roles, no edges, no exact re-find of the original unit.
- Similarity ingest — chunk the source, embed the chunks, retrieve “nearby” text at query time. Familiar. Still mostly unary: each chunk describes itself, not the work of understanding how it joins the rest of the canon.
Industry guidance already treats naive fixed windows as a weak default when semantic understanding matters.1 Hierarchical retrieval work keeps rediscovering that flat chunks behave like islands, divorced from their place in a larger argument.2 Those are retrieval symptoms of a deeper failure: you never recovered the design. You only stored more convenient residues of the text.
The move I care about is different. Treat prose and code as compiled forms of thought and intention. Writing linearises an argument for a human reader. Shipping code linearises decisions into executable order. In both cases structure still exists — headings, scopes, functions, tests — but the relational design is no longer first-class. Semantic decompilation is the reverse pass: recover the symbol table, recover the call graph of ideas, keep the debug symbols if you were smart enough not to delete them, then force doctrine and implementation to answer to each other.
What this book owns
This is a method book with a heavy proof burden. Part I builds the spine: structure as symbol table, claims as functions, three directions of engineering. Part II spends a serious fraction of the page budget on an end-to-end worked path — one ebook decompiled with stable scope IDs, and one implementation path from prompt to observed behaviour, including a real discrepancy and a correction. Part III generalises the cascade, turns mismatch into an ongoing product, and shows how to build without flooding a canon with prose.
What it does not own is just as important. Attention-native publishing already owns the interrupt doctrine for feed surfaces. Semantic Refraction owns relational grain. The unary-metadata critique of RAG, and the architecture of quote storage and promotion, belong to later pieces in this series — name them when useful, do not smuggle their whole arguments in. This is also not a course in reverse-engineering machine code. The software analogy is load-bearing; the binary-hacking tutorial is not.
The enemy has a friendly face
The incumbent belief is not evil. Summaries help humans skim. Chunk stores help chatbots answer quickly. Both optimise for time-to-first-orientation. Semantic decompilation optimises for something colder: can you recover the design well enough that a claim can be inspected, joined, implemented, and tested against behaviour? If you only ever need a gist, stop here and keep summarising. If you are trying to run a living doctrine across prose, prompts, and code, gist is how your system goes quietly wrong.
Next we start where every serious reverse engineer starts: not with interpretation, but with a trustworthy map of the cuts.
Key takeaways
- Summary compresses; decompilation recovers design.
- Whole-document “relations” are usually averages that destroy edges.
- This book owns source-mapped reverse engineering and round-trip lint — not refraction, not feed interrupts, not binary RE tutorials.
The Symbol Table of Source Structure
Before the model judges anything, something non-magical has to tell you where the safe cuts are.
Software people already know the grain problem. A whole repository is usually too large to reason about as one object. A single line is usually too small. A file, class, or function often gives the right working unit — not because it is independent of the rest of the system, but because it has a purpose, a boundary, and a way to name its neighbours. You open one source file instead of the whole project. That is not laziness. It is encapsulation with a map.
Long prose is no different. An ebook is a project. A chapter is often still a multi-idea region. A heading-plus-body, a callout, a closed list, a setup with its punchline — those are the functions. The difference is that publishing tools taught us to treat structure as decoration. Fonts, boxes, hierarchy: “formatting.” In a decompiler, that structure is the first half of the symbol table.
Deterministic disassembly
↓ deterministic structure
chapters → headings → blocks → stable IDs
↓ candidates (+ allowed combinations)
↓ AI judgment
claim · role · significance · candidate edges
The deterministic layer answers two boring, essential questions: where can I safely cut? and how do I find the exact source again? It does not get to decide what is interesting. Interestingness is judgment. Judgment is model work. Confuse the two and you smuggle editorial taste into a layer that should be mechanical — or you force the model to invent boundaries it has no right to invent.
HTML is an especially honest disassembly map when you denoise it. Strip the chrome that exists for browsers, not for meaning. Keep hierarchy. Keep emphasis as hints, not as silent filters. Give every scope a stable identifier. Preserve almost all source words so scored text, rendered output, and validation gates can point at the same bytes. That pattern — a denoised semantic DOM — already proved itself in text-shaped selection pipelines: structure without premature taste.
Code found the pieces. AI judged what the pieces could mean.
Natural keys of intent
Headings are not neutral furniture. They are the author’s own declaration of where one idea ends and the next begins. Lists group failures. Callouts mark load-bearing contrasts. Those boundaries were drawn by a human with intent. A fixed token window cuts across that intent and then wonders why the pieces feel hollow. Source-native cuts inherit the author’s segmentation first, then allow the model to recombine when meaning does not fit a single box.
This is the same soft-data discipline as preferring a natural key over a resemblance score when the source already named the join. Folder names, stable DOM IDs, turn markers, AST symbols: boring handles beat clever similarity when you need provenance. Compilers learned the same lesson with intermediate representations — you need a stage where structure is explicit before you lower again.3
Division of labour
| Layer | Owns | Must not own |
|---|---|---|
| Deterministic structure | Safe cuts, stable IDs, exact extraction, recoverability | Interestingness, canon membership, “what it means” |
| AI judgment | Claim, role, significance, combinations, cross-corpus joins | Inventing source text; inventing fake IDs |
| Graph / IR | Concluded relationships with provenance | Re-deriving the same join from scratch every query |
Call that the Deterministic–AI Pendulum if you already know the name: for each sub-problem, decide whether it is ground truth or judgment, and be willing to move a component when evidence says the wrong side owns it. Decompilation is one of the purest applications. Boundaries want physics. Meaning wants taste. Provenance wants a receipt.
Microsoft’s own RAG guidance already warns that fixed-size chunking is not recommended where semantic understanding matters.1 That is necessary hygiene. It is not yet decompilation. Parent documents and semantic windows still mostly aim at better context packing. Semantic decompilation aims at a reconstructed map of functions and calls.
What the symbol table must preserve
A usable symbol table for prose is not a pretty outline export. At minimum it must preserve: hierarchy (what nests under what), identity (stable IDs that survive re-render), extractability (whole-scope text by ID without drift), and combination hooks (permission to name multi-scope selections without inventing new prose). Lose any one of those and reverse engineering becomes sticky notes on a PDF.
Conversation logs need a different native map — turns, speakers, tool calls — but the rule is identical: cut where the medium already declared a boundary, then judge. Source code already has ASTs; do not pretend a sliding window over a .py file is more sophisticated than the grammar the language gave you for free.
Once every unit has an address, the next question stops being “what chunk looks like the query?” It becomes “what claim lives here, what is it for, and what does it call?” That is the next chapter’s job.
Key takeaways
- Structure is a disassembly map, not decoration.
- Deterministic code owns cuts and exact recovery; AI owns judgment.
- Author-declared boundaries beat fixed token windows as the first cut.
Claims Are Functions
A claim is not a shorter paragraph. It is a bounded unit with a purpose and declared relationships to other units.
Once structure has given you addresses, the temptation is to stop at “interesting excerpts.” Excerpts are useful exhibits. They are not yet the intermediate representation you can reason over as a system. The unit that behaves like a function in an idea system is the claim: a normalised proposition with an intelligible purpose, dependencies, and a stable identity that can survive paraphrase.
That last point matters. A quote should not survive paraphrase — its job is exact language and framing. A claim should. Mix the types and you will either flood a navigation graph with prose, or you will lose the rhetorical exhibit that made a human lean in. Type confusion is how canons get fat and brittle at the same time.
The correspondence table
Hold this table. Later chapters will reference it rather than rebuild it.
| Software | Idea system |
|---|---|
| Repository | Corpus |
| Project / package | Ebook, large conversation, product surface |
| Source file | Chapter or wiki page region |
| Function | Claim or mechanism |
| Function call | Typed edge |
| Interface | Stable claim identity + provenance |
| Runtime behaviour | Published artefact or agent action |
| Tests / receipts | Evidence, path replay, regression gates |
| Debug symbols | Prompts, transcripts, intent notes, acceptance tests |
Writing linearises thought. The call graph of ideas gets flattened into narrative order optimised for a reader. Claims are the recovered symbol table. Edges are the recovered call graph. Public practice is already edging toward claims-and-edges knowledge bases with lint for contradictions and orphans.4 That rhyme matters less as celebrity citation and more as proof that navigation wants functions, not slabs.
Roles, not just topics
A function is not merely “about payments.” It validates a card, charges a customer, or emits a receipt. Likewise a claim is not merely “about prompting.” It may be a premise, a mechanism, a consequence, an exception, or a proof. Role is part of the reconstruction. Without it, every edge looks like a vague related-to, and vague edges are how graphs become hairballs.
Ask sharper questions once the unit is tractable:
- What exact proposition is present?
- What function does it perform in the parent argument?
- Which other claim does it extend, contradict, or evidence?
- Which prompt or code path is supposed to embody it?
- What other unit combines with it to close meaning?
That last question is not optional. Pure atomisation destroys meaning. A heading without its body, a punchline without its setup, two mutually reinforcing boxes split into lonely orphans — the model needs permission to recognise combinations and overlaps while still recording every contributing scope ID. Deterministic candidates without combinatorial judgment is scissors. Judgment without multi-scope provenance is a rewrite.
Most reverse engineering recovers structure and guesses intent. Exhaust-preserved decompilation recovers both.
Debug symbols are a strategic choice
Binaries ship stripped. Names, comments, and intent vanish at compile time. Teams reverse-engineering legacy systems with modern models still spend most of their effort reconstructing purpose from behaviour and residual structure.5 That work is heroic because the intent stream was discarded for years.
AI-native work does not have to repeat that mistake. Prompts, transcripts, agent instructions, tests, and decision notes are the .pdb files of a thought system. Keep the bronze layer — the representations a later decompiler may need — and reverse engineering becomes source-mapped instead of speculative. Throw them away and you are back to watching behaviour for months and calling the guess a specification.
Vector search can still act as a weak sensor for things that might be nearby and currently lack an edge. It is bad structural steel. Multi-hop connection is exactly where pure similarity shows its ceiling.6 Claims and typed edges are how you stop paying the join tax at every query as if it were free.
A claim record that can travel
In practice a reconstructed claim needs enough fields to leave the page that birthed it: normalised text, role, provenance scopes, candidate edges with relation types, status (candidate versus accepted), and a pointer to any implementing path if known. Miss provenance and you cannot falsify it. Miss role and every edge collapses to related-to. Miss status and advisory sketches pollute navigation as if they were settled law.
That record is deliberately heavier than a quote card and lighter than a chapter. It is the IR grain between bronze territory and runtime use — small enough for one cognitive move, rich enough to connect without reopening the whole ebook.
Functions without a build-and-test loop still drift. The next chapter adds the directions: forward from idea into implementation, reverse from behaviour into principle, and the cross-check that turns mismatch into lint.
Key takeaways
- Claims are functions; edges are calls; interfaces need identity and provenance.
- Allow combinations; record every scope that contributed.
- Debug symbols (prompts, transcripts, tests) make decompilation source-mapped.
Forward, Reverse, Cross
One direction documents hope. Two directions create a test. Three directions make the test honest.
By now the spine is in place: structure supplies addresses, claims behave like functions, edges recover the call graph, and exhaust acts as debug symbols. That is still only reconstruction. Reconstruction without a verification loop is expensive annotation. The distinctive product of semantic decompilation is the round trip — engineering the idea in more than one direction and treating disagreement as a first-class finding.
If you keep conceptual doctrine in one place and implementation evidence in another, you already have both ends of the wire. The mistake is to use each surface only for its local job: essays for persuasion, repos for shipping. The interesting work starts when you ask, of the same idea: what have we said this means? and what have we actually built that behaves this way?
Forward engineering
Forward starts with a conceptual claim and traces it into implementation until behaviour answers back.
↓
selection or agent prompts
↓
deterministic scopes + IDs
↓
model judgment (singles, combinations, overlaps)
↓
rendered artefact + regression evidence
Doctrine predicts shape. If the doctrine says truth remains in source bytes, the render path should not paraphrase. If the doctrine says the deterministic layer makes no interestingness judgment, no pre-filter should be silently ranking candidates. Forward engineering is not “write more docs.” It is a predicted chain of representations, each of which can fail in a named way.
North Star style prompting is often the top of that chain: tight intent, looser method, evaluation moved where it can see outcomes. The point here is not to re-teach that prompting style. It is to treat the prompt package as an engineering artefact that must survive reverse inspection.
Reverse engineering
Reverse starts with what exists: stable IDs, extractors, prompts, scores, cards, tests, logs. From those materials you recover a latent design principle — including principles that were only half-articulated when the system was built.
+ combination support + contextual scoring
+ observed outputs and gates
↓
latent principle
(e.g. structure supplies handles; AI supplies judgment; truth stays in source)
Reverse is where honesty lives. Teams are excellent at forward narratives. They are worse at admitting what the code actually optimises for. A reverse pass that only restates the README is theatre. A reverse pass that can point at a function, a prompt clause, and a failing candidate set is evidence.
Cross-engineering: mismatch as lint
Cross compares both directions:
↔
wiki claims and edges
↔
prompts and code
↔
observed behaviour and tests
When those layers disagree, one of them is lying — or all of them are incomplete. Either way, the disagreement is information. Software already treats round-trip engineering as cross-validation. Idea systems need the same differential test. You are not documenting a thought twice for the pleasure of bureaucracy. You are building a lint neither surface can run alone.
The idea explains the code; the code tests the idea; the discrepancy refines the wiki; the refined wiki improves the next implementation and publication.
What the loop compounds
Done once, the round trip corrects a bug. Done as practice, it becomes how a canon stays executable. Archive material lowers into a joinable IR; agents run against that IR rather than against raw sludge. Claims-with-edges already define the wiki grain that makes reverse results worth filing. Semantic decompilation is the method that keeps filling that IR from real artefacts without pretending a summary was the work.
Component boundaries still matter. Pipeline stages should meet at dumb, stable seams so reverse inspection does not require reading a ball of mud. The three directions are not three teams. They are three passes over one system with explicit comparison points.
Minimum comparison surface
You do not need perfect omniscience to start. You need four sentences you can place in a row: the public claim, the wiki claim (if any), the prompt or code clause that should implement it, and the last observed behaviour that counts as evidence. If any row is missing, that absence is already a finding. If all four exist and disagree, you have a ticket. If all four agree, you have a receipt worth keeping — rare enough to celebrate without turning it into theatre.
Part I ends here on purpose. The method is only as real as a worked path. Next we decompile one ebook scope by scope, then follow one claim into implementation until a discrepancy has nowhere to hide.
Key takeaways
- Forward predicts implementation from doctrine.
- Reverse recovers principle from prompts, code, and behaviour.
- Cross treats mismatch as epistemic lint — the product of the round trip.
Flagship: Decompile One Ebook
Method without retained scope IDs is a speech. This chapter keeps the IDs.
Part I gave you vocabulary. This chapter spends it on one source. We will not re-introduce decompilation as a concept. We will open a structural map, reconstruct units, and leave a trail a second engineer could replay.
The source is a field-guide ebook in the North Star prompting family — long enough to hold multiple mechanisms, written with real heading hierarchy, callouts, and lists. The publishing stack has already produced a denoised HTML form with stable block IDs. Names below are representative of that class of artefact; the method is what must transfer.
Bronze and silver before gold
Source package
- Bronze: full ebook HTML (immutable words)
- Silver structure: chapter regions + block IDs (
nsp-ch4-*) - Not yet gold: reconstructed claims remain advisory until review or repeated use
We focus on Chapter 4 of that guide — the region that argues for tight intent and loose method. A chapter-level summary would flatten several independent moves into one paragraph about “better prompting.” That is exactly the average we refuse.
Scope inventory (deterministic)
| Scope ID | Form | Structural note |
|---|---|---|
| nsp-ch4-h-01 | h2 | Title: tight intent, loose method |
| nsp-ch4-p-03 | paragraph | Mechanism body under h-01 |
| nsp-ch4-box-02 | callout | Specification vs orientation contrast |
| nsp-ch4-li-05 | list item | Failure mode: procedure cosplay |
| nsp-ch4-li-06 | list item | Failure mode: no evaluation surface |
| nsp-ch4-li-07 | list item | Failure mode: goal fog |
| nsp-ch4-p-11 | paragraph | Boundary: audit moves downstream |
| nsp-ch4-combo-07 | combination | h-01 + box-02 selected together |
Notice what the deterministic layer did not do: it did not drop plain paragraphs because they lacked a coloured box. It did not score interestingness. It minted addresses and kept extraction exact. Combination nsp-ch4-combo-07 is not a new invented ID for free-floating text; it is a selection record pointing at two real scopes.
Reconstructions (AI judgment, advisory)
Unit A — nsp-ch4-combo-07
Claim: Purpose should be constrained more tightly than procedure.
Role: mechanism
Significance (advisory): Moves quality pressure upstream to intent, and evaluation pressure downstream to outcomes, instead of micromanaging steps in the prompt.
Edges (candidate):
extends→ concept.north-star-promptingsupports→ concept.attention-budgetcontrasts-with→ concept.procedural-prompt-leash
Provenance: scopes nsp-ch4-h-01, nsp-ch4-box-02 · source ebooks/north_star/index.html
Unit B — nsp-ch4-li-05
Claim: Detailed step lists can simulate control while leaving the goal underspecified.
Role: exception / failure mode
Edges (candidate): warns-against misuse of Unit A; evidences Unit A by negative case
Provenance: nsp-ch4-li-05 only
Unit C — nsp-ch4-p-11
Claim: When method is left loose, evaluation and audit must still exist somewhere explicit.
Role: boundary condition
Edges (candidate): qualifies Unit A; requires concept.outcome-gates
Provenance: nsp-ch4-p-11
Unit D — nsp-ch4-p-03
Claim: Orientation beats exhaustive procedure when the environment is too wide to pre-script.
Role: mechanism expansion
Edges (candidate): depends-on Unit A; implemented-by (pointer reserved for Ch6 path)
Provenance: nsp-ch4-p-03
Units E & F — list cluster li-06, li-07
Claims: (E) Loose method without an evaluation surface is abandonment, not empowerment. (F) Goal fog cannot be fixed by more procedural bullets.
Role: paired failure modes
Edges: both qualify Unit A; E links-to Unit C
Six units, six different relational lives. The chapter summary would have said something true and thin. The decompilation yields premises, mechanisms, exceptions, and boundaries that can be joined separately — which is the operational payoff of addressability already argued next door in Semantic Refraction, applied here as engineering rather than marketing architecture.
Three approaches on the same chapter
| Approach | What you get | What you lose |
|---|---|---|
| Fixed ~500-token window | Retrievable similar text | Cuts across heading intent; no roles; weak provenance |
| Prose summary of Ch4 | One general gist | Exceptions and boundaries averaged away |
| Source-native scopes | Addressable units, combination records, exact re-find | Requires a structural pass and judgment discipline |
The HTML was not formatting. It was a deterministic disassembly map.
Significance notes above are compiled judgment, not source text. They keep a pointer or they are marketing. None of these units are automatically graph citizens. They are working drawings.
What we deliberately left un-promoted
Several scopes in the chapter were reconstructed and then left as satellite material: illustrative anecdotes, transitional sentences, and a metaphor that repeated Unit A without adding a new role. That restraint is part of the method. Decompilation increases addressability; it does not obligate you to mint a citizen for every address. The failure mode is not missing a pretty quote. The failure mode is a map so dense that nothing can be walked.
The next chapter takes Unit A into the implementation path that is supposed to embody it — and finds the place where doctrine and code quietly disagree.
One more discipline on the inventory: every candidate edge above is typed and provisional. extends, supports, qualifies, and warns-against are not synonyms for “seems related.” Vague edges are how decompilation quietly reverts to a tag cloud. Prefer fewer typed candidates you can defend over a long list of soft associations.
Key takeaways
- Retain scope IDs, roles, candidate edges, and provenance on every reconstructed unit.
- Combinations must point at real child scopes, not free-floating rewrites.
- Source-native decompilation preserves distinctions a summary and a fixed window erase.
Flagship: One Implementation Path and a Discrepancy
The claim from Chapter 5 is only honest if the shipping path still implements it.
We already reconstructed Unit A: purpose should be constrained more tightly than procedure, with provenance on nsp-ch4-h-01 and nsp-ch4-box-02. This chapter does not reopen the ebook map. It follows that claim through the implementation path that is supposed to embody the same design choice in the publishing stack: North Star intent, selector prompt, deterministic extraction, model selection, render, regression.
Private repository names do not matter. The path shape does. Treat the following as a generalised implementation repository for attention-shaped compilation from long-form sources — the same family of systems that make quote cards falsifiable against exact source text rather than against a model’s memory of a paraphrase.
End-to-end path
intent: earn attention honestly; exact source text; combinations allowed
2. prompts/selector.md
return scope IDs + reasons; prefer load-bearing tension; no rewrite
3. tools/dom_extract.py
denoised DOM → nsp-ch4-* IDs; whole-scope text by ID
4. model selection run
selects nsp-ch4-combo-07 (h-01 + box-02)
5. render/card_builder.py
card bytes = exact concatenation of selected scopes
6. tests/test_source_fidelity.py
card text must be subset of source scopes; IDs stable across runs
What forward doctrine predicted
- Deterministic layer owns boundaries, IDs, extraction, and recoverability.
- Deterministic layer makes no interestingness judgment.
- AI layer owns judgment of interest, role, and combinations.
- Truth remains in source bytes; the model returns references, not rewritten prose.
- Emphasis markup may appear as hints, never as silent deletion criteria.
Those predictions are the same division of labour named in the pendulum work: ground truth on one side, judgment on the other, components free to move when evidence says ownership is wrong.
What reverse engineering found
Most of the path matched. IDs were stable. The selector returned scope references rather than free prose. Combination selection worked: Unit A’s two scopes shipped together. Regression caught a paraphrase attempt when a prompt regression accidentally asked for “cleaner wording.” That gate is doing real work.
Then the candidate set itself looked strangely thin. Plain paragraphs that should have been eligible almost never appeared. Callouts and high-emphasis blocks dominated every run. The selector prompt still talked about judgment. The data said judgment was happening earlier.
Discrepancy ticket — DET-JUDGE-01
| Doctrine claim | Deterministic layer makes no interestingness judgment |
| Reverse claim | dom_extract.py pre-filters scopes by CSS emphasis class before the model sees them |
| Evidence | Scopes with plain paragraph styling absent from candidate JSON; filter branch logged as “low visual weight” |
| Impact | Quiet premises cannot enter combinations; Unit C-style boundaries under-selected; doctrine lies in production |
| Correction | Remove pre-filter; pass all scopes; keep emphasis as annotation hint only |
| Verification | Re-run: plain scopes appear in candidates; model still prefers high-signal units; exact-byte gate unchanged; combo including a plain boundary scope now possible |
When forward doctrine and reverse-engineered implementation disagree, the disagreement is information.
Why the discrepancy is the product
A documentation pass would have restated the pendulum and moved on. A code review might have called the pre-filter a “performance optimisation.” Only the cross pass forced the question: does this line implement the doctrine or quietly replace it? The answer was replace. The fix was small. The epistemic gain was not: we now know the system can lie in the seam between structure and judgment, which is exactly where decompilers should look first.
Observed behaviour after correction is the other half of proof. The model continues to select load-bearing units; it is not required to treat every plain paragraph as equal in interest. What changed is who decides. Deterministic code stopped pretending to be an editor. AI judgment regained the full candidate set. Source fidelity tests still pin truth to bronze bytes.
Retained trail (minimum proof set)
- Scope IDs:
nsp-ch4-h-01,nsp-ch4-box-02, combination recordnsp-ch4-combo-07 - Prompt pointers:
prompts/north_star.md,prompts/selector.md - Claim: purpose constrained more tightly than procedure
- Edges: extends north-star-prompting; supports attention-budget; contrasts procedural leash
- Behaviour: card bytes match scopes; post-fix candidate set includes plain scopes
- Lint: DET-JUDGE-01 opened, fixed, re-verified
That is an end-to-end decompilation of one ebook region and one implementation path with a documented doctrine/implementation disagreement. Part III generalises what you do with the cascade, the ongoing diff practice, and the build path that does not turn every excerpt into canon sludge.
Key takeaways
- Trace the same claim from scopes through prompts, code, render, and tests.
- File mismatches with doctrine claim, reverse claim, evidence, correction, verification.
- The round trip’s product is the lint finding — not a second brochure about the system.
The Compiler Cascade
There is now a compiler before the compiler. Generated source is often an intermediate artefact, not the origin of intent.
Chapter 6 followed a claim into prompts and code. Zoom out one level and the same shape appears as a cascade of stages, each of which is “source” relative to what sits below it and “compiled output” relative to what sits above. Confusing those stages is how teams reverse-engineer the wrong layer.
Source code before the source code
↓
specification + prompts + context + tests + starting state
↓ AI agent
generated source code
↓ conventional compile / interpret
runtime behaviour
Generated code is simultaneously the compiled output of an intention layer and the source input of a conventional toolchain. That is not a metaphor for poets. It is an operational claim about what you must keep if you want later decompilation to be source-mapped. If the only durable artefact is the generated file, reverse engineering collapses back into legacy mode: watch behaviour, infer purpose, hope you guessed right.
A useful maturity test is brutal on purpose:
Delete the code. Can you regenerate equivalent behaviour from the upstream package?
- No → code still contains trapped source-level judgment. It is not yet a disposable IR.
- Yes → code has become a compiled artefact relative to intent, prompts, tests, and worldview context.
The same cascade shows up in publishing. Working thought compiles into an ebook. The ebook compiles into addressable units. Units compile into surfaces. Attention-native systems already treat long-form as the falsifiable proof surface behind short interrupts; the cascade explains why that proof surface must remain recoverably structured, not merely archived as a PDF blob.
The prompt is source before source code.
A later piece in this series will sharpen that sentence into its own article — The Prompt Is Source — without needing a live URL yet. Here the point is cascade literacy: when you decompile, name which stage you are standing on. Decompiling only the generated Python while discarding the prompt package is like reverse-engineering a binary after shredding the repository that produced it.
Why cheap layers win demos
Retrieval-augmented generation became popular because it has excellent deployment fit for chatbot products: cheap to stand up, broad coverage quickly, one fast call at query time. A wiki-shaped IR that stores claims and edges front-loads comprehension and looks slower in demos even when it is cheaper per resolved question. Popularity tracks visible cost and product mix, not necessarily richer design recovery.
That observation is not a full attack on metadata arity — that narrow explanation belongs later. It is a warning for decompiler builders: do not confuse the layer that wins the first demo with the layer that can answer what each part does, what it depends on, and whether implementation still matches doctrine.
Keep representations the next decompiler may need
Keep-the-bronze is cascade hygiene. Transcripts that explain why a decision was made, prompts that encoded the intent, tests that pin behaviour, structured HTML that preserves author boundaries — each is a symbol table for a future reverse pass. Delete them to save storage and you will pay in archaeology.
The archive → wiki IR → agent runtime frame already describes how heterogeneous material becomes joinable and runnable. Semantic decompilation is how you keep feeding that IR from real stages of the cascade without pretending the generated middle is the origin of meaning.
Where reverse passes pay first
Not every stage deserves equal reverse attention on day one. Start where drift is expensive: prompts that gate published output, tests that define “correct,” and any generated code that still cannot be deleted and regenerated. Leave low-stakes scripts alone until the high-stakes seams are honest. Cascade literacy is a prioritisation tool, not a mandate to reverse-engineer the universe before lunch.
Once you see stages, you can schedule reverse passes where drift is most expensive. That practice is the next chapter.
If you keep only one mental model from this chapter, keep relativity of source: the ebook is source to the quote card and compiled thought relative to the intent package that produced it. Debates about “what is the real source of truth” that ignore stage are category errors. Name the stage, then argue.
That naming habit also prevents a common tool-shopping failure: buying another retrieval product when the missing asset is upstream exhaust, not a better embedding model.
Key takeaways
- Intent and prompts are upstream source relative to generated code.
- Code is a disposable IR only if regenerable from the upstream package.
- Name the stage you are decompiling; keep bronze symbols for the next reverse pass.
Mismatch Is the Product
Engineers are trained to hide discrepancies. Decompilers are trained to file them.
Chapter 6 produced one ticket. That was not a special-effects sequence for the flagship. It was the unit of ongoing practice. Semantic decompilation without a diff habit becomes a museum of reconstructed claims — neat, dated, and slowly false.
What to run on a schedule
Pick paths that matter: anything that publishes externally, anything that gates money or safety, anything whose doctrine you still quote in public. For each path:
- State the forward doctrine claim in one sentence.
- Reverse from prompts, code, and recent behaviour.
- Write the reverse claim in one sentence.
- If they differ, open a ticket with evidence pointers (IDs, files, logs, tests).
- Correct the lying layer — or revise the doctrine if the implementation discovered a better truth.
- Re-verify and write the edge back into the IR so the next agent does not re-discover the same seam from scratch.
That last step is where decompilation compounds. Filing hard answers back into the map is how walks become cheaper over time rather than eternally heroic.
What counts as a mismatch
- Doctrine says X; code does Y.
- Wiki claim has no implementing prompt or path.
- Implementing path has no claim — behaviour without doctrine.
- Tests pass while published mechanism is contradicted (the tests may be testing the wrong thing).
- A layer assigned to ground truth is quietly performing judgment (or the reverse).
- Reconstructed claim cannot point at bronze scopes it allegedly came from.
Ticket fields that travel
id · doctrine_claim · reverse_claim · evidence_pointers · impact · correction · verification · edges_written_back
When forward doctrine and reverse-engineered implementation disagree, the disagreement is information.
Two silos is not a round trip
The cheap fake is documenting twice without comparing. A beautiful doctrine page and a working repository can coexist for years while drifting. Cross-engineering is the comparison step, not the existence of two folders. If nobody ever places the sentences next to each other, you have dual documentation, not a differential test.
A second fake is the post-hoc LLM story. After a run, a model can narrate why the system did what it did. That narrative is not a causal trace unless it is grounded in the same retained exhaust you would use for reverse engineering: versions, retrieved context, structured outputs, actual mutations. Treat ungrounded explanation as a draft hypothesis, not as the audit log.
Correction can cut either way
Not every mismatch means the code is wrong. Sometimes implementation discovered a constraint doctrine never admitted. Sometimes the public sentence is aspirational and the path is honest. The rule is not “always change code to match the essay.” The rule is: make them meet, on purpose, with a receipt. Either revise the claim, revise the path, or document a temporary known divergence with an expiry. Silent divergence is the failure mode.
Conversation-sized claims help here. A document is the wrong grain for a dialogue with your own system; a claim with edges is small enough to put on the table and large enough to matter. Diff practice inherits that grain: ticket one claim at a time, not one repository at a time.
Shame is the wrong emotion
Teams hide mismatches because mismatch feels like incompetence. In a decompilation culture, mismatch is a sensor reading. The incompetence is refusing to look. Attention-native publishing already treats silence and interrupts as ledger events rather than as calendar obligations; treat doctrine/implementation gaps the same way — rare, explicit, and closed with evidence, not with tone.
Cadence without bureaucracy
A monthly reverse pass on three hot paths beats a quarterly “architecture review” that produces slides. Keep the ticket queue visible. Close tickets with verification, not with vibes. If the same seam reopens twice, the correction was incomplete — raise the test, not the rhetoric. Diff practice scales with the number of load-bearing claims you actually care about, not with the number of files in the monorepo.
You now have the method, a worked path, a cascade map, and a lint habit. The last chapter is the build path that starts tomorrow without flooding the canon with prose.
Publish the diff culture inside the team the way you would publish an incident process: short template, visible queue, no blame theatre. The first time someone is rewarded for filing a doctrine/implementation gap instead of papering it over, the system starts telling the truth without waiting for a crisis.
Key takeaways
- Schedule reverse passes; file mismatches with evidence and verification.
- Two documents without comparison is not a round trip.
- Correction may revise code, doctrine, or both — silence is the failure.
Build a Decompiler Without Flooding the Canon
Smarter decompilation does not begin with a grand platform. It begins with structure you can already name.
The fear is always the same: if we add finer resolution, we will put prose in the wiki. That fear is correct if you confuse reconstruction with citizenship. It is solvable if you treat early outputs as working drawings — source-faithful, advisory, reverse-linkable — and promote only the meaning that proves it can travel.
A practical ladder
- Pick one source with real structure (ebook HTML, not a blob of plain text if you can help it).
- Mint stable scope IDs at author-native boundaries. No interestingness filter.
- Run isolated reconstruction on selected units: claim, role, zero to three candidate edges, significance note with pointer.
- Allow combinations while retaining multi-scope provenance.
- Store working drawings cheaply outside primary navigation citizenship if your map is not ready for them.
- Trace one implementation path that should embody a reconstructed claim.
- File the first mismatch; fix; re-verify; write the edge back.
- Promote normalised meaning through use and review — not every excerpt that scored well once.
You do not need a full semantic decompiler theatre on day one. If you already run a structural pass for selection or publishing, file that output instead of throwing it away after the card renders. The expensive part is often already paid.
Guardrails that prevent sludge
- Claim ≠ quote. Propositions navigate; exhibits prove. Different janitor rules.
- Pointers on significance. Brilliance without a receipt is marketing.
- Deterministic ownership of bytes. Models may judge; they may not silently become the source of truth for text.
- Combination without amnesia. Multi-scope records or it did not happen.
- Promotion through use. Recurrence earns residency; existence does not.
Adjacent work in this series will handle finer points you should not smuggle in here: the arity problem of unary metadata versus relational meaning, and the authority rules for quotes that must never outrank source-backed claims. Name those problems when they appear in conversation; do not pretend this book already settled their architectures. For relational grain and constellation publishing, cite Semantic Refraction rather than re-deriving it.
What you can refuse
You can refuse fixed-window chunking as your theory of understanding. You can refuse summary-as-ingest when the job is design recovery. You can refuse to call reverse engineering complete after reading generated code with the prompts deleted. You can refuse to grant every energetic excerpt a seat in the main map. Those refusals are not aesthetic. They are how a system stays inspectable as AI multiplies both prose and code.
Wiki-shaped memory still wins when the job is synthesis across related questions rather than one-shot chat recall. Semantic decompilation is one of the ways that substrate stays honest: continuously re-derived from real artefacts, continuously checked against behaviour, continuously willing to admit when doctrine and implementation have drifted.
RAG retrieves the source. Semantic decompilation reconstructs the design — and the design is trustworthy only when forward and reverse can be forced to agree.
Close the loop
Start with structure. Reconstruct claims as functions. Keep the debug symbols. Run the three directions. Treat mismatch as a product. Promote meaning, not sludge. If you do only that much, you will already outrun teams that are still arguing about chunk size while their published doctrine and shipping prompts quietly become strangers.
A first week, not a fantasy roadmap
Day one: choose the source and mint IDs. Day two: reconstruct a handful of units, including at least one combination. Day three: pick the claim you will stand behind and find its implementing path. Day four: reverse the path and write the four-line comparison. Day five: open the mismatch ticket or the rare agreement receipt. That is enough to know whether the method fits your stack. Tooling can come after the first honest disagreement.
The reader question this book set out to answer was practical: how do you reverse-engineer the ideas inside prose, prompts, and code without reducing them to summaries? The answer is now operational. Build a source-mapped semantic decompiler. Use round-trip disagreement as a test of both the implementation and the idea. Then keep the source map — because the next decompiler, human or agent, will need it.
If the first week only produces six claims, one combination, one path, and one ticket, you are ahead of teams with elaborate taxonomies and no reverse pass. Complexity is allowed later. Honesty is required first.
Key takeaways
- Working drawings first; citizenship later.
- One source + one path + one mismatch is a complete first system.
- Hold the sentence: retrieve the source, reconstruct the design, force agreement.
References & Sources
The evidence base behind every claim — primary research, industry analysis, and technical specifications
Research Methodology
This ebook draws on primary research from standards bodies, independent research firms, enterprise technology vendors, and consulting firms. Statistics cited throughout have been cross-referenced against primary sources.
Frameworks and interpretive analysis developed by Scott Farrell / LeverageAI are listed separately below — these represent the practitioner lens through which external research is interpreted, and are not cited inline to avoid self-promotional appearance.
Primary Research & Standards Bodies
Microsoft Azure Architecture Center — Develop a RAG solution: Chunking phase [1]
Fixed-size chunking not recommended where semantic understanding matters
https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-chunking-phase
Sarthi et al. — RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval [2]
Flat chunks lose position in larger argument structure
https://arxiv.org/abs/2401.18059
Cornell CS 4120 — Intermediate Representations [3]
Layered IR preserves structure for analysis between source and machine code
https://www.cs.cornell.edu/courses/cs4120/2023sp/notes/ir/
LeverageAI / Scott Farrell — Practitioner Frameworks
The interpretive frameworks, architectural patterns, and practitioner analysis in this ebook were developed through enterprise AI transformation consulting. The articles below are the underlying thinking behind those frameworks. They are listed here for transparency and further exploration — not cited inline, as this is the author's own analytical voice.
Scott Farrell — Semantic Refraction — Why the Pieces Can Mean More Than the Pillar
Relational grain and meaning-complete units as interfaces
https://leverageai.com.au/wp-content/media/articles/152-semantic-refraction.html
Scott Farrell — Attention-Native Publishing — The Article Compiled for an Interrupt
Interrupt doctrine; long-form as proof surface
https://leverageai.com.au/wp-content/media/articles/151-attention-native-publishing.html
Scott Farrell — Text Is the Model's Home Turf
Denoised semantic DOM; Deterministic–AI Pendulum
https://leverageai.com.au/wp-content/media/articles/66-text-is-the-models-home-turf.html
Scott Farrell — The Soft Join
Natural keys and deterministic joins over resemblance
https://leverageai.com.au/wp-content/media/articles/88-the-soft-join.html
Scott Farrell — Keep the Bronze
Never discard a representation a later pass may need
https://leverageai.com.au/wp-content/media/articles/92-keep-the-bronze.html
Scott Farrell — AI Legacy Takeover
Behaviour as oracle when the intent stream was lost
https://leverageai.com.au/wp-content/media/articles/48-ai-legacy-takeover.html
Scott Farrell — North Star Prompt
Tight intent, loose method; orientation over procedural leash
https://leverageai.com.au/wp-content/media/articles/70-north-star-prompt.html
Scott Farrell — Your Life Compiles to One Language
Archive as source, wiki as IR, agent as runtime
https://leverageai.com.au/wp-content/media/articles/104-life-compiles-to-one-language.html
Scott Farrell — The Index Is the Data
Claims and edges as the navigable IR
https://leverageai.com.au/wp-content/media/articles/63-the-index-is-the-data.html
Scott Farrell — Hora's Watchmaker
Stable components joined by cheap mechanical seams
https://leverageai.com.au/wp-content/media/articles/74-horas-watchmaker.html
Scott Farrell — Cache the Significance
Significance is compiled judgment that must keep a pointer
https://leverageai.com.au/wp-content/media/articles/90-cache-the-significance.html
Scott Farrell — RAG Was Built for Chatbots — Agents Need a Wiki
RAG fits chatbots; wiki stores claims and reasoned edges
https://leverageai.com.au/wp-content/media/articles/69-rag-was-built-for-chatbots-agents-need-a-wiki.html
Scott Farrell — File Back the Walk
File multi-hop answers so the map improves through use
https://leverageai.com.au/wp-content/media/articles/80-file-back-the-walk.html
Scott Farrell — Conversation Is the REPL
Claim-with-edges as dialogue-sized unit of thought
https://leverageai.com.au/wp-content/media/articles/132-the-conversation-is-the-repl.html
Industry Analysis & Vendor Research
Andrej Karpathy — LLM Wiki [4]
Claims-and-edges knowledge base with lint for contradictions and orphans
https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f
Aspire Systems — Reverse Engineering with AI [5]
Recovering business logic from legacy code when intent symbols are missing
https://www.aspiresys.com/blog/digital-software-engineering/agile-software-solutions/reverse-engineering-with-ai-will-generative-models-unravel-30-year-old-codebases/
Neo4j — How to improve multi-hop reasoning with knowledge graphs and LLMs [6]
Vector search weak on how facts connect across hops
https://neo4j.com/blog/genai/knowledge-graph-llm-multi-hop-reasoning/
About This Reference List
Compiled July 2026. All URLs verified at time of compilation. Regulatory documents and standards specifications are subject to revision — check primary sources for the most current versions.
Some links to academic papers and vendor research may require free registration. Government and standards body publications are freely accessible.