Eight Rounds: The Security Reviewer That Rewrote Its Own Method

I have a security-review pipeline for WordPress. It is unglamorous: it walks an inventory of plugin files, hands one immutable PHP file at a time to a model, and stores what comes back. By the time this experiment started it had completed 7,206 file jobs — and produced 1,523 results marked needs_context, where the model had seen something interesting and could not finish the thought with one file in front of it.

That is an ordinary state for a scanner to be in. What I did next was not: I pulled my own toolbelt out of the middle of the pipeline, put a different agent runtime in there, and let it loose. I gave it a couple of ideas — build tools that look at more files at once, try different linters and strategies, stop the myopic look at one idea at a time — then told it it could do what it wanted, as long as it heavily reviewed four or five plugins the scan had already flagged.

And I added one instruction that mattered more than all of that:

I told it to keep a log of the ideas it tried. Try an idea, test it on the projects. If you like the idea, keep it. If not, reject it. So I can have a look through later at what it's done.

What came back was not a list of vulnerabilities. It was a dated, append-oriented experiment log in eight rounds, in which the agent repeatedly decided that the method of reviewing was inadequate and that it would improve the method while pursuing the review — proposing a change to the apparatus, running it against real code, inspecting the failure mode, and recording a keep or a reject before moving on.

The final code tells you what survived. The experiment log tells you why it has that shape.

The gap the run is actually about

There is a specific asymmetry in what models are good at here, and it drives every round that follows.

Models are strong at spotting a dangerous primitive — code that would be harmful if an attacker could reach it. Where semantic understanding beats pattern matching they outperform rule-based static analysis convincingly: one 2026 benchmark reports "SQL injection (95% vs. 32% recall), command injection (83% vs. 24%)".² They are correspondingly weak at the second half: proving the primitive is reachable in this deployment, by this attacker, through this path. A June 2026 benchmark of frontier models found that "every frontier model produces 10–50% false positive rates in white-box detection, systematically over-predicting vulnerabilities", and separated confirmed exploitability from a category it labelled theoretical — not realistically exploitable.³

My own database said the same thing more bluntly: 34 high-severity files, several of them vendored-library behaviours with no established path from a WordPress request to the dangerous code at all — a state of affairs the log described as "demonstrating why a dangerous primitive and an exploitable site path must be separated."

So the run's subject is closure, not detection. The eight rounds are eight attempts to get from this code is scary to here is the path, here is the exhibit, and here is what I could not verify — or, just as often, to an honest refusal.

What the agent was allowed to touch, and what it wasn't

The instruction was loose on method and hard on fences, which is the ratio that makes this kind of run survivable. The agent could rewrite selectors, schemas, prompts, validators, migrations, its own scanning tools — anything that shaped how it perceived the problem. It could not move the things that bind: source identity, byte and hash verification, the aggregate budgets, the rule that every cited path and range must validate against the actual supplied bundle, and the decision about what gets published as a finding.

That split is the architecture my sibling piece on agent-native computing argues for: a fluid, machine-native middle where the model may restructure its own cognition, and a narrow authority boundary outside it that is mandatory and not redrawable from inside.⁴ This run is one dated instance of that boundary holding under pressure — evidence, not doctrine. I am not going to re-derive the architecture here.

Infrastructure, for calibration: gpt-5.6-luna via LiteLLM at medium reasoning effort; logging redaction enforced; input manifests and hashes stored, never source. Baseline before the run: 25 pytest tests passing, compileall clean, every dashboard PHP file passing php -l. Goal time used: 4,433 seconds — the only hard performance number this article has, used exactly as the log recorded it.

Rounds 1–2: the unit of reasoning was wrong

Round 1 — follow the model's own file hints

The first hypothesis was the obvious one, and it was mine as much as the agent's. The completed reviews contained 6,517 related-file hints — the model, mid-review, saying I would need to see this other file. Of those, 2,973 were exact inventory paths and 226 more resolved uniquely by suffix. Surely you just resolve those, bundle them, and ask again.

The agent built it properly: same-plugin only, current versions only, per-file byte and hash verification either side of the call, hard budgets, a persisted context fingerprint, unresolved hints retained as coverage gaps rather than quietly dropped, and multi-file results stored separately so nothing was retroactively reinterpreted.

Then it ran the caching plugin. The planner selected the flagged file plus all three exact hints: four files, 158,599 bytes, no unresolved hints. The call succeeded operationally — 46,958 tokens, 21.8 seconds — and returned needs_context at confidence 0.96. It had correctly identified an unauthenticated log-streaming primitive, and correctly rejected an unrelated HMAC-protected route in a sibling file as a red herring. And it could not prove the class it was worried about ever gets registered at all. Independent source review afterwards found the original alert real but conditional. The three files the model asked for were not the reason it failed.

That split — keep the infrastructure, reject the doctrine — is the template for every round that follows, and the first sign the log is doing something a changelog would not: it separates the thing we built from the belief we built it on, and only one of those was wrong.

Round 2 — close the bundle before you ask the question

The change was small and structural. For a standard plugin, deterministically fetch the canonical main file before any model-suggested sibling. Add a versioned semantics pack covering active-plugin loading, init timing, the ordering of admin authentication, menu capability scope, nonces and REST permission callbacks.

The new plan selected five files and 159,788 bytes: the flagged file, the canonical bootstrap, and the three original hints. This time the model returned findings — two conditional unauthenticated log-disclosure findings, context_complete=true, confidence 0.99, 49,764 tokens. It connected the bootstrap's instantiation to the register call, applied the framework fact that init runs before the admin authentication redirect, and kept the logging flags as preconditions rather than suppressing a source-level finding because runtime state was unknown.

One deterministic caller edge plus a page of framework semantics beat three model-selected sibling files. That is not a model-quality result. It is a grain result, and we have a name for it: semantic closure — the smallest deterministic bundle whose meaning closes, reconstructed from the medium's own structure rather than handed over as whatever the storage layer happens to export.⁵ A pipeline that sends file → AI, file → AI, file → AI is using storage grain as if it were semantic grain. Close the meaning first, then ask the model to think.

Rounds 3–4: the conclusion was right and the evidence was broken

Round 3 — the 1212 problem

Round 2 also surfaced a failure no amount of better context would have fixed, and it is my favourite specimen in the whole log.

The finding paths were correct. The reasoning was correct. But six of the model's citations into the main class file had dropped the leading digit from line numbers in the 1200s: the log gate at lines 1212–1235 was stored as 212–235. The validator did not catch it, because it only checked that the cited range existed somewhere inside the supplied file — not that the cited lines contained the claimed code. The log's framing: “This is a persuasive-evidence integrity failure even though the vulnerability conclusion is correct.”

The obvious fix is to tell the model to be more careful. The agent tried the equivalent, and the raw model response in the very next round still dropped the leading 1 — 212, 217, 227. This is what the literature predicts: Google DeepMind and UIUC found that "LLMs struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction."⁶

So the agent stopped asking the model to fix itself and changed the evidence protocol instead. Every location must now carry a minimal verbatim source excerpt; the application searches for it in the supplied redacted source, rejects it if it is absent, ambiguous or over-wide, and derives the authoritative line range from the unique occurrence. Model line numbers are demoted to hints. The same broken 212/217/227 response then persisted correctly as 1212–1215, 1217–1220, 1228–1235 and the rest, and a bootstrap citation given as a three-line range narrowed to the single line that matched. The log's verdict:

This experiment strongly supports excerpt anchoring: asking the model to improve line numbers did not fix its systematic error, but deterministic source anchoring did.

That is the deterministic–AI boundary moving one notch, on evidence, mid-run: line arithmetic went from the model to code, semantic judgment about reachability stayed with the model. It is the mechanism my other sibling piece calls the generative pendulum⁷ — performed here four separate times, unprompted, in the right direction each time. And what the new protocol produces has an older name: an evidence package is a claim, a verbatim exhibit, a resolvable pointer, and a confession of what could not be verified — the discipline that turns a sub-tool from an oracle you must trust into a witness you can check.⁸ A citation of line 212 for code at line 1212 is a dead pointer, and a dead pointer is worse than none: it looks like a receipt.

Round 4 — findings that stand up alone, and the cost of being too strict

The next defect was subtler. The second log-disclosure finding cited a guard and a transform, and its transform excerpt did contain the terminal exit — but it never independently cited the registration path and never labelled a sink. It was leaning on the previous finding's attack path. As the log said: “A stored finding must be self-contained; readers should not have to borrow the previous finding's attack-path locations.” So the schema grew invariants, enforced locally: every finding must independently cite a registration or entrypoint, a sink, and at least one location in the focused root file, and a discovery living only in a related file cannot leak out of a focused root review.

And then the new rule immediately failed, which is the part worth reading. The first run under the stricter schema failed closed because an exact add_action excerpt occurred twice in the file and the model's truncated line hint did not overlap the real one. Rejecting every repeated excerpt proved too brittle to ship. The fix was narrow rather than permissive: use the claimed range only as a bounded disambiguation hint — including a deliberately scoped trailing-digit projection for the observed 1212 → 212 failure — accept only a unique best match, and let ties fail closed.

Three findings then validated, each with its own entrypoint, registration, guard and sink. Two matched manual review. The third — a missing capability check on an admin-side callback — the model rated HIGH and I put closer to MEDIUM, because the bundle never established how a lower-privileged user obtains the required nonce or reaches the hook at all.

Round 5: a verifier that is not allowed to invent

The fifth round produced the one component I would port to another domain unchanged: a separate verification stage with its own prompt, tables and strict schema, and a hash-verified reload of the exact same source versions. It receives a closed set of already-validated claims, and for each must return exactly one of supported, downgraded, unsupported or needs_runtime_test, with a recommended severity and explicit missing preconditions. Crucially: it cannot invent new findings.

On the caching plugin, the verifier supported both log-disclosure findings as conditional on logging configuration, file readability, routing and content. On the third, weaker claim it returned needs_runtime_test at confidence 0.87 — agreeing the callback checks a nonce but no capability and reaches impactful actions, while refusing to assume a lower-privileged role could reach the hook or obtain a valid nonce. That is the conclusion my manual review had reached, arrived at independently, and it converted an overconfident publication candidate into an explicit test requirement without discarding the underlying control weakness.

That terminal state is doing real work. An unknown that arrives typed — with a precise assertion and a named next consumer — is a deliverable rather than a shrug.⁹ needs_runtime_test is not the pipeline failing to answer; it is the pipeline handing you a specific experiment to run.

The caveat the log raised against itself

Here is the part I want to put in front of anyone who is about to build the same thing, because the agent wrote it before I could:

Do not call it model-independent: it currently uses the same Luna model with a different skeptical prompt, so errors remain correlated. Future bake-offs should compare a different verifier model or a deterministic/runtime verifier.

This is the correlated-checkers pitfall, named correctly and refused. Running the same judge twice and treating the agreement as verification is "an echo, not a check" — same model, same blind spots, consensus purchased at double the price. Independence has to be mechanical: a different property, checked a different way, against something outside the model.¹⁰

So the honest accounting is this. The genuinely independent checks were the deterministic ones: hash verification either side of every call, excerpt anchoring against actual source, budget enforcement, schema invariants, and my own manual source review. The sceptical pass added something real — it caught the severity overreach — but it is a second opinion from a correlated mind, not verification. That distinction is why the published output of this run is so small.

Rounds 6–7: two ways of finding nothing

Then the run did something I had not asked for and would not have thought to specify: it deliberately chose specimens that should fail differently.

Round 6 — the reachability control

The first was a negative control: a firewall plugin's vendored cryptography class, flagged high-confidence by the layer-zero scan. The context was minimal on purpose — the 21.9 KB vendored class plus the plugin's 5.4 KB canonical bootstrap. The model returned no publishable finding and needs_context, in 12,825 tokens. It agreed the suspicious primitive exists, then required actual call sites, attacker-controlled input, and a security-sensitive protocol or sink before it would treat that primitive as a WordPress exploit.

That is the correct answer, and it corrects my own database's HIGH rating rather than confirming it. Plugin bootstrap alone does not establish reachability into a deep vendored class.

Round 7 — the absence you cannot prove

The second was a small custom template module on one of my own sites, where the layer-zero scan had raised a HIGH on a shared header template. The agent built an opt-in identifier expander: scan only the root module under explicit budgets, treat PHP as data, mask comments and strings, verify inventory size and hash, store no scanned source, and select a candidate only when it actually includes the template — with variable assignments improving the ranking only when they occur before that include, at the same lexical brace depth.

It picked exactly the right immediate caller, cheaply. The model then removed the original HIGH as a publishable finding — the colour variables were fixed literals, metadata was escaped, a deployment capability it noticed sat behind a separate trust boundary — but stayed at needs_context, because two other template variables had no discoverable producer. My own full-source and runtime review went further and judged the file clean. So the pipeline was correct and more conservative than necessary, and the agent's reasoning about why is the best line in this section: a module-local assignment search cannot prove that inherited include-scope variables are absent. It is a selector, not a safety verdict.

That last rejection is the one I would frame. The agent had a working lexical trick and declined to grow it into something that would look like a call graph and behave like a guess. It wrote down what the real version needs — token-aware parsing, complete-scope coverage metadata, dynamic-construct flags, scan-manifest fingerprints — and then did not build a fake one in the meantime.

Its summary of the three specimens is the experiment-design lesson of the whole run:

Three deliberately different targets were more informative than scanning more similar files: a real cross-file exploit, a deep vendor primitive, and a template false positive.

“A real cross-file exploit” is the log's own shorthand for a source-level path the bundle could close. It remains conditional and unconfirmed. The word is the agent's; the caveat is mine, and it stands.

The audits, and the result I like least

Between rounds, two independent code audits found real problems — bootstrap selection still guessed the conventional main-file path instead of reading WordPress's actual active-plugin entries; budget-skipped hints lived only in database metadata rather than being shown to the model as missing source; a deferred verification result could still raise a discovery's severity. All were fixed before the experiment closed, and the identity chain moved to SHA-256 finalised over ordered path, version, role and digest data.

Then the clean end-to-end canary produced the result I like least and would have been tempted to leave out. On identical source, with the exact-source chain now verified, the same two conditional disclosures came back at MEDIUM, where earlier runs had said HIGH. The discovery pass and the sceptical pass agreed with each other within the run. They just disagreed with the previous run.

The code path and evidence are stable; the label is not.

Model consensus is not a severity rubric. Severity needs an explicit policy — a CVSS-like scale, human judgment, runtime evidence — and this run did not produce one. It produced the discovery that it needs one, which is a smaller and more useful thing than a number would have been.

Round 8: the boundary that more context cannot cross

The last round tested the opposite failure. Not too little context — too much, in the wrong place.

The subject was a connection-management plugin's user-create endpoint: 2.8 KB of code, flagged HIGH by the original triage on the claim that a caller holding only the WordPress create_users capability could supply an arbitrary role. Rather than expand the lexical spike into arbitrary class-symbol resolution — explicitly rejected as manufacturing false certainty — the agent added a controlled bridge: an analyst may supply a source-confirmed exact path, but it must exist in the same plugin inventory, keep every budget and hash check, be labelled analyst-supplied, and participate in the context fingerprint. Missing or cross-plugin paths fail closed.

The first bundle, four files, added the exact helper that creates the user. The model found the role-assignment primitive and published nothing: endpoint registration, inherited authentication and transport attacker control were all missing. So the run escalated properly — a second bounded bundle of seven inputs and 215,890 bytes, supplying the exact registration, the endpoint base, the legacy dispatcher and the request processor. That established a great deal. The endpoint is registered and does pass requested roles through to WordPress's user-creation functions; normal user authentication does check the capability; and the registration also deliberately enables a site-token authentication mode that bypasses that capability check, but only for a request the upstream remote-management service has authorised.

And there the evidence stopped, because the deciding fact is not in the codebase. The local bundle cannot establish the upstream service's authorisation policy, or that a hostile caller could obtain or use such a trusted request. The model emitted no finding and retained needs_context, at 79,457 tokens.

This is the right publication decision: a dangerous privileged primitive at a remote-management trust boundary is not independently a site exploit. The test also showed diminishing returns: adding 195 KB of local framework code could not answer an authorization decision enforced upstream.

This is the one I would put on a wall. Every earlier round was fixed by better context, and it would have been easy to learn the rule “when in doubt, add files” and apply it until the budget ran out. Instead the run worked out that context is only the answer when the missing evidence is inside the audit boundary — and that “not observed within the audit boundary” is not proof of absence outside it.⁹ Naming the boundary is the finding.

The payload: what the log is actually for

The honest tally of what this run published: two conditional, unconfirmed log-disclosure findings on a caching plugin, whose severity label its own runs disagreed about; one control weakness downgraded to a runtime test requirement; and three cases where the correct output was nothing. That is a thin afternoon's security work, and a very good afternoon's method work — the difference is the argument.

The product of the run is not merely "three security findings." It is also a better security cognition machine than existed at the beginning of the run.

The machine is real, and it is also stuck to WordPress. The migrations, the plugin-basename queries, the init-before-auth_redirect semantics pack, the MU-plugin loader quirks — none of that survives contact with npm or PyPI. Hand that implementation to an agent building the same thing for another ecosystem and it will spend the first afternoon carefully porting decisions whose reasons it cannot see.

What ports is the other artefact. Six lines, written as the run went:

Read those again as a set rather than as a list of mistakes. They are not six things that went wrong. They are a map:

not this
not this
this works under these conditions
this fails at this boundary
this requires a different evidence class

Every one of those six lines is ecosystem-independent. “Validating that a citation range exists is not validating that it contains the claimed code” is true of every language with line numbers. “Bootstrap-only context cannot establish reachability into a vendored dependency” is arguably more true of npm than of WordPress. And “stop adding local source when the missing precondition is enforced by another service” is a statement about trust boundaries, not about PHP.

So the answer to what do I feed the next build is not the repository:

You don't want it copying the WordPress implementation. You want it consuming the rejected design space and recompiling the underlying principles.

This is the John West move, applied to an architecture rather than a decision. Keep the options you rejected and why, because “receipts are not just audit artefacts — they are future-question infrastructure”.¹¹ A record of the surviving code answers exactly one question — what did we build? A record of the rejections answers the one that actually comes up: why does it look like this, and what breaks if I change it?

The findings weren't the asset. The rejections were.

What this is evidence for — and what it isn't

Two recent pieces of mine both claim, in the abstract, that an agent can restructure its own machinery inside a run: one on the architecture that makes it safe⁴ and one on why better judgment produces more short-lived deterministic code, not less.⁷ This run is a receipt against both, and subordinate to both — it demonstrates, it does not extend.

On the first: the authority boundary held. The agent rewrote its selectors, schemas, prompts, validators and tooling freely, eight times, and never once got to decide what shipped as a published finding. The gate was narrow, mandatory and outside the cognition — and the fluid middle is where the improvement came from.

On the second: the pendulum swung in both directions, on evidence, four times in one afternoon — line arithmetic and context selection moving from the model to code, reachability judgment staying with the model, severity moving from the model to an unresolved policy question. Nobody planned that sequence.

What it does not do is mint anything. My development wiki already carried two concepts before this run — bounded security-context escalation (broad single-file detection first, then exact-version multi-file proof under hard budgets) and claim-bounded adversarial verification (a verifier receives a closed set of existing claims and may support, downgrade, reject or require runtime evidence, but may not invent new ones). This is the narrative of those two being derived under pressure, not their introduction. No new framework, no new name, deliberately.

And it does not show that the method generalises. One run. One model. One ecosystem. One operator, who was also the one grading it. The distinction that matters is smaller than any of the grand claims:

Old reflexive agent design: the human watches the agent's path and redesigns the architecture from the evidence. New: the agent watches its own path and redesigns some of the local architecture during the run. You still need the human for promotion and durable judgment, but the latency between defect discovery and architectural mutation collapses.

That is the whole delta. I already did this manually — keep the debug, preserve the walks, ask the agent every few days how it used the system, then change the tooling. That loop ran at hours-to-days. This one ran at minutes.

Run it yourself

The instruction costs nothing and you can add it tonight. But the instruction alone gives you a tidy diary, not a research notebook. Here is what has to be in place.

1. Give it real specimens, and make them differently shaped. The most valuable design choice in this run was not mine — the agent chose three targets that fail in different ways: one with a genuine cross-file path, one deep vendored primitive with no established reachability, one where the right answer was that the original alert was a false positive. Four similar targets would have taught it almost nothing.

2. Make the fences deterministic and unmovable. Hashes in and out. Hard file and byte budgets. Validation that every cited path and range resolves against the actual supplied bundle. A schema the model fills rather than authors. If the agent can rewrite the fence, you have a policy that happens to be obeyed, not a boundary.

3. Type your unknowns. needs_context, needs_runtime_test, unsupported, downgraded — each asserts something precise and hands something specific to a named next consumer. An untyped unknown is unbounded labour; a typed one is a deliverable. Here, the typed states are what let three no-finding outcomes count as results.

4. Keep discovery and verification in separate stores, and be honest about correlation. The verifier must not overwrite discovery history and must not invent claims — a verification pass that can add findings is a second discovery pass wearing a sceptical tone. And if that pass is the same model with a different prompt, write it down where the result is stored, exactly as this log did. Treat the deterministic checks as your real independence, and put a mechanically different verifier on the roadmap rather than in the marketing.

5. Write the log hypothesis-first. This is the one that turns a diary into an instrument — the loop the run followed, and the one I would hand to any agent doing methodology work:

propose review idea
      ↓
implement it
      ↓
run against real specimens
      ↓
inspect failure mode
      ↓
keep / modify / reject
      ↓
change the apparatus
      ↓
run next experiment

The order is the mechanism. Because the agent must state what it expects before it runs, every outcome becomes a legible failure rather than a bad vibe — and a legible failure is something the next round can design against. That is why the method improved eight times in one run, and would not have improved at all under an instruction to “review these plugins carefully.”

One posture note on publishing any of it. The review is a finding, not a verdict; it is derived, not primary evidence, and never outranks its own exhibits; it runs parallel to, not in place of, the judgment that decides.¹² Hold those three and the fact that your findings are unconfirmed stops being an embarrassment and becomes the reason anyone can read them.

The part I'd want you to take

The interesting thing that happened on 8 August 2026 was not that a model found something in some heavily-used code. It might have. I genuinely don't know, and neither does the pipeline — which is what it says, at length, in its own record.

The interesting thing is that an agent given a real problem, a tight intent, a loose method and one standing instruction spent 4,433 seconds conducting methodology research while simultaneously applying the methodology, and left behind a dated derivation history: what it tried, what it kept, what it rejected, and why. The code is one ecosystem's answer. The log is the question, worked out in the open, with the wrong turns still in it.

If you are running agents on real systems, add the instruction. Then go and read what they rejected — it is the part you would otherwise have thrown away, and it is the part that ports.

Scott Farrell is an AI strategy and architecture advisor at LeverageAI, working with Australian mid-market leadership teams on governed AI portfolios. He writes about agent architecture, evidence discipline and the AI/deterministic boundary. scott@leverageai.com.au · leverageai.com.au