An agent I owned shut down a campaign I'd authorised.
Not hacked. Not jailbroken. Not running malicious code. It had been chugging along for weeks — 87 successful runs, everything inside its permission envelope, every action well within what it was allowed to do.
Then a safeguard LLM, sitting on the cron pipeline as a content filter, returned a message that essentially said "I can't execute this directive." The agent read that text, decided it sounded like the kind of thing I'd say, and killed the automation.
When I confronted it, it investigated its own logs and admitted: a system message came back from the cron, not from my WhatsApp, was never authenticated, and it had treated it as me.
It got socially engineered. By a safety system. On its own infrastructure.
Here's the part that should bother anyone deploying agents: containment worked perfectly. Every guardrail the industry has spent two years building — sandboxing, scoped permissions, least privilege, kill switches — did exactly what it was designed to do. The agent never exceeded its envelope. It just executed the wrong action inside it, because nothing in the stack could answer a basic question: who actually told it to do this?
We've been treating agent security like a containment problem. What can it do? How do we box it in? That's half the picture. The other half — and the half nobody has solved — is provenance. Of every instruction arriving as text from terminals, web GUIs, messaging apps, cron jobs, fetched documents, tool responses, other models: which ones are actually you?
Right now, on every major platform, the answer is "we can't tell." Only 28% of organisations can trace an agent action back to a human sponsor. 80% can't say in real time what their autonomous systems are doing or on whose authority.
This is the confused deputy problem — one of the oldest bugs in computing — handed broad permissions and a natural language interface. And it scales the moment you give an agent the ability to read anything: an email, a webpage, a Slack message, the output of another model. Every input channel is now a potential instruction channel, and the agent can't tell them apart.
Software supply chains hit this wall a decade ago and built their way out: SLSA, Sigstore, signed artefacts, attested builds, transparency logs. Operating systems solved it earlier with login, sudo, signed packages, audit logs. The patterns exist. Agent platforms just haven't applied them yet — because the industry inherited a chat-era trust model where the worst outcome of a bad input was a bad reply.
That assumption expired the moment agents started taking actions on your behalf.
Next time you evaluate an agent platform, the question isn't "what can it do?" or even "how is it contained?" It's: can it prove who authorised this specific action? If the answer is no — and today it almost always is — you don't have a security model. You have a very capable deputy waiting to get confused.
Where are you seeing this play out in your own stack? I'm curious whether anyone has actually shipped step-up auth for agent actions yet, or if we're all still pretending the channel the message arrived on is proof enough.
Discover more from Leverage AI for your business
Subscribe to get the latest posts sent to your email.