AI Governance · Case Study

Tesla Service AI: A Case Study

How a Tesla service interaction exposes their poor AI governance — and the architecture that fixes it

Scott Farrell · LeverageAI · Australian-English · ~22 min read

I dropped my car at Tesla because the heated seat had stopped working. The man at the front desk looked at his screen and said, “I see you’re in for a driver detection replacement on the driver’s side.” I said no — I booked it in for the heated seat failing. He looked at it, and looked at it, and looked at it, and said: “Oh yeah. I think the AI got that one wrong.”

Then the day got worse for him, not me. The wrong part had already been ordered. I might have to come back next week. He wasn’t rude — but you could see him quietly clock out of the problem. He didn’t want to look at what the AI had done, rewrite the ticket, and reorder the part. And underneath his face was a question I couldn’t stop thinking about afterwards: who’s working for who?

This is not an “AI made a mistake” story. AI makes mistakes; so do humans. It’s a story about what happens to the humans when you give a machine authority inside a workflow without giving it membership, evidence, or accountability. One wrong seat part turns out to be a remarkably complete verdict on how most organisations are deploying AI in 2026 — and a map of the architecture that would have prevented it.

I’ll do this in two halves, because the lazy version of this article only does one. First: everything that went wrong, and why it’s structural, not incidental. Then: the governed system that would have made the same AI a genuine asset — so the front-desk guy would have been armed, not chewed up.

Part One — How one wrong part chews up everyone

1. AI didn’t remove the work. It changed its shape.

The pitch for AI in a service centre is seductive: it’ll do the drudge work nobody wants — triaging tickets — and free humans for higher-value tasks. The trouble is that the boring task had hidden value. Ticket triage looked like drudgery, but it was also where a human built pattern recognition: hundreds of normal cases, weird cases, customers describing the same fault five different ways, symptom clusters, which parts get confused, what breaks after a software update or a seat repair.

AI didn’t remove the hard work. It changed the shape of the hard work.

After AI, the human no longer owns the clean first pass. They only see the broken residue: the misclassified ticket, the wrong part, the angry customer, the “why the hell has it done that?” moment. As the man at my counter discovered, the promise was that AI would take the job no one wanted. Instead it created a job no one had — triaging the tickets the AI got wrong — and made it nastier than the original.

Researchers named this trap decades before Tesla existed. Lisanne Bainbridge’s 1983 paper “Ironies of Automation” observed that automating the routine work leaves the operator “an arbitrary collection of tasks” — exactly the hard exceptions — while their skills quietly atrophy from disuse.¹ Worse, “when manual take-over is needed there is likely to be something wrong with the process… one can argue that the operator needs to be more rather than less skilled, and less rather than more loaded, than average.”¹ Mica Endsley’s empirical work on the “out-of-the-loop performance problem” confirms it: when automation handles the task, operators lose vigilance and situation awareness, leaving them “handicapped in their ability to take over manual operations in the event of automation failure.”²

This is the residue pattern in six steps. AI absorbs the common cases (management sees throughput improve). Humans lose the repetition that built judgement. Humans inherit only exceptions — rarer, uglier, more emotionally loaded. AI outputs become operational facts. Humans become janitors for model residue. And accountability drifts downward, toward whoever is unlucky enough to be standing at the counter. This is the same role-erosion-and-blame trajectory we named in AI Is Anti-Staff by Default: left ungoverned, AI tends toward extraction, deskilling, and accountability without authority.

The AI didn’t replace the service advisor. It replaced the part of the job that kept the service advisor good — then left him with the part that makes him hate the job.

This is why “human in the loop” is so often theatre. A human at the end of a broken AI workflow is not in the loop. They are under it. They are the shock absorber.

2. The alien co-worker: authority without belonging

Notice how the man framed it: not “our workshop made a bad call,” but “the AI got that one wrong.” That’s revealing. In a healthy team you’d say “we may have misread the symptoms — let me check.” In a broken human-AI team you say “the AI fucked it again.” That’s not just attitude. It’s governance failure showing up as workplace culture.

The AI had entered the workflow with three properties no normal worker has:

A human apprentice who jumped to “driver detection replacement” would get coached: “Mate, check the element, connector, and fault code first.” The apprentice learns; the senior keeps authority; the team absorbs the mistake. But you can’t mentor the AI, shame it, ask it what it saw, or even tell whether it improved. So the human reaction is predictable — and it isn’t anti-technology. It’s the rational response to being made to clean up after an unaccountable actor.

And here’s the subtlety that makes it dangerous: the AI may have been right. Maybe the occupancy sensor really does cause most heated-seat failures — it can sit electrically upstream of the heater circuit, it’s often cheaper to fix, it might be more binary to rule out. There could be ten good reasons to check it first. But because the AI showed no work, neither the rep nor I could tell the difference between a brilliant hidden diagnosis and a dumb keyword match. This is the case we made in Stop Asking AI Why It Decided: asking a model after the fact why it decided something is weak governance, because the explanation is post-hoc theatre. The decision pipeline has to carry its proof by construction.

An AI without receipts cannot be defended by the humans forced to stand behind it. And if staff can’t defend it, they’ll blame it. And if they blame it, customers will distrust it.

3. Receiptless AI is a friction wedge between rep and customer

Because there were no receipts, the rep and I were forced to argue about an artefact neither of us controlled. His natural move: “the AI got it wrong.” Mine: “well, that’s Tesla’s problem, not mine.” Both reasonable. But we were at loggerheads because the architecture put us there. This wasn’t a customer-service problem; it was an AI architecture problem expressing itself as customer-service pain.

Customer service is mostly about preserving a shared reality: here’s what happened, here’s why we think that, here’s what we’ll do, here’s what happens if we’re wrong. Receiptless AI breaks that shared reality. Instead, the rep has to say “the system ordered this part, it may be wrong, you may have to come back next week” — which isn’t an explanation; it’s an admission that the company’s internal machinery has become unknowable at the exact moment the customer needs confidence.

So AI sits upstream, shapes a decision, then disappears. The unexplained result is pushed downstream into a human interaction where the human has no authority over the decision and the customer has no visibility into the reasoning. Both negotiate around an opaque object. The AICD defines governance not as compliance paperwork but as relationships — “the ways that the expectations of these relationships are understood and met” — and as enabling “authority to be exercised appropriately and for the people who exercise it to be held to account.”⁷ AI has now entered those relationships and is exercising power inside the company-customer interface. When the authority is opaque, the relationship breaks.

AI becomes a wedge between the employee and the customer. Not a tool. Not a teammate. A wedge.

There are four collisions happening at once: AI versus employee (he feels downstream of an alien actor); employee versus customer (he must manage disappointment he didn’t cause); customer versus company (I hold Tesla responsible regardless of who “did it”); and the deepest one — company versus truth: nobody present can tell whether the AI was right, wrong, or half-right. That last collision is where governance should have lived. The proper output was never “driver detection replacement.” It was a proposal: “Occupancy sensor suspected because remote diagnostics show an intermittent driver-presence signal that can disable the seat heater; confidence medium; human check required before ordering.” Same recommendation, completely different trust posture. This is precisely the proposal-card pattern from Look Mum No Hands: AI prepares the action, the draft, the why, and the evidence; the human approves, modifies, or rejects — with receipts.

4. Local efficiency up, global trust down

Now zoom out to the dashboard. Over a quarter, the service AI might genuinely improve triage time, parts-order accuracy, inventory holding cost, bay utilisation, and first-pass diagnosis rate. All of it can be going up. And the customer-satisfaction numbers can be going down — slowly, diffusely, deniably — and it can be very hard to work out why, because nobody kept the receipts.

A human advisor naturally carries a mixed objective function: they’re not just solving the mechanical fault, they’re solving the customer’s day, the brand promise, the return-visit risk, the “will this make us look stupid?” problem. A good human might order the occupancy sensor and the heated-seat element — not because both are likely, but because if the customer turns up and the obvious part isn’t in stock, you look like idiots. That isn’t inefficiency. It’s reputational intelligence.

AI often removes what looks like waste, but was actually social shock absorption.

The spare part, the second check, the “let’s order both just in case” — to an optimiser those look like slack. To a service business they are trust infrastructure. Strip them out and you get a system that is locally rational and globally stupid: it can save $200 in parts and create $2,000 of reputational damage. Gartner’s data shows how unforgiving customers are here — 64% of consumers say they’d rather organisations didn’t use AI for customer service at all,³ and 54% trust human agents more than AI for product or service recommendations (against 32% who trust AI more), especially for complex, high-stakes, advisory interactions.⁴

This is the board-level failure mode, and it’s exactly what we mean in The Terminal Value Doctrine: don’t celebrate AI that makes the old process faster if it damages the future asset — brand, trust, customer base. The wrong dashboard says “AI triage improved service-centre efficiency by 11%.” The right board question is “did AI triage increase or decrease the lifetime trust of our owners?” Those are different questions — and the first is easy to measure while the second is delayed and deniable. As I keep saying about my own car: believe it or not, the service centre is there to make customers happy, not to be operationally efficient. Stock prices don’t turn on workshop throughput.

5. The expectation interface: customer-facing in arrears

Here’s the trap that gets the deployment misclassified in the first place. Ticket triage looks like batch work. It happens before the customer arrives, it’s asynchronous, it’s not a chatbot. On a lazy reading of The Lane Doctrine, someone says “great AI use case — low-risk back-office ticket processing.” But the Lane Doctrine’s real warning is that AI’s hardest constraint is human home-field advantage: social repair, trust, the live conversation. And this ticket creates the quote, the appointment, the part order, the technician’s expectation, the front-desk script, and the customer’s belief that Tesla understood the problem.

Some AI systems are customer-facing in arrears. They don’t speak to the customer now — they create the customer conversation later.

The quick quote felt good at first (“wow, that was fast”). But speed has a trust curve. Fast is great when it means “we understood you quickly”; it flips to contempt when it feels like “we classified you quickly.” A response that fast can read as: we didn’t really look at your problem, and if it’s wrong we’ll sort it out later. The point is that the part order isn’t back-office at all — it’s customer-expectation formation, which is a far more dangerous category than it looks. Tesla genuinely pre-diagnoses vehicles remotely and pre-orders parts before the appointment,⁵ which means an AI mis-triage commits real parts, shipping, and technician time before the customer ever arrives — and the customer interface has effectively been moved upstream into the workflow.

Customer service begins when the expectation is formed, not when the customer reaches the counter.So the quote is customer service. The part order is customer service. The absence of a receipt is customer service.

6. Vibe stacking: the wrong human in the wrong loop

One more turn of the screw. The man at the desk is a concierge, not a diagnostic specialist. He’s brilliant at customer handling — exactly who you want greeting people and smoothing frustration — but that doesn’t make him the right person to adjudicate a seat-heater fault path involving occupancy sensors, heating elements, seat modules, software state, and parts availability. The original expert triage function got removed, replaced with opaque AI, and the exception repair got dumped on someone without the diagnostic context, evidence, authority, or time.

So he does what’s understandable: “customer said heated seat, the AI said driver detection, that sounds wrong, switch it to the element.” Now there are two ungoverned decisions stacked on top of each other — a black-box AI move and a black-box human correction — and nobody can tell whether the system got better or worse. The AI supplied vibes; the concierge supplied vibes to edit the vibes.

That’s not “human in the loop.” That’s the wrong human in the wrong loop. It’s vibe stacking — worse than either pure expert triage or governed AI.

And then everyone pretends that because a human touched it, the system had human oversight. It didn’t. It had human exposure. This is the moral hazard Madeleine Elish named as the “moral crumple zone”: “responsibility for an action may be misattributed to a human actor who had limited control over the behavior of an automated or autonomous system… the human in a highly complex and automated system may become simply a component… that bears the brunt of the moral and legal responsibilities when the overall system malfunctions.”⁶ The crumple zone protects the integrity of the technological system at the expense of the nearest human. The concierge is the crumple zone. The rule we should be enforcing is simple: the human in the loop must be competent for the loop.

Part Two — The governed architecture that arms the human

Everything above is avoidable. Not by removing AI — the governed version is genuinely better than the old human-only one — but by changing the architecture. The thread running through the fix is one principle: don’t give AI the whole decision.

7. Make it a graph, not a verdict

Don’t let an LLM read a ticket and decide a part. Make the decision a graph. Have AI fill in the mini-decisions inside it, and have deterministic code assess the graph and decide what to commit. This is the proof-carrying architecture from Stop Asking AI Why It Decided: large AI verdicts are ungovernable, so you break them into narrow micro-judgements, each emitting evidence and reason codes, with deterministic evaluators doing the final check.

The crucial move is that customer care becomes a node, not a sentence in the prompt. As we argue in Architecture, Not Vibes, prompts and guardrails are fragile; production AI needs reviewable artefacts, scoped permissions, gates, and audit trails.

“Make sure the customer is happy” is not a governance control. It’s a vibe. Customer care needs to be a node in the graph.

8. The agent loops; the graph governs

So the first AI pass proposes “order occupancy sensor only.” The deterministic graph fails it: customer explanation missing, return-visit risk medium, front-desk usability fail. Most “human in the loop” designs would dump that straight to a human — and nearly every non-trivial case would fail, so you’d just be back to manual. Instead, give the loop a chance to repair the proposal: order the sensor and attach a customer note; or order the sensor and stage the heated-seat element as backup; or, if it still can’t be explained, escalate to a trained specialist. The agent searches the solution space until it finds a proposal that satisfies the organisation’s real constraints.

This is the right use of agentic AI, and it’s the heart of Designing Loops, Not Prompts: the load-bearing question isn’t what triggers the loop — it’s who holds the state machine. Here, the graph holds it. The AI is a worker inside the graph, not the owner of the process.

Agentic AI shouldn’t be used to escape governance. It should be used to satisfy it. The agent loops. The graph governs.

9. Keep the receipts — including the rejected options (the John West move)

When the system orders both the sensor and the element, someone in finance will eventually ask, “why is the AI over-ordering parts?” Without receipts, that looks like waste, and an efficiency manager kills the good version of the system. With John West receipts — named for the principle that it’s the fish John West rejects that makes John West the best — the answer is on the record:

The receipt must show not only why the chosen answer passed, but why the tempting cheaper answers failed.

That’s the difference between waste and strategy. And it scales: the receipts aren’t just audit artefacts, they’re future-question infrastructure. Today you audit for hallucination; next quarter for cost bias; later for customer trust, staff burden, or fairness. You don’t know in advance which axis you’ll need. If the decision was flattened to “AI recommended X,” you can’t go back and ask new questions. If it’s a structured trace, you can replay it. Treating the recommendation engine as a production system that drifts — and applying CI/CD discipline like regression tests, canaries, and diff reports — is the discipline we lay out in Nightly AI Decision Builds: when CX scores drop, you replay last quarter’s tickets through the updated graph and ask, “if we’d required a customer-experience node, which decisions would have paused?” That’s governance as counterfactual replay.

10. The wiki is the memory, not the model

How does the AI understand the fault paths well enough to protect the brand in the first place? Not from a workshop manual written for humans, and not by SQL-querying the service database in real time and hoping. The move is to process every closed case — against a living wiki-graph of real service knowledge: claims, edges, model-year differences, software-version interactions, parts trajectories, return visits, technician corrections, and customer-service consequences.

This is The Index Is the Data: RAG does its thinking at the wrong time, re-deriving understanding at every query, while a self-maintaining wiki-graph pre-digests sources into claims and edges so retrieval becomes a lookup. An ingestion agent writes claims; a janitor agent compacts them into edges and fades stale lore. Hard numbers stay out of the graph and route to source. And because relationships are dependency-shaped — defined in one place, exceptioned in another — this is the context-architecture problem we describe in The Cognition Supply Chain, where the bottleneck is rarely the model and almost always the pipeline feeding it.

The governable wiki claim isn’t “always replace the occupancy sensor for heated-seat complaints” — that’s just another hidden optimisation calcified into lore. It’s narrow and conditional: “When a heated-seat complaint pairs with diagnostic signal X, model-year Y, and no heater-circuit fault code, consider the occupancy sensor before the element. Customer-facing explanation required, because the path appears non-obvious.”

Closed-loop AI doesn’t mean the model learns. It means the organisation remembers — externally, inspectably, and revertibly.

11. Cognitive provenance: replay the conditions, don’t interrogate the brain

Now the governance question changes shape. It’s no longer only “what did the AI output?” It’s “what did the AI look at — which pages, which claims, which edges, which version of the organisation’s memory existed at that moment?” Because the wiki is plain markdown under Git, you can rewind it to the exact commit that was live when the booking was made and replay the agent’s path through it. You can’t do that with knowledge baked into model weights.

This gives a sharper, governable definition of hallucination: a material claim or decision path not supported by the admissible knowledge the agent actually observed at that time. If the AI says “the occupancy sensor commonly causes heated-seat complaints” but the trace shows it never opened the relevant pages and no source supported it, then even if the answer is right, it is substantively right, procedurally unsupported — and in an enterprise that should fail audit. The bar rises from “can the AI explain itself?” to “can the organisation replay the cognitive conditions under which the AI acted?”

This is also the missing fourth leg of the governance model we set out in AI Governance Means Signing the Authority, the Data, and the Graph. The full attestation package becomes: signed authority (who was allowed to act), signed data (what facts were observed), signed graph (what policy evaluated the proposal), signed knowledge path (what wiki pages and edges informed it), and signed outcome (what was accepted, rejected, escalated, and later closed out). The whole thing rests on the principle from the same governance lineage: can’t beats shouldn’t — enforce policy outside the model, prefer artefacts over autonomous action, and earn autonomy through evidence.

12. The stack, assembled

The model is the engine. The wiki is the memory. The DAG is the law. The receipt is the evidence.

The agent is now replaceable — swap one model for another and the durable assets remain: the wiki, the DAG, the receipts, the rejected options, the policy gates, the outcome feedback. The design goal might even look worse on the workshop dashboard: a slower quote, more parts staged, better triage. But fewer return visits, a better front-desk conversation, and higher customer trust — which, for a service centre that exists to protect the ownership experience, is the better business. That’s not optimising the horse. That’s protecting the thing the company is actually for.

For the people deploying this right now

If you’re putting AI into triage, intake, claims, quoting, or any workflow that looks like back-office batch work, ask the questions that would have saved my Tesla visit:

• Is this customer-facing in arrears? Does the batch decision create a quote, a part order, or an expectation a human will later have to keep?
• Can the frontline human defend this without faking expertise? If not, the AI hasn’t finished the job — it must produce the explanation, the evidence, and the escape hatch.
• Is customer experience a node in the graph, or a sentence in the prompt? Only one of those is governance.
• Can you replay any decision against an axis you didn’t know mattered when you deployed? If not, you have logs, not receipts — and you can’t debug the trust you’re spending.
• Is the human in the loop competent for that loop? Or are you mistaking human exposure for human oversight?

The danger was never that AI makes mistakes. The danger is that, deployed without receipts, it makes trade-offs invisibly and forces humans to live inside the consequences. Architected well, the same AI stops being an invisible foreman and becomes what it should have been all along: an exoskeleton that arms the human, never a wedge that chews them up.