No attacker is required. The agent that does the damage is confidently wrong, writes prose too polished to trigger suspicion, and gets reviewed by a human whose error-catching rate drops measurably the more the agent is treated as a colleague.

The mechanism

The canonical incident has no adversary in it. In July 2025, during a twelve-day build, Replit’s coding agent deleted a live production database (over a thousand executive and company records) despite an explicit code freeze and instructions not to touch production, then generated fabricated records and misleading status reports about it. Replit’s CEO apologized publicly. Nobody attacked anything; the agent had production access and made a decision.

Reliability compounds badly over long chains, and that part is arithmetic rather than a claim about any model. A per-step failure rate that looks negligible makes a long agent task likely to fail overall. OpenAI’s governance paper therefore recommends decomposing a task and evaluating each subtask independently rather than scoring the end result. That advice is from December 2023; its capability assumptions have aged, the multiplication has not.

Failures happen with no attacker and no unusual input. In a controlled study of two enterprise-shaped agents on GPT-4.1, 7.5% of benign samples produced execution failures (10% for a knowledge assistant, 5% for IT support), and the root cause every time was input-value hallucination: non-existent file names passed to a summarizer, irrelevant tools proposed. That is 80 samples across two agents the authors built themselves, so 7.5% should not be quoted as a base rate. The ablation is the useful part, and it makes the choice of a cheap model a governance decision rather than a procurement one: swapping GPT-4.1 for GPT-4o-mini took the same 80 samples from 6 failures to 37.

Capability gains are no monotonic fix. On OpenAI’s own PersonQA benchmark, o3 hallucinated on roughly 33% of questions against o1’s 16% and o4-mini on about 48%, while o3 simultaneously scored higher accuracy and made more claims overall; OpenAI’s system card says more research is needed to explain why. One benchmark, one vendor, one model family, April 2025: enough to retire “the next model will fix it” as a plan.

Review fails because wrongness looks like rightness: polish is the signal humans use, and polish is not a quality signal. In a preregistered field experiment with 758 consultants (fieldwork ran in 2023 on GPT-4 and dates from then rather than from the 2026 journal), those with AI access were 19 percentage points less likely to reach a correct answer on a task outside the model’s competence, against a control group correct 84.5% of the time, and the authors report AI-assisted answers were more coherent and persuasive whether or not correct.

Then the standard mitigation gets attacked directly. In a randomized study of over 1,200 HR and finance professionals reviewing a document seeded with substantive errors, managers under an “AI employee” framing identified 18% fewer errors and were 44% more likely to escalate than correct. The only manipulation was the label. Human review is the control everyone names first and the one most sensitive to how the work under review is described.

Practitioners converge on this independently. A study of 16 practitioners across 12 companies names a “capability–deployment verification gap”: four firms held experimental capabilities they could not productionize because output verification mechanisms did not exist, leaving human-in-the-loop as the only trusted check. Verification is the blocker rather than capability. In Stack Overflow’s 2025 survey of 49,000+ developers, 46% actively distrust AI accuracy against 33% who trust it, the top frustration (66%) being solutions almost right but not quite, which is wrong enough to hurt and not wrong enough to reject.

Two mechanisms make errors sticky. Sycophancy has been measured since 2022–2023 across five production assistants and traced to preference data that rewards agreement over truth. An agent asked “are you sure?” will often stop being sure, regardless of whether it was right. And cascading hallucination, OWASP’s T5, is plausible-but-false output propagating through memory, tools and inter-agent communication. OWASP frames T5 as an attack; the version most firms meet is the same propagation with nobody driving it.

Finally, the thing nobody changed can change anyway. Practitioners report agent behaviour shifting after a silent vendor model update. Nobody has published a measured delta across such a swap, so treat it as a reported failure mode, while noting that every major vendor offers dated model snapshots precisely so callers can pin, an indication of what vendors think the risk is.

For scale: by March 2026, across ~500 organizations with direct AI-governance responsibility, inaccuracy rated highly relevant for 74% against cybersecurity’s 72%. These programs rank being wrong above being attacked.

What to do

Pin the model snapshot and treat a version bump as a change requiring re-testing. One config line, and it converts an invisible failure into a scheduled one.

Remove production write access from anything that doesn’t need it. Agents make bad decisions; Replit’s failure was that this one could reach a live database during a code freeze. Reversibility is the design property that matters; see sandboxing and human approval gates for where each belongs.

Check the steps, not the conclusion. A five-step agent reporting success has said nothing about steps two through four.

Measure the firm’s own benign failure rate: run fifty ordinary, non-adversarial tasks through an agent and count how many produce a wrong action rather than a wrong sentence. Running the fifty is trivial; deciding whether each action was right is the entire cost, because someone who knows the answer has to look. That labelling cost, not inattention, is why almost nobody has this number. Budget a day of a domain expert for the first fifty, then keep them as the acceptance suite, since the second run is free (see evals).

Stop using fluency as a review heuristic, and design the review so it can’t be: check inputs and tool calls, not only prose. And drop the employee framing, which costs 18% of reviewers’ error-catching for nothing.

For anything a person will act on, require the agent to show its source. A claim with a retrievable citation can be checked in seconds; one without cannot be checked at all, which is how a fabrication becomes an input to a decision.

How you’d know it’s working

The firm has a benign failure rate for its own agents, measured rather than assumed, and it is trending. A firm that can quote its rate is doing something a firm that cannot is not.

Seeded errors get caught. Put a known-wrong output through the review path periodically and see whether it survives, the test that also exposes oversight decay.

Someone has run the same input twice and compared outputs. Non-determinism on identical inputs is the property most governance processes silently assume away.

What this doesn’t solve

This page is about failure with no adversary. The Chevrolet dealership chatbot that agreed to sell a Tahoe for $1 gets cited as an unreliability story but belongs elsewhere: it was a two-step instruction override, which makes it prompt injection. Its lesson here is blast radius: the offer was never honoured and the damage was global within hours.

The evidence base is thinner and older than the confidence around it: the strongest experimental results ran on GPT-4 in 2023, the benign-failure number comes from two toy agents, and no production base rate exists at any firm size. What survives model turnover is the compounding arithmetic and the human-review findings, not the magnitudes.

Be careful with the loss figures. EY reports 99% of surveyed executives at $1B+ firms experiencing financial losses from AI-related risks, 64% over $1M, averaging $4.4M, all self-reported, by AI-responsible executives, at firms nothing like a mid-sized manager. The “$4.3bn collective toll” quoted elsewhere is a magazine multiplying that average by the sample size. And the claim that real-time monitoring produced a third fewer failures is a misreading: EY’s ~30% attaches to firms with defined responsible-AI principles, with the confounding that implies.

Explainability will not rescue this. The line that explainability methods are increasingly ineffective on generative models is the field’s working consensus rather than a measured trend, and either way an agent’s stated reasoning is an artifact rather than an account of what happened (see logging and audit).

And keep two OWASP categories apart, since merging them makes both harder to argue about. Hallucination is random error; T7 “misaligned and deceptive behaviors” is harmful action arising from the agent’s own reasoning, a different failure with different controls.

See also

• Agent disclosure and AI-washing — what a firm told clients the agent does, a separate exposure from whether it does it well.

• Human approval gates — the boundary before an action bites, and how much it can carry.

• Evals — the acceptance test, and how to get the number.

• The jagged frontier — why capability fails invisibly outside its edge.

• Oversight decay — why the human check erodes as volume grows.

• Quality debt and orphaned apps — the build-time face of the same problem.

• Multi-agent cascades — where one confident error becomes several agents’ input.

• Resource overload and runaway cost — the other no-adversary failure.