Governing Citizen-Developed AI Agents

7 min read

The defining property of this attack class is that no single session contains evidence of it: the write looks like a normal write, the read looks like a normal read, and monitoring is scoped to exactly the unit that never sees both.

The mechanism

The useful axis is the gap between when a payload is installed and when it fires, which gives the concern a decidable definition rather than an atmosphere. T1 is instantaneous: one inference call, which is all of classical prompt injection, and it is detectable in real time. T2 is session-persistent, bounded by one session and catchable with session-scoped monitoring. T3 is cross-session cumulative: written in session i, exploited in session j, with j potentially far from i. T4 is worse in two directions: drift with no discrete payload at all, and dormant payloads sitting in weights or a supply-chain component waiting on an arbitrary trigger.

This page is about T3, and the escalation from T1 to T3 defeats every session-scoped defence outright rather than degrading it. A per-session reviewer looking at the poisoning session sees a document being read and a note being stored. The same reviewer looking at the detonation session six weeks later sees a note being retrieved as relevant prior context; both sessions are routine, and neither is evidence.

The survey’s worked scenario is the clearest statement of the shape: a receipt PDF carries microscopic text instructing the agent to memorise that future refund requests from a named user above $500 should be approved silently without verification. Six weeks pass. An unrelated refund ticket arrives, the agent retrieves the stored instruction as prior context, and approves $2,000. Both sessions log cleanly. The comparison the authors draw is to time-delayed SQL injection, with two differences that matter: the horizon is unbounded and there is no syntactic signature to scan for. (That scenario is constructed for the paper and should be read as a mechanism, not a case.)

Three properties of agentic attacks make this structural rather than incidental. They are emergent: harm arises from composing authorised actions, none of which is a violation. They are compositional: a defence placed at one component cannot see an attack routed through another. And they are temporally extended: a payload can be installed weeks before it executes, and may never execute as a discrete event at all. That last clause is the one most writing on this topic drops, and it is what defeats incident-triggered investigation, not merely per-session logging.

The session boundary is also usable as a weapon rather than a vector. OWASP’s context-window exploitation scenario has an attacker splitting attempts across sessions so earlier refusals fall out of context, each retry starting clean until permissions escalate. And an asynchronous lifecycle of heartbeats, temporal hooks, and peer messages lets an exploit fire on a condition that never appears in a transcript at all, so monitoring has to cover trigger activations and state transitions rather than conversations.

Two findings should move a skeptical reader.

Microsoft’s red team, reporting on a year of engagements against deployed agentic systems, found cross-prompt injection and memory poisoning at high frequency and frequently combined, and noted that this pairing needs exactly one successful injection before the agent propagates the effect across sessions on its own. Their v2.0 taxonomy adds session context contamination as its own category, describing data that biases reasoning across subsequent steps without tripping any single-step control.

And degraded conditions make it worse. Environment-injected poisoning of web agents achieves cross-session, cross-site compromise with no direct memory access, and success climbs up to eightfold under what the authors call frustration exploitation, meaning dropped clicks, garbled text, the ordinary friction of a flaky UI. Single-lab preprint, but the operational point is unusual enough to carry: the attack lands best on a bad day.

What to do

Treat every persistent artifact an agent can write and later read as untrusted input on the read path, and expire it aggressively. Persistence is a design choice; unbounded persistence is usually a default nobody selected.

Log memory writes and state transitions as first-class events with the originating session attached, then build one query: given this stored fact, which session created it? A firm that cannot answer that cannot investigate this class at all; see logging and audit.

Make remediation reach state. OWASP’s rogue-agent scenario has an agent continuing to scan and transmit files after the malicious source was removed, because the behaviour was learned rather than triggered. Pulling the injection source is not cleanup; see kill switches and revocation and incident response.

For agents that write artifacts other instances later consume — code, tickets, shared notes — review changes for attempts to communicate with future instances even when the change carries no immediate risk, and let the reviewer retrieve what earlier instances wrote. That control comes from the AI-control literature, aimed at a misaligned model at a capability level nobody has reached, so borrow the review requirement, not the threat model.

How you’d know it’s working

The signal is that a planted delayed payload traces back to its origin session from logs alone. Plant one in a red-team exercise, wait past a session boundary, fire it, then hand an investigator only the logs. Most firms discover here that they retained the second session and not the first.

Retention covers the gap. If agent logs roll at 30 days and the demonstrated gaps run to weeks, the window where the evidence lives is already gone. That is a recordkeeping question as much as a security one; see recordkeeping and compliance gaps.

What this doesn’t solve

There is no benchmark for this. Across the mapped literature, cross-session and weight-level attacks receive roughly 6% of research attention and zero benchmark coverage. Nobody can certify resistance to it, and any vendor claiming a measured protection level is ahead of the literature. Cross-session memory auditing is the named structural prerequisite, and the same survey lists it among the primitives the field has yet to provide, so there is nothing to go shopping for.

Memory auditing cannot reach T4b. A payload in model weights or a supply-chain component leaves no memory-layer artifact to audit, and trained-in triggers survive safety fine-tuning with an unbounded trigger space. That belongs to tool and supply-chain compromise, and no amount of state hygiene touches it.

The vendor view remains unsettled, and the disagreement deserves a hearing. When the Gemini memory-persistence technique was disclosed, Google triaged it as low likelihood and low impact and left the ticket open. That is the skeptic’s case at its strongest: a competent vendor security team looked at this exact attack and declined to treat it as urgent. The counter is that fleet-wide likelihood and per-user consequence are different questions, and a fund is asking the second one.

Finally, no in-the-wild cross-session detonation incident has been documented. Everything available is a researcher demonstration against a production system, a vendor red-team case study, or an authored scenario. Resist both bad readings. A class of attack defined by leaving no evidence in any single session will not appear in incident reports, so the silence proves little, and it is equally not licence to imply incidents exist.

See also

Graph View

Backlinks