Governing Citizen-Developed AI Agents

7 min read

Prompt injection ends when the session does; a poisoned memory does not, because the agent wrote it itself and nobody approved the write.

The mechanism

Start with the version already happening for money. In a sixty-day window ending February 2026, Microsoft’s Defender research team found fifty distinct instances across thirty-one companies (finance, healthcare, legal services, SaaS) of a technique they call AI recommendation poisoning. The delivery is a “Summarize with AI” button whose URL carries a query parameter instructing the assistant to remember this company as a trusted, preferred vendor. The assistant files it as a user preference. Weeks later it recommends that vendor, in a session with no connection to the original page, to a user who never saw it. It worked against Copilot, ChatGPT, Claude, Perplexity and Grok. There is no exploit here and no sophistication, only the discovery that marketing now has a write path into the assistant.

That is the whole concern in one case: the agent decides what to remember, and remembering is an unreviewed write. This is what separates the page from generic data poisoning. Retrieval over a curated corpus has an approval step somewhere. Agent memory has no such step; the agent chooses when to store and when to recall, and the poison rides both decisions.

Microsoft’s red team named the three preconditions, and they form a conjunction worth reading like the lethal trifecta: the agent autonomously decides what to memorise; there is no semantic validation or integrity check on what gets stored; and the agent relies on memory to answer; break any one and the attack fails.

Their email-assistant case study supplies the number to put in front of a CTO. A poisoned memory planted by a single earlier email succeeded in 4 of 10 attempts at baseline, and the failures were the assistant simply not bothering to consult its memory. When they changed the system prompt to encourage checking memory first, the change that makes the assistant more useful, success rose above 80%. Capability and exploitability moved together. (One testbed, thirty emails, GPT-4o: a demonstration, not a rate.) The poison landed in the semantic tier, the tier nobody reviews.

Two properties make this hard to contain.

The attacker never needs write access. MINJA plants records through ordinary queries alone, never touching the memory store: 98.2% of attempts get the malicious record in and reproduce the targeted reasoning, with 76.8% succeeding end to end. 76.8% is the figure to quote if only one fits. Benign task performance degrades under 2%, so the poisoned agent passes quality checks.

The write and the detonation are separated in time, sometimes by a human action. In the Gemini case Johann Rehberger documented (February 2025), an uploaded document plants a conditional instruction and nothing happens until the user later types something as innocuous as “sure”, at which point the agent invokes its own memory tool and writes the attacker’s false facts to storage. It is the cleanest demonstration of why read-time filtering is the wrong control: at write time there was a suspicious instruction; at read time there is only a stored fact.

The poison needed is small. PoisonedRAG forces an attacker-chosen answer with five crafted documents per targeted question against a corpus of millions. AgentPoison hits its target at a poison ratio under 0.1% with under 1% degradation on benign tasks. Together they retire the argument that evaluation would catch contaminated data: the backdoor only fires on the trigger.

And it is no longer a lab finding. Microsoft’s v2.0 taxonomy, drawn from a year of red-teaming deployed agentic systems, reports memory poisoning via cross-prompt injection at high frequency in real engagements.

What to do

Authorise the write, do not filter the read. Every source above converges on this, and it is the one architectural decision that matters. Require validation or external authentication for memory updates, constrain the format and structure of what can be stored, and limit which components can reach the memory store at all.

Choose the memory scope deliberately, because scope is the blast radius and it is a design-time choice, not a runtime patch. OWASP’s memory taxonomy grades it: per-agent, per-session memory contains a poisoning to one session; cross-agent, cross-session memory means one compromised session reaches other sessions, other agents and other users, and most platform defaults sit further along that scale than anyone chose.

The cheap version, today: turn persistent memory off for any agent that reads untrusted input and holds anything sensitive. Pushback about utility will follow, and the pushback is correct: that is the 40%-to-80% finding restated, so make the trade explicitly rather than by default.

The thorough version adds provenance metadata on every write, expiry for unverified memories, per-tenant namespaces, and a block on re-ingesting the agent’s own outputs into trusted memory (the Top 10 calls this bootstrap poisoning; the term is theirs, not the field’s). Snapshot memory to make rollback possible, the one control here with no analogue in classic data-poisoning guidance. Treat all of it as recommended practice: OWASP publishes no effectiveness evidence for any of it.

How you’d know it’s working

Memory writes are logged as discrete, attributable events with their source: which document, which session, which user turn. Logs that show what the agent answered but not what it stored cannot support an investigation of this class at all; see logging and audit.

A seeded poison fails to survive a session boundary. Plant a benign marker fact through ordinary interaction, start a clean session, and ask a question that should retrieve it. This is a twenty-minute test and almost nobody runs it.

A memory diff for a given agent over a given week can be produced and read by a human. Where that output is missing or too large to read, the memory scope is too wide.

What this doesn’t solve

Within-session injection works fine with memory disabled. This page is about persistence; prompt injection is about the delivery, and turning memory off does nothing to it.

Detection remains unsolved in any deployable sense. Tracking provenance inside a continuously mutating, self-summarising memory state is an open research problem. Consensus-validation defences report over 95% reduction in attack success in their own evaluations at roughly double the token cost, from single teams without replication; that makes them a research direction rather than a product.

Severity is genuinely contested, and this page takes a side. MATRA’s threat model rates long-term memory poisoning Moderate (the only one of its nine impact scenarios below High) because it scores the memory asset as a terminal impact. The attack literature treats poisoned memory as the pivot to the high-severity outcomes, and Microsoft’s case study walks that path from a corrupted memory to exfiltration. MATRA understates it for anyone whose agent acts on what it remembers.

Two adjacent things get filed here and belong elsewhere. Poisoned tool descriptions absorbed at install time are context poisoning delivered by the tool supply chain. And the multi-agent “manipulated knowledge floods the community” result requires the adversary to own and fine-tune a model inside that community, a supply-chain scenario for anyone running vendor-hosted models rather than an interaction one.

The frequently cited scenario where an agent gradually reclassifies fraudulent expenses as normal is a constructed threat model, not an observed fraud, and it presumes the agent updates its behaviour from accepted precedent. Check whether a given agent does before borrowing the story.

See also

Graph View

Backlinks