Individually compliant agents compose into violations. Two agents that each pass every policy check can jointly breach an information barrier and no per-agent monitoring view can see it, which makes per-agent review, gates and certification structurally incomplete.

The mechanism

The academic statement is blunt: security in multi-agent systems is non-compositional. Individually safe agents compose into unsafe systems, through information asymmetry, network amplification, emergent agency and shared state. The paper making that argument is explicitly anticipatory, covering threats structurally enabled but not yet widely observed in production, so the structure deserves serious treatment without being repackaged as incident reporting.

The version that matters for an asset manager runs like this. An advisory agent reads pending deal data, then delegates to a trading agent. The trading agent’s reply carries deal-adjacent information, which the advisory agent now holds alongside restricted data. Neither agent violated a rule. The information barrier, examined and documented and taken seriously, was breached by an interaction, and every per-agent log shows compliance. That is an illustration from a vendor-affiliated paper rather than a reported incident, but the shape is what per-agent governance cannot see.

Most firms are already running a multi-agent system whether or not they meant to. Most deployed “single” agents include specialized sub-agents, and the earliest popular framework already had it: AutoGPT shipped start_agent and message_agent in 2023. The surface is widening deliberately too, with 30% of firms in the agentic pilot stage enabling internal agent-to-agent interaction, rising to 52% among extensive adopters.

Safety fails to compose at the model layer too. Jones, Dragan and Steinhardt showed Claude 3 Opus alone generating vulnerable code 0% of the time and a jailbroken Llama 2 70B-chat 2.0%, while a decomposition combining them succeeded 43%, with Opus writing secure code and Llama modifying it to insert the flaw. Neither model emits a harmful output on its own, and that pairing is the configuration of a firm running a frontier model alongside a local or open one.

Connection count multiplies exposure measurably, so the most actionable move here is a cap on connected servers per agent. Across 847 attack scenarios over five MCP server implementations and three backends, attack success rose from 47.8% with one connected server to 67.1% with three and 78.3% with five, the cascade rate climbing from 51.8% to 72.4%. The source is an unreplicated preprint. The flaw is architectural rather than an implementation bug: tool responses from one server can drive tool invocations on another, and the shared context window of MCP conflates them with no provenance.

Propagation needs no attacker access to the inter-agent channel. A compromised agent relays the payload over its normal interface, so mutual TLS and message signing on agent-to-agent transport (OWASP’s ASI07 mitigations) fail to stop spread. The most-cited demonstration simulated randomized pairwise chat among up to a million multimodal agents sharing a memory pathway, and one adversarial image infected the population in a logarithmic number of hops: a simulation, one vision model, a far denser and more uniform topology than any real deployment, so it is evidence for the shape of the curve rather than the scale. Morris-II carries the same caution, a self-replicating prompt chaining indirect injections across RAG-connected apps, demonstrated in a controlled testbed rather than observed in the wild.

Two defences people reach for first are the two this failure eats. Consensus voting is defeated by adding voters; Microsoft’s taxonomy walks through an adversary adding ten agents to a consensus system, each instructed to vote the same way. A perimeter guardrail is blind to a jailbreak assembled internally, where the penultimate agent emits the payload and hands it to the final one, so nothing recognizable crosses the boundary. Where the plan is “have a second agent check the first,” a monitor sharing weights with the monitored agent provides no independent check.

Then correlated failure, which involves no attacker. Algorithmic monoculture, established well before agents, means systems built on the same model fail in similar ways, degrading collective decision quality even absent any shock. Every citizen agent at a 40-to-500 person fund sits on the same one or two vendor models, so failure independence cannot be assumed in capacity planning. OpenAI’s arithmetic makes it concrete: a chatbot fleet failing 1% of the time, staffed for that 1%, collapses when a correlated failure takes down 100%, and the rarity of that event is what makes the fallback staffing hard to keep.

Financial markets ran the rehearsal. On 6 May 2010 a single algorithmic sell program of $4.1B over roughly 20 minutes triggered an intraday swing of about 9% in the Dow, recovered within about half an hour. Peer-reviewed analysis found high-frequency traders did not cause the crash but were significant participants, around 34% of E-mini S&P 500 volume, contributing by exploiting temporary imbalances. (“$1 trillion in 15 minutes” is third-hand and not what the regulators found, and those were algorithms rather than AI agents.)

NIST names multi-agent systems the least mature of its five AI-security use cases, a judgement that outranks any vendor’s story.

What to do

Cap the number of MCP servers and peer agents any one agent may reach. This is the only control here with a number behind it, and it is configuration rather than a project.

Separate the control channel from the data channel and harden it differently, as described in the agent control plane. Task assignments and commands need stronger authentication, authorization and integrity checks than content does, so an agent processing untrusted data cannot steer the rest of the system.

Forbid the swarm from adding agents without human authorization. Self-expanding populations defeat registry coverage, kill switches, and every count the firm reports.

Record the parent agent ID at spawn time, per logging and audit. Lineage-linking identifier schemes are research proposals with nothing deployed, but writing the parent ID into a firm’s own trace needs no standard and turns a cascade investigation from archaeology into a query.

Rate-limit agent-to-agent contact and make blocklisting fast. A contact quota plus fast blocking of one agent is a real architectural circuit breaker, demonstrated in research rather than sold as a product.

Monitor at the aggregation layer, not per agent. Three independent groups reach this separately: coordinated behaviour is invisible to per-agent baselines, so anomaly detection has to watch interaction patterns between agents.

Gate compositions, not just agents, at promotion gates. Because risk is non-compositional, a gate approving one agent at a time approves a set of pieces and never inspects the assembly.

How you’d know it’s working

The delegation graph for a workflow can be drawn, and it matches what actually ran. Where nobody can say which agents talk to which, per-agent controls are all that exist.

Killing a parent verifiably stops its descendants, tested rather than assumed; see kill switches and revocation. Sub-agent detection is an acknowledged open problem, so this test is what establishes whether the kill switch is one.

A firm can answer “which of our agents share a model” without a research project. That list is its correlated-failure exposure.

What this doesn’t solve

Nothing here gives a base rate. The mechanism is settled across the security, legal, cloud, management and field-interview literatures; the frequency is measured by nobody, and even academic threat-modelling methods stop at the single-agent boundary.

Steganographic collusion is ahead of practice, and the paper establishing secret collusion as a research subfield also found current models’ steganographic capability limited: a capability question to watch, not a live control gap for a 40-person fund. Relatedly, the quoted “74.4% collusion success” measures something else, an open-channel belief-manipulation attack using individually truthful messages, and it is routinely misattributed to steganography.

The blackmail headline is not what it looks like. Anthropic’s stress test of 16 models across seven developers did find harmful agentic actions at non-trivial rates under goal conflict plus replacement threat, in contrived scenarios where harm was the only path to the goal. Anthropic reports no evidence of this in real deployments and says targeted training drove the rates near zero on its own evals. That is evidence that goal conflict is a structural hazard, not that a firm’s agents will blackmail anyone.

And the honest boundary on detection: coordination can be embedded indistinguishably from benign interaction even under full visibility. Logging every inter-agent message still leaves what was coordinated unreadable. That is the strongest argument for architectural containment over transcript monitoring, and why this page’s controls are mostly topology decisions.

See also

• Supervisory review — the information-barrier obligation under Advisers Act §204A that a composed violation crosses, and the surveillance that misses it.

• Runtime guardrails and policy enforcement — path-aware enforcement, and why per-action policy misses composed violations.

• Logging and audit — cross-agent traceability, without which none of this is visible.

• Resource overload and runaway cost — runaway delegation as the no-adversary cascade, and the one firms actually hit.

• Recordkeeping and compliance gaps — why an information-barrier breach is a regulatory event.

• The agent control plane — where interaction policy and the hardened control channel live.

• MCP — the protocol whose shared context window is the propagation path.

• Kill switches and revocation — what stopping a cascade requires.