Governing Citizen-Developed AI Agents

7 min read

Two things surprise people here. The clock that matters most starts at the vendor’s awareness rather than the firm’s, so detection latency is already spending someone else’s budget. And the conversation about the incident is probably not privileged — including, where it was typed into an agent, the part everyone assumed was.

The checklist

1. Write the trigger list before it is needed. Escalation is an examinable governance element: supervisors ask “who approves vendor relationships and how risks are escalated,” and the expected answer is a document, not a habit of calling counsel when something feels bad. The list should name who decides, not only what qualifies. At minimum: any agent touching customer information, any agent output that reached a client, any capability change on a tier-2-or-above agent, any vendor incident notice, and any request from a regulator. IOSCO’s supervisory question bank puts “incorrect outputs” alongside breaches and biases as potentially notifiable, so a hallucination with client impact is on the list too.

2. Know which clock applies, and name it. Three different 72-hour deadlines appear in this material and they are unrelated: the Reg S-P service-provider notification, Form PF current reporting, and the contractual term a firm imposes on a vendor. Saying “we have 72 hours” without saying which will mislead everyone in the room. The two that govern an agent-caused data exposure at an adviser: the service provider must notify the adviser within 72 hours of the provider’s awareness, and the adviser must notify affected customers as soon as practicable and no later than 30 days after the adviser’s awareness that unauthorised access has occurred or is reasonably likely to have occurred. Note that the trigger is likelihood rather than confirmation. Compliance dates were 3 December 2025 for larger entities and 3 June 2026 for everyone else, so this is live.

3. Treat the clock-start asymmetry as a contracting problem. The vendor’s 72 hours run from the vendor’s awareness. The adviser’s 30 days run from the adviser’s. A provider that sits on a notice burns the adviser’s budget before anyone at the firm knows a budget is running, and receipt of that notice starts the firm’s own incident response. That is why an AI vendor’s notification SLA is a compliance artifact rather than a procurement nicety; see supply-chain vetting. Note also that delegating the notification work leaves the duty where it is: the logistics can be handed over, the obligation cannot.

4. Be precise about privilege, in both directions. Two separate points, and most firms get both wrong. First, entering text into a hosted AI tool can waive attorney-client privilege, because the act of entering it is disclosure to a third party. The failure mode is the disclosure itself rather than any leak by the model. A chat with an agent about pending litigation, an internal investigation or the firm’s own regulatory exposure is the specific thing to avoid, and it is exactly what an eager builder will do first. Second, and pointing the other way: compliance reporting carries no privilege. The Commission has said that reports required by its rules “are not subject to the attorney-client privilege, the work-product doctrine, or other similar protections,” because they are meant to be available to the staff. Read the scope narrowly: that statement sits in a recordkeeping footnote about fund board packets and CCO annual reports under Rule 38a-1, it is an assertion in an adopting release rather than a judicial holding, and it stops short of converting every compliance document into a public one. The counterweight belongs in the same breath: staff responses in the examination program are generally kept confidential under Advisers Act §210(b). Non-privileged still falls short of public, and candour with a firm’s own compliance function remains the right call.

5. Escalate on aggregation, not only on severity. The sharpest answer to “each of these was minor” comes from the Commission itself: serious issues go up promptly and cannot wait for an annual report, and “individual compliance matters that, taken in isolation, may not be material may collectively suggest a material compliance matter, such as a material weakness.” Five trivial agent misfires sharing one root cause are one material weakness, and they cross the threshold together; without a periodic aggregation review on the calendar, nothing surfaces the pattern.

6. Report the event, not the response to it. The Commission considered limiting reporting to matters that resulted in remedial action and declined, out of concern that firms “might abuse the limitation and fail to impose remedial actions in order to avoid having to report.” Reportability turns on the event regardless of whether anyone fixed it. A team that quietly patches an agent and reports nothing has selected the one path the regulator explicitly anticipated.

7. Price the self-report. In the off-channel communications sweep, individual penalties ran from $600,000 for a firm that self-reported to $12 million for firms that did not. That spread is the economic argument for a fast escalation path, and it is a number a CTO can take to a CFO who is asking why this needs a process.

8. Give counsel the architecture, not just the incident. Legal reviewers need enough mechanism to allocate liability. A lawyer who cannot see where the autonomy sits cannot advise on it. Brief them on the agent’s trust zone and tool access once, in peacetime, rather than during the call. See the reviewer pool.

How you’d know it’s working

Someone can state, without looking it up, which clock applies to an agent that emailed a client list to the wrong address, when it started, and who decides whether it is notifiable.

The firm’s AI vendor contracts contain the 72-hour notification term, and someone has checked rather than assumed. If the contract is silent, the clock still runs.

At least one incident has been escalated on aggregation rather than on its own severity. If none ever has, either the firm is unusually clean or nobody is running the aggregation review.

Nobody has pasted an investigation into an agent. This is unmeasurable directly, so it belongs in training rather than in monitoring; see training curriculum.

What this doesn’t solve

This page is the escalation path rather than the incident mechanics. Containment, evidence and the technical sequence are incident response.

The privilege points are stated at the strength their sources support and no further. Counsel writing on AI say information entered into these tools “can be exposed to third parties, thus waiving attorney-client privilege,” and cite no authority for it; we could not find a decided case applying waiver doctrine to a hosted AI vendor. The direction of the risk is clear, the doctrine remains unsettled, and anyone claiming otherwise is ahead of the law.

The two secondary sources describing the Reg S-P attorney-general exception disagree about whether it is a law-enforcement or a national-security delay. The primary rule text settles the 30-day and 72-hour clocks, which we verified; we did not resolve the exception’s exact scope, and it is narrow enough that it should not feature in a firm’s planning.

Whether the firm must notify a regulator of an agent failure is largely unanswered in the US. There is no adviser equivalent of a mandatory AI-incident report; the withdrawn Rule 206(4)-9 would have created a 48-hour Form ADV-C filing, and it never took effect. Firms with EU exposure have DORA, where a major ICT incident carries reporting timelines and client notification, so the US position should not be assumed to hold across every entity.

This page assumes an adviser. Broker-dealers have additional escalation duties, including SAR filing where cyber events surface suspicious activity, and FINRA has cited firms for failing to escalate red flags detected outside the AML programme.

See also

Graph View

Backlinks