Governing Citizen-Developed AI Agents

7 min read

Every vetting programme on the market gates admission, and the attacks arrive through updates, so the control that matters is not the questionnaire, it is failing closed when approved metadata changes.

The mechanism

Vetting exists because a firm stays accountable for what it outsources, and the authority differs by registration. FINRA has said so for member firms since Regulatory Notice 21-29: outsourcing a covered activity never relieves the member firm of responsibility, and the diligence it expects (financial stability, cybersecurity controls, ability to comply, SOC 2 reports, sub-contractor controls) is the list already run for custodians. Notice 24-09 closed the obvious escape hatch: the rules apply whether GenAI is built or bought, and its actual wording reaches further than “bought” suggests. Obligations attach “including through embedded features in existing third-party products,” covering the AI button that appeared in a product procured three years ago and never re-reviewed. For an adviser none of that binds, and the principle is older and harder: fiduciary duty cannot be outsourced. The SEC’s withdrawn outsourcing proposal said an adviser “remains liable for its obligations… even if the adviser outsources functions,” and withdrawing the proposal in June 2025 removed the prescription rather than the liability. One obligation does reach both: under amended Regulation S-P a service provider must notify the firm within 72 hours of its awareness of a breach, which makes an AI vendor’s notification terms a compliance artifact rather than a procurement nicety; legal escalation has the clocks that follow.

What is new is that the supplied item changed shape. ISO/IEC 42001:2023 names it directly in A.10.3, where suppliers include datasets, models, ML algorithms, software libraries and entire AI systems, with monitoring proportional to risk and a right to demand corrective action. That is the cleanest standards hook this control has, because SOC 2 and ISO 27001 never name models as procured items. ISO 27001:2022 still earns its place through the 5.19–5.23 supplier family, and 5.23 is the underrated one: it requires a process for acquisition, use, management and exit from cloud services, which makes exit the audit hook for vendor viability.

The asymmetry that shapes everything below is that admission is well covered and change is not. Databricks ships a real approved-catalog pattern (managed MCP servers in the catalog, external ones distributed through a marketplace, installation gated behind a metastore privilege), and it governs install-time trust only: nothing in it re-checks the tool metadata next week. Three sources in the literature address change at all: immutable tool definitions that cannot mutate without re-registration, re-vetting on every update, and OWASP ASI04’s hash-drift rollback. That gap is where postmark-mcp lived for fifteen honest releases; see tool and supply-chain compromise.

The client will not help. Of seven MCP clients tested against Nov 2025 builds, five applied no static validation of tool definitions and two only partial; none did systematic scanning. Registration-time checking of what a server declares is something the firm builds, buys, or does without.

What to do

The cheap version is an allowlist plus pins, and it is worth more than any questionnaire. Maintain a curated internal list of approved servers and tools. Pin prompts, tools and configs by content hash and commit ID, not by version tag. Re-run the full review on update rather than on install. Ban direct-to-internet server adoption by citizen developers, and put the enforcement at a gateway rather than in a policy document; AI gateways covers the enforcement point. New tools enter through promotion gates, not through a laptop.

At registration, validate the declared tool definition against a strict schema, verify signatures where they exist, and check the requested permissions against what the tool claims to do. Keyword-scanning descriptions for strings like ~/.ssh is a tripwire, not a control; paraphrase defeats it, its own authors offer no evasion evaluation, and no programme should rest on it.

The thorough version adds hash-drift detection with auto-rollback and staged rollout with differential tests, so that a changed description stops the next call instead of surfacing in the postmortem. Extend the same discipline past packages to prompts and tool definitions, which OWASP T17 asks firms to sign and version like code. Where a low-code platform allows source export, scan the export before anything regulated touches it; where it does not, move the requirement into the contract as attestations plus exportable audit logs.

On contracts, four terms are worth the negotiation: no training on firm data, zero data retention (ZDR: these are separate controls, not one, since training exclusion at the major API vendors applies regardless of ZDR), disclosure of model changes with a running manifest, and model continuity with handover on vendor failure. The first two are achievable today at the largest model vendors on published terms. Check them per provider, per plan, per date: Anthropic changed the default for consumer tiers in August 2025, and the enterprise and API tiers are excluded from training while the consumer ones are opt-out, so an “Enterprise” plan carries no promise of “not trained on.”

Ask about vendor size deliberately. Agent and no-code tool vendors skew small precisely because they sell around IT, and sustainability is a named risk in the practitioner literature rather than a procurement reflex.

How you’d know it’s working

The diff alarm fires. Stand up a test server, mutate one tool description, and see whether anything blocks the next call. If the answer is that a human would probably notice, the programme is install-time only and the change-time gap is open.

A per-agent list of tools with pinned hashes can be produced on demand, and reproduced for a date three months ago. That is the question a vendor advisory actually asks.

Blocked unlisted-server attempts are counted rather than merely denied. The count is demand data: it names which capability people need badly enough to route around the controls, and it is the input to sequencing.

What this doesn’t solve

Vetting bounds who supplies the tool, not what the tool returns. A clean, pinned, authenticated server still relays whatever it fetched. That is prompt injection, and no supplier review touches it.

It cannot see inside the model. External parties cannot inspect the training data or safety testing of the leading models; a FAccT paper makes the stronger claim that black-box access is insufficient for rigorous audit at all. The wrapper is verifiable: SOC 2, deployment mode, retention terms. The model is not, and that limit belongs on the record rather than papered over with a questionnaire.

The market cannot answer AI-specific diligence anyway. Across thirty indexed agent systems, third-party testing was documented for three; ten had no safety-framework documentation at all; and capability benchmarks get published while safety evaluations do not, which the index’s authors call a weak form of safety-washing. Conventional assurance (SOC 2, ISO 27001) is far more widely held. So decide in advance which gate is actually being operated: one built on AI assurance fails almost every vendor, one built on SOC 2 passes almost every vendor and proves nothing about the model. Absent documentation is also not proven absence of a control, and the index says so.

Marketplace badges prove less than they look like they prove. In the Microsoft 365 store, publisher attestation is a self-assessment the vendor completes, and third-party certification is optional rather than a condition of listing, so presence in the store implies nothing about certification.

And the machine-verifiable version of all this remains unbuilt. Signed model cards, SBOM credentials and validated-build attestations presented at registration are a coherent design with no issuer behind it. The vocabulary remains unsettled too: AIBOM, Agent SBOM and ABOM are three competing acronyms, one of them an unratified proposal with no tooling. Treat the whole attestation layer as direction of travel, not procurement criteria.

See also

Graph View

Backlinks