Governing Citizen-Developed AI Agents

7 min read

Governance effort should scale with autonomy × authority × criticality, per agent. That model has been arrived at independently at least five times under different names, which is evidence it’s right. The version most builders will meet is not a framework at all, it is shipped product documentation from Microsoft.

The mechanism

Three axes, and each one alone mispredicts risk. Autonomy is how much runs without a human. Authority is what the agent may do: read, write, move money. Criticality is what depends on it. A highly autonomous meeting summarizer over documents its user already sees is a lower-risk object than a low-autonomy agent with a trade-execution tool, and any single-axis rubric gets that backwards.

Autonomy is a design decision, not a property the system arrives with. Feng and colleagues define five escalating levels by the user’s role (operator, collaborator, consultant, approver, observer) and treat the level as something the designer sets, separately from the agent’s capability and its environment. Note the tension worth keeping: other frameworks index autonomy to environment risk, constraining it in high-risk, adversarial or client-facing settings and relaxing it in controlled in-house ones. These reconcile, since the environment doesn’t determine the level and the designer sets it in light of the environment. Citing the autonomy ladder as authority for the risk-indexing rule nevertheless runs two different claims together.

The convergence is the strongest argument on this page. The same three-tier structure appears in a 2022 field study of citizen development at an energy company (green/orange/red), in Shell’s DIY program (green full-DIY, amber partnered with a coach, red professional development for large or risky apps), and in Microsoft’s shipped Copilot Studio guidance, which plots zones on risk × technical complexity:

  • Zone 1: anyone builds personal or team productivity agents over content they already have access to. Read-only permissions, private, sharing disabled, first-party connectors only, agents run in the user’s own context.
  • Zone 2: IT-approved makers build for other teams under coach oversight and IT-managed review, with advanced connector policies, ALM pipelines, and admin approval to publish.
  • Zone 3: professional developers build mission-critical agents under standard ALM, the strongest controls, and SLAs.

Those zones are shipped documentation for the platform most citizen builders will use rather than a proposal, and adopting the structure costs nothing in argument. Two cautions carried from verification: Shell’s amber zone is described in one book as time-boxed to a year with red triggered by sensitive data or an external internet connection, and neither detail is corroborated anywhere public, while Forrester’s account of red is capability-and-scale based. Use the structure, flag those specifics as single-source. And there is no public evidence that PMI adopted Shell’s scheme, whatever the retellings say.

Untiered governance fails in both directions. Apply the mission-critical posture to everything and personal-productivity agents go into the shadows; apply the personal-productivity posture to everything and departmental agents run unwatched. One heavyweight gate for all builds gets circumvented; that is the observed pattern in citizen development generally rather than a prediction.

A de facto ladder already exists whether or not anyone wrote it down, and where firms actually sit on it is measurable and low. Asked how much they trust agents with end-to-end core processes, 6% of 603 respondents said fully, 43% limited to routine tasks, 39% supervised or non-core only, and 8% not at all (fielded July 2025, sponsored by an orchestration vendor and AWS, so date it and discount it accordingly).

Two refinements worth stealing. Progressive governance: controls scale with demonstrated autonomy and authority rather than intended autonomy, so an agent earns its tier by behaving rather than by being described. And runtime de-escalation: reduce an agent’s permissions automatically when anomalies appear, which makes the tier a live property instead of an intake decision. The capability-indexed control-levels literature makes the same argument from the other end: calibrate control to what the system can actually do, and cheaper controls allow evaluation to run more often and across more threat models, an argument from the authors rather than a measured result.

One correction, because it circulates widely. Anthropic’s published guidance names three per-action permission states — always allow, needs approval, block — and describes plan-level approval. It does not use traffic-light framing and does not tie autonomous execution to reversibility or to a demonstrated track record. Those criteria are a consultancy’s operationalization. The reversibility idea is sound and belongs on this page; the attribution does not.

What to do

Score three axes at intake, in five minutes, on a form. Autonomy: does a human see each action, the plan, or nothing. Authority: read, write, or irreversible. Criticality: personal, departmental, or firm-critical. The highest of the three sets the tier: a single high axis outranks two low ones.

Make the tier physically real rather than documentary. A tier that is only a field in a registry degrades into a label; a tier implemented as an environment with different connector policies, different credentials and different sharing defaults enforces itself. That is the three-zone architecture, and it is why Microsoft ships zones rather than a rubric.

Keep the bottom tier genuinely light and say so out loud. At least one large manufacturer running democratized building at scale deliberately keeps its lowest tier low-friction, on the reasoning that self-reliance is the point. Every hour of governance imposed on a personal summarizer is an hour spent manufacturing shadow agents.

Bind tier to a concrete control set, not to a level of concern. Tier 3 means these five controls, named. If moving from tier 2 to tier 3 leaves the applicable controls unchanged, the tiers are decoration.

Re-tier on scope change, not on schedule. The trigger is a new tool, a new data source, or a new audience. See tiering for the process and promotion gates for the checkpoint.

How you’d know it’s working

Tier distribution in the registry matches control coverage: no tier-3 agent missing its control set, and no tier-1 agent carrying tier-3 friction.

Tier changes trace to scope changes rather than to audits. A tier that only ever changes when someone reviews it is being assigned rather than maintained.

Someone can point to an agent that was demoted. Progressive governance is bidirectional in theory and almost never in practice.

What this doesn’t solve

This is the per-agent axis only. Organizational maturity, the Crawl/Walk/Run/Fly tables in the dimension hubs, is a different scale measuring the program, and conflating the two produces the common error of tiering an agent by how mature the firm is.

Tiering cannot rescue a mis-scored intake. Everything downstream inherits the number a builder picked in five minutes, which is an argument for making the axes concrete enough to be hard to get wrong, and for sampling intake scores rather than trusting them.

The basis for promotion is genuinely contested. One popular framing treats an agent as a junior employee earning autonomy through track record; a randomized experiment finds the employee framing measurably degrades human review, and a separate survey finds 76% of executives already think of agents as more coworker than tool. This wiki takes the position that earned autonomy is the right mechanism and the employee metaphor is the wrong language; the disagreement is live and worth naming rather than resolving silently.

There is a real counter-position that most citizen-built applications fall well below the threshold that warrants serious governance, from an interviewee in a peer-reviewed study of 30 practitioners. It deserves engagement rather than dismissal, with one qualification: that intuition was formed about low-code forms and dashboards, before the artifact in question could read untrusted text and call tools. An outer-ring “system of action” was the safe zone in the low-code era; for agents it is exactly where untrusted input and egress meet.

See also

Graph View

Backlinks