A demo is an anecdote; an eval is an acceptance test — only 4 of 30 indexed deployed agents ship agent-specific safety evaluations, so the eval a firm writes is probably the first one its agent has ever faced.

Start with what vendors provide, which is less than most firms expect, and evaluate the tools in use rather than conversational safety. In a February 2026 index of 30 deployed agentic systems, four had agent-specific system cards, 25 disclosed no internal safety results, 23 gave no third-party testing information, and nine documented no guardrails. The mechanism behind the gap matters more than the counts: safety-critical behaviour emerges from planning, tools, memory and policies rather than model capability, so model-level evaluations fail to transfer to a firm’s own configuration, and the index’s authors warn that buyers risk false assurance from model-only documentation.

The second calibration point is how far below expectations grounded accuracy sits. On a benchmark built from roughly 89,000 pages of Treasury bulletins, frontier models answered about 2% of questions correctly without corpus access, confidently and wrongly. With a standard RAG pipeline over the documents they plateaued below 45% in December 2025; the March 2026 successor benchmark puts the best agent at 48.1% on the full PDF corpus, rising to 66.9% when handed oracle pages and pre-parsed documents. It is the vendor’s own benchmark and it motivates their product, so date whatever is quoted. The ceiling moves, and it is nowhere near where a citizen developer assumes.

The checklist

1. Write a task eval from the agent’s real inputs before promotion. Twenty cases with pass/fail scoring beats nothing by a wide margin, and public benchmarks are no substitute: built for static academic settings with predefined tasks and deterministic outcomes, they rarely capture ambiguous success criteria or dynamic workflows.

2. Decompose, then prioritize. OpenAI’s documented example splits an operations agent into information gathering, calculation, and reasoning, and evaluates each independently, because long action sequences compound rare per-action failures. Where full decomposition is too expensive, its prioritization rule transfers directly to a fund: evaluate the high-risk actions first, financial transactions before anything else.

3. Then run it end-to-end, as close to production conditions as can be managed. That is the current state of the art, and OpenAI’s own follow-on sentence is the one to carry: because evaluation is immature, lean harder on other practices such as human approval for high-stakes actions. A model developer arguing for architectural controls over measurement is worth quoting to anyone who thinks evals alone clear a gate.

4. Add a security pack scaled to tier. Red-teaming scope has to cover privilege escalation through tools, memory poisoning and plan manipulation, not injection prompts alone. AgentDojo, Agent Security Bench and Agent-SafetyBench are real, dated harnesses; AgentPoison is an attack paper with testbeds, not a scoring harness. None has finance-grade scenarios, so all are necessary and not sufficient at a gate. The practitioner minimum circulating in hedge-fund forums is sound: injection probes, citation-accuracy checks, data-leak probes, canary traffic, a rollback path.

5. Evaluate the transport actually deployed. Attack success rates rise when the same scenarios run over MCP rather than direct tool APIs. The paper reporting this leaves its own figures unreconciled; the abstract claims a relative increase, the body a percentage-point one. Take the finding and not the number: benchmark results from direct-API harnesses understate risk for MCP-mediated agents.

6. Probe for what quality evals structurally cannot see. A memory-poisoned agent processes unrelated work correctly, with no noticeable performance side effects. Two independent lines of evidence agree, and one attack achieves it at a sub-0.1% poison ratio with under 1% benign degradation. No amount of task-performance regression testing surfaces that. Targeted probes and direct memory inspection are needed; see memory and context poisoning.

7. Re-run on every model or tool change, and accept that this is unsolved. Capability findings expire; researchers say so in their own limitations sections, and practitioners report that model churn makes eval systems hard to stabilize. NIST’s March 2026 report on monitoring deployed AI systems is the honest anchor: it names drift as the problem, then names the absence of ground-truth datasets and of a reliable drift-detection method as unsolved. The federal standards body is saying the field cannot reliably detect what this page tells firms to watch for.

8. Pick graders deliberately. Anthropic’s public guidance is a reasonable default: code-based grading is fastest and most reliable but lacks nuance, model-based grading scales to judgement calls, human subject-matter review covers the rest, and volume of automatically graded cases beats a small hand-graded set, with three constraints from evidence on top of it.

Never grade with the model family that produced the output. In a field experiment, a GPT-4o-mini judge rated GPT-4o-co-created ads as equal or better on every axis while human raters saw an image-quality drop the judge missed entirely.

Validate the judge before trusting it. The best worked example validates an LLM complexity classifier against 1,000 problems at r=0.65 against human solve times. That is what a validated grader achieves, which should calibrate expectations for an unvalidated one.

Price the human alternative honestly. One defensible human panel took 22 blind expert evaluators, 1,595 evaluations over 550 solutions, roughly three ratings each, reaching an ICC2 of 0.452. That number is what makes “we’ll just have a reviewer score it” look naive.

9. Grow the eval set from real failing traces, and score dev and production with the same scorers. A vendor’s architecture rather than an industry finding, and still the right shape: the cases that matter are the ones the agent already got wrong.

How you’d know it’s working

Evals block promotions and catch regressions after model swaps. A suite that has never failed anything is measuring something other than the firm’s risk.

Coverage can be stated as arithmetic. A 2,000-task evaluation cannot detect a strategy that triggers on 1 task in 100,000, which a 100,000-task deployment will hit. That comes from work on scheming frontier models: the threat model stays there, the arithmetic carries over to citizen agents, and it converts “we tested it” into a number.

Error and override rate tracked as a standing production metric, not a launch artifact.

The people who quit are counted. A reported 84% positive productivity rate meant 84% of continuing users; abandoners had left the denominator.

Throughput is always paired with quality. In the support-agent study most often cited for productivity gains, the effect on issue resolution against an 82% baseline was economically modest and statistically insignificant. Speed metrics alone hide quality loss.

None of this is common. As of January 2025, 39% of C-suite leaders used benchmarks at all, and of those, 17% ranked fairness, bias, transparency, privacy or regulatory measures as most important.

What this doesn’t solve

Evals sample behaviour; they do not bound it. Evaluations establish lower bounds on capability only, and composing separately evaluated agents voids even the lower-bound reading, because emergent properties exceed the sum of the measured parts.

There is a measurement vacuum exactly where this wiki’s hardest concerns live. Of ten surveyed agent-security benchmarks, none evaluates cross-session or drift/dormant threats; every published one terminates at single-session attacks. Slow-burn compromise is therefore structurally unmeasurable, and a vendor claiming cross-session coverage cannot be empirically distinguished from one claiming none. That is a single-author preprint working from its own sample frame, so attribute it; its three derived demands nonetheless make an excellent vendor questionnaire: report against adaptive as well as oblivious adversaries, report false-positive rates next to true-positive rates, and treat 95% mitigation at a 40% false-positive rate as impractical.

Whatever the eval leaves unmeasured is what the agent will sacrifice to pass it: Goodhart, restated for agents by Kolt, with the sharper corollary that goals resisting pre-deployment metrics resist post-hoc measurement too. And per-step evaluation against fixed thresholds misses cumulative drift where each individual step scores within tolerance; that is a well-posed hypothesis with a proposed detection method, not a measured phenomenon.

For an eval report that fails every bar on this page, consider a vendor claim of 100% precision in agent decisioning against under 95% for humans under four-eyes review. The 100% covers approvals only, at one unnamed client, with ground truth defined as a single compliance officer’s agreement, no sample size, no dataset, no independent evaluation. The companion figure the marketing omits is 90% precision on declines: one in ten agent declines judged wrong, so ask for the denominators.

See also

• Promotion gates — where eval results become gate evidence.
• Unreliable output acting on your systems — the failure class evals exist to catch.
• Observability and eval platforms — the tooling category for running evals at scale.
• The jagged frontier — why capability must be measured per task rather than assumed.
• Runtime guardrails and policy enforcement — the runtime complement to pre-deployment evals.
• Memory and context poisoning — the failure a passing quality eval will not surface.