Traces and evals are two products, not one: decide where the traces live before comparing anything else, because the trace store holds a firm’s prompts and tool outputs and is therefore a recordkeeping system before it is a debugging tool.

What this category solves (and what you did before it)

Before it: application logs that recorded the HTTP call and lost the reasoning. Now: step-level traces of non-deterministic runs, plus regression evaluation against fixed datasets.

Keep the halves distinct, because vendors bundle them and buyers conflate them. Tracing answers “what did this agent actually do on this request”: spans and payloads, retrospective, per-request, and the thing an incident responder or an examiner reads. An eval harness answers “does this agent still do the job”: a fixed dataset, a scorer, a number a release can be gated on. Different buyers, different retention profiles, different failure modes. Human annotation is the bridge: feedback attaches to traces, and good production traces graduate into evaluation datasets.

The decision that dominates all others

Traces contain the full prompt and every tool output. For a portfolio or research agent that means positions, counterparty names, deal data and client identifiers. Eval datasets can be synthesized or redacted; traces cannot, because a redacted trace stops being evidence of what happened. So the trace backend is a confidentiality and books-and-records system that happens to render flame graphs. Decide where traces live first; everything else is a feature comparison.

Verified deployment postures, from vendor documentation read on 6 August 2026, with SigNoz added on 22 August:

• MLflow: Apache-2.0, self-hosted by definition. No vendor certification to inherit; the entire control burden moves to the adopting firm’s infrastructure.
• Langfuse: SOC 2 Type II and ISO 27001, annually audited, reports on request. Cloud, self-hosted MIT open source, or self-hosted Enterprise Edition; self-hosting can run air-gapped; retention, masking and deletion documented. The best certification-plus-sovereignty combination found in this review.
• Phoenix: ELv2, fully self-hostable, with Arize carrying SOC 2 Type II, ISO 27001, HIPAA, GDPR and PCI DSS. Arize’s trust center is silent on deployment modes, retention and no-training terms for the commercial product.
• Braintrust: a genuine split-plane architecture. The data plane (traces, spans, datasets, prompts, logs) deploys into the customer’s own cloud with official Terraform modules for AWS, GCP and Azure, while the vendor runs only a control plane holding UI, authentication and non-sensitive metadata. The documentation states its servers and employees have no need for data-plane access.
• SigNoz: MIT core, with ee/ and cmd/enterprise/ reserved by a separate SigNoz Enterprise License requiring a paid production subscription. Cloud and self-hosted, SOC 2 Type II and HIPAA published, no ISO 27001. Audit logs and finer RBAC are marked coming soon on the Enterprise tier, and SAML is absent from Community. Its “Zero Trust BYOC” model keeps customer data and compute inside the customer’s own cloud account and makes control-plane communication pull-only so the vendor cannot initiate a connection inward, a stronger posture than SaaS and one SigNoz specified itself with nothing auditing conformance to it. It covers the tracing half only: OpenTelemetry-native spans with agent and tool-call coverage through OpenLLMetry, OpenLIT or OpenInference, opt-in prompt and completion capture per attribute, token and duration histograms, and no datasets, scorers or regression gating. Log search and trace correlation overlap with a SIEM’s; detection content, case management and investigative retention do not ship with it.
• LangSmith self-hosted: an Enterprise-plan add-on running entirely in customer infrastructure. Its architecture page states no compliance certifications and no data-residency guarantees, so a SOC 2 posture cannot be assumed from it.
• Datadog LLM Observability: SaaS only, and explicitly unsupported on the government sites. For a fund that cannot export prompt contents this is out, unless redaction-before-egress is acceptable, a materially weaker assurance than the data never leaving.
• Weights & Biases: Cloud, Dedicated Cloud and Self-Managed tiers, but the hosting documentation states no SOC 2 or ISO 27001 and leaves open whether Weave, the tracing and eval product, runs on Self-Managed. Verify that before shortlisting it.

What is not publicly establishable for any of them, and therefore left unasserted here: SCIM provisioning, RBAC granularity, environment separation, exportable audit logs, explicit zero-retention and no-training-on-customer-data terms, and audit rights. Those come from the DPA and the enterprise plan documents; ask for them in writing.

What actually differentiates products

Beyond deployment mode: agent-native trace depth (tool calls, sub-agents, memory operations), the eval workflow (datasets, LLM-as-judge, regression gating, a prompt registry), and whether compliance can export any of it. Dashboards differentiate least.

The portability story is real and premature. OpenTelemetry’s GenAI semantic conventions cover model spans, agent spans and MCP, so a vendor-neutral wire format is genuinely converging, the most important structural fact about the category. But the conventions were relocated out of the main semantic-conventions repository into their own, and the whole set carries status Development: OpenTelemetry’s pre-stability tier, where breaking changes are permitted. Nothing is Stable, including the agent-span conventions portable agent telemetry would require. Meanwhile Phoenix emits Arize’s own OpenInference attribute set rather than the OTel conventions. A converging standard, a widely-deployed near-standard beside it, and Development status together are the precise shape of this category’s immaturity. Instrument to the conventions anyway, pinning a version and owning the instrumentation layer, because attribute names will break.

Tier

Day 1 for the trace floor, because retrofitting traces after an incident is reconstruction rather than observability. Day 2 for eval depth.

The enterprise bar

The standard list, with deployment mode promoted to the top for the reason above. Two category-specific additions: what is the retention and legal-hold behaviour of the trace store, and can traces be exported in a form that survives the vendor? No established practice exists for retention or legal hold of trace payloads containing regulated material, a gap that firm policy has to close on its own. See logging and audit and recordkeeping and compliance gaps.

Benchmarks are not the eval harness

No platform’s “evals” tab ships AgentDojo, Agent Security Bench or AgentBench, and the corpus habit of listing all of them as eval benchmarks overstates what exists: AgentPoison is an attack paper with released red-team code, not a scoring harness. AgentBench measures capability; AgentDojo and ASB measure security under injection. None contains finance-grade scenarios (ASB’s finance scenario is generic) and all use simulated tools. Firm-specific eval tasks remain the adopting firm’s own problem, and the dataset and scorer features are exactly what a buyer pays for to solve it.

One convergent finding from those benchmarks is worth carrying into the architecture argument: more capable models complete more tasks and comply with more injections, and guardrails trade utility against attack-success rate without reaching zero. See evals.

M&A state and category maturity

Crowded and consolidating, with the open-source cores changing the exit calculus. As verified in this project’s vendor research on 27 July 2026: Langfuse was acquired by ClickHouse (announced 2026-01-26, core stays open source); Weights & Biases was acquired by CoreWeave (closed 2025-05-05); TruLens’s originating company was acquired by Snowflake (announced 2024-05-22, closed June 2024), with TruLens remaining MIT-licensed under Snowflake stewardship. An MIT or Apache core materially softens an acquisition, the main reason to prefer one here.

No credible leader anchors this category the way Splunk or Datadog anchor conventional observability, and four signals explain why. The wire format is unfinished, with a competing attribute set in wide use. The strongest free option is a project rather than a vendor: MLflow covers tracing, evaluation, judges, datasets, human labelling and a prompt registry, so every commercial product must differentiate above a capable zero-cost floor. The products disagree about what the product even is: keeping customer data out of the vendor’s hands, folding LLM traces into an existing APM estate, an open-core funnel, MIT self-hosting, and an ML-experiment-tracking install base are five different theories of the same category. And the certification picture is uneven, with two vendors publishing SOC 2 Type II and ISO 27001 openly while three others’ postures could not be established publicly from their own documentation.

Treat the vendor choice as reversible, and make it so by owning the instrumentation layer.

Where the category is immature

Unstable telemetry conventions with a competing attribute set; evaluation that bottoms out in LLM-judge scores whose own error rates are rarely published; no vendor documentation stating zero-retention or no-training terms without going to contract; no established practice for retention and legal hold of trace payloads; and no finance-grade agent benchmark to score against.

See also

• Logging and audit — the control this category implements.
• Evals — the recurring practice the eval half of these products serves.
• The agent control plane — observability as one leg of the convergent architecture.
• Unreliable output acting on your systems — the failure class evals exist to catch before production does.
• Cross-session delayed detonation — the attack zone only cross-session tracing can see.
• Recordkeeping and compliance gaps — why the trace store is a regulated system.