0 cumulative citations
View corpus contextA new benchmark asks AIs to build production-ready agents and finds they can produce runnable systems but rarely reach deployable quality — the best model configuration passes only 23.9% of evaluation simulations against an 82.2% expert ceiling.
$τ^τ$-Bench: An Environment for End-To-End, Realistic Agent Construction
ττ-bench is a new, realistic benchmark that frames end-to-end agent construction as the task for developer agents, showing current LLM configurations can create running agents but fall well short of expert performance on held-out simulated production tasks.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
LLM agents are rapidly becoming production software, deployed to handle customer service, adjudicate disputes, and operate internal systems. Notably, the work of building them is increasingly handed to coding agents, yet existing benchmarks say little about whether an AI system can deliver one under the conditions of a real client engagement. We introduce $τ^τ$-bench (pronounced hyper-tau-bench), a benchmark that makes agent construction the task. A developer agent is given the records a business actually keeps, a client who holds requirements, a production API that operations must run through, a codebase to inherit, and limits on serving cost and models: the same starting point a real engagement provides. From these it must deliver a complete customer-service agent, scored by deploying that agent against held-out simulated users. Across 53 tasks spanning four domains, the strongest configuration, Claude Opus 5 under Claude Code, passes just 23.9% of evaluation simulations. Meanwhile, an expert-authored reference ceiling scores 82.2%. The failures mirror ones human agent developers see: models issue shallow queries in place of deep comprehension of the records, communicate almost nothing to the client, and experiment too little with agent architecture and serving spend, shipping the first design that runs. We aim for $τ^τ$-bench to turn the work of cooperative agent building into a measurable target for coding agents.
Summary
Main Finding
τ τ-bench is a new benchmark that frames "agent construction" itself as the task: an LLM-based developer is given the realistic mess of a business (multimodal records, a live-but-imperfect client API, an interactive client, an inherited codebase, and strict cost/model constraints) and must deliver a complete customer‑service agent. When evaluated end-to-end by running the submitted agent against held‑out simulated users, current coding-agent configurations can produce running agents but generally not production-ready ones: the best measured system (Claude Opus 5 + Claude Code) passes only 23.9% of evaluation simulations, versus an expert-authored reference ceiling of 82.2%.
Key Points
- Task framing: the developer agent must (1) recover the specification from scattered, multimodal business artifacts and an interactive client, (2) build or extend an agent implementation inside a sandboxed workspace, respecting a fixed model menu and per‑conversation credit budget, and (3) submit the agent which is scored by deployment against held‑out simulated users.
- Realism levers: tasks vary across seven independent dimensions (evidence surface modalities, client simulator presence, client REST API fidelity, starting codebase, model menu & budget, a live-experiment sample, and phrasing/judging rules). This is intended to mirror real engagements.
- Failure modes observed: developer agents tend to stop deep requirement elicitation early (keyword searches vs. reading), fail to interview the client adequately, under-explore design alternatives (ship first-running architecture), misallocate serving budget (too cheap or too expensive), and validate against self-authored tests that miss real deployment needs.
- Measured gap: best automated configuration passes 23.9% of held‑out simulations; expert reference scores 82.2% — indicating large capability gaps in automated agent construction today.
- End-to-end, implementation-agnostic scoring: performance is judged by final conversation outcomes and final database state against ground-truth task outcomes (rubric-driven), so any internal architecture can win if behavior is correct.
Data & Methods
- Task components:
- Corpus A: multimodal artifacts produced by transforming domain policies into materials a business would actually keep (documents, support transcripts, spreadsheets, screenshots, slide decks, recordings).
- Client simulator C: LLM-based simulated client that holds facts removed from the corpus and reveals them only through multi-turn questioning; the client prompt is deterministic and fact-limited (making elicitation measurable).
- Client REST API T: an OpenAPI-style API acting as the production store; some tasks include deterministic, realistic API defects drawn from a nine-class catalog (schema drift, timeouts, async commits, etc.).
- Starting implementation π0: some tasks seed the workspace with imperfect real-style codebases (stale values, partial coverage) to require repair/extension.
- Model roster & budget M, b: shared model menu across frontier/mid/small tiers (examples: GPT-5.6-sol, Claude Opus 5, Gemini 3.1 Pro; mid/small alternatives) and domain/difficulty-specific credit budgets per served conversation (examples: airline easy/med/hard = 0.30 / 0.061 / 0.022 credits).
- Construction process:
- Policy decomposition into atomic facts; artifacts are generated by prompting models to produce business-style carriers for those facts; all outputs are audited so every fact is recoverable and no unsupported facts are introduced.
- 2,868 distinct artifacts across four domains (documents 6%, conversations 55%, UI/process visuals 28%, recordings 7%, operational exports 3%); text artifacts alone exceed 5.5M tokens.
- Evaluation:
- Developers can self-evaluate by authoring τ-bench-style local simulations (these are the only source of pre-submission feedback).
- Final submissions are sealed and evaluated by running the constructed agent on held-out τ-bench-style tasks (53 public release tasks; a separate 53-task held-out set is kept private).
- Scoring is mean task reward (1/0 pass per task based on outcome & messaging), minus penalties for overspending the credit budget: S = max(0, mean_reward − penalty).
- Experiments & findings:
- Best automated configuration (Claude Opus 5 under Claude Code) achieved 23.9% pass rate; an expert-authored reference agent scores 82.2%.
- Analysis attributes the gap to missing engineering disciplines (deep reading, client interviews, design search, budget-aware serving design, external validation).
Implications for AI Economics
- New, high‑value benchmarked task: agent construction is an economically valuable engineering activity distinct from writing code or running an agent; benchmarking it makes automation of a high-margin enterprise activity measurable.
- Current automation limits imply continued human labor value: because coding agents frequently fail to elicit requirements, explore designs, and validate appropriately, firms are likely to retain human engineers for oversight, specification, and final validation for the near term. This preserves human labor demand in higher‑value parts of agent delivery.
- Productivity and cost dynamics: if improved, automated developers could reduce the cost and time to deploy domain agents substantially (lower engineering hours, faster iterations), but current model limitations mean automation mainly produces prototypes, not deployable systems. The benchmark’s explicit credit-budget accounting ties model selection to economic trade-offs (quality vs cost) — a useful lens for ROI calculations.
- Incentives for model/provider features: the failures highlighted (deep comprehension, interactive elicitation, robust testing, budget-aware routing, orchestration/search) point to provider opportunities — invest in models and toolchains that support (a) long-context, precise document understanding; (b) structured interactive elicitation; (c) automated design-space search and experiment-running primitives; and (d) cost-aware model-routing and reliable code generation with test harnesses.
- Risk and governance: automated construction that misses hidden requirements or mismanages budget/behavior could cause costly operational errors or customer harm. Firms should pair automated developers with human-in-the-loop checkpoints, verification suites, and clear accountability until automated performance approaches expert ceilings.
- Research & market signals: τ τ-bench turns previously fuzzy engineering practices (elicitation, validation, architecture search) into measurable objectives; that should accelerate R&D and commercial tooling targeted at closing the gap, and give purchasers clearer metrics for comparing coding-agent products and estimating potential labor substitution and cost savings.
Assessment
Paper Typedescriptive
Evidence Strengthn/a — The manuscript introduces a benchmark and reports evaluation metrics of models on that benchmark rather than making causal claims that require identification; results are descriptive performance measures, not evidence of causal effects.
Methods Rigormedium — The benchmark design is careful and well-documented: tasks are generated from decomposed atomic facts, artifacts are audited, client and user simulators and held-out evaluation sets are used, and multiple model configurations are evaluated. Limitations include reliance on LLM-based client and user simulators whose fidelity to real stakeholders is uncertain, a finite set of domains (53 public tasks across four domains with a private held-out set), possible selection artifacts in the generated evidence, and dependence on rubric-driven human judgments for messaging quality.
SampleBenchmark suite of 53 released tasks (with a separate private held-out set of 53) across four domains (airline, retail, telecom, banking), producing 2,868 multimodal evidence artifacts (documents, conversations, operational exports, UI/process visuals, recordings) totaling millions of tokens; simulated human client (LLM) that holds some facts out of the corpus; client REST APIs (some with deterministic defects); starting codebases sampled from imperfect implementations; evaluations run by deploying constructed agents against held-out simulated users and scored by rubric-driven judges; experiments reported across a roster of production LLMs and model-budget tiers.
Themeshuman_ai_collab productivity
GeneralizabilityRelies on LLM-based client and user simulators which may not capture the full nuance of real human stakeholders and customers., Tasks cover four domains and 53 public points in configuration space — may not reflect all industries, workflows, or rare/edge-case business practices., Artifact generation is prompt-driven and audited but may not fully reproduce the diversity or messiness of real enterprise records., Model roster and budget constraints reflect contemporary large models; results may change as model capabilities or pricing evolve., Evaluation uses simulated users and rubric-based judgments; real-world deployment can surface operational issues (security, legal, integration) not captured here.
Claims (10)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Across 53 tasks spanning four domains, the strongest evaluated configuration, Claude Opus 5 used through Claude Code, passed 23.9% of evaluation simulations. Output Quality | negative | Share of held-out evaluation simulations passed by the constructed agent. |
Reading fidelity
high
Study strength
medium
|
n=53
23.9% passed
|
| An expert-authored reference agent achieved an 82.2% pass rate on the evaluation simulations, substantially exceeding the strongest developer-agent configuration. Output Quality | positive | Share of evaluation simulations passed by the expert-authored reference agent. |
Reading fidelity
high
Study strength
medium
|
n=53
82.2% passed
|
| Current AI systems can construct agents that run, but their performance is not yet comparable to that of deployable, production-ready agents. Output Quality | negative | Readiness of constructed agents for correct end-to-end deployment behavior. |
Reading fidelity
high
Study strength
medium
|
n=53
23.9% versus 82.2% pass rate
|
| Developer agents frequently fail to recover the full specification, instead querying business records by keyword rather than deeply interpreting them. Task Completion Time | negative | Completeness and accuracy of requirements/specification recovery. |
Reading fidelity
high
Study strength
low
|
not reported
|
| Developer agents often fail to interview the client, thereby shipping systems that omit requirements that could have been surfaced by asking a single question. Output Quality | negative | Recovery of client-held requirements. |
Reading fidelity
high
Study strength
low
|
not reported
|
| Developer agents tend to explore the design space insufficiently, patching the first architecture that runs instead of searching for a better architecture. Organizational Efficiency | negative | Extent and effectiveness of architectural exploration during agent construction. |
Reading fidelity
high
Study strength
low
|
not reported
|
| Developer agents commonly validate against self-authored tests that encode their own blind spots rather than against the deployment behavior they are ultimately building toward. Error Rate | negative | Validity of pre-submission testing relative to held-out deployment performance. |
Reading fidelity
high
Study strength
low
|
not reported
|
| The benchmark contains 53 public release tasks and a second held-out set of 53 tasks. Other | positive | Benchmark task-set size. |
Reading fidelity
high
Study strength
medium
|
n=106
53 release tasks and 53 held-out tasks
|
| The benchmark's four-domain transformations produce 2,868 distinct evidence artifacts, and the text-format artifacts alone contain more than 5.5 million tokens. Other | positive | Scale of the benchmark's evidence corpus. |
Reading fidelity
high
Study strength
medium
|
n=2868
2,868 artifacts; over 5.5 million tokens
|
| Benchmark artifacts were reviewed by three human auditors per transformation. Ai Safety And Ethics | positive | Human review coverage of generated benchmark artifacts. |
Reading fidelity
high
Study strength
medium
|
n=3
3 human auditors per transformation
|