A new large-scale benchmark finds proprietary autonomous agents outperform open-source counterparts on complex, long-horizon real-world tasks, while exposing wide variation in resource efficiency, self-correction, and tool use — underscoring the need to co-design models and agent frameworks.

Main Finding

AGENCYBENCH is a new, large-scale benchmark for autonomous LLM-based agents that measures long-horizon, real-world agentic capabilities. It comprises 32 scenarios (138 tasks) across 6 capability domains and requires on average ~1 million tokens and ~90 tool calls per scenario. Using an automated pipeline (user-simulation + Docker sandbox + rubric-based scoring), the authors find a sizable performance gap between closed‑source and open‑source models (48.4% vs 32.1% aggregate score). Frontier models still struggle with these demanding tasks, and models show distinct resource, feedback, and tool-use behaviors. The benchmark and evaluation toolkit are released open-source.

Key Points

• Scope and difficulty
  • 32 real-world scenarios → 138 tasks covering: game development, front-end, back-end, code generation, research, and MCP tool use.
  • Average scenario: ~1M tokens, ~90 multi-turn tool calls, hours of execution time.
  • Tasks are hierarchical and sequential (1–5 tasks per scenario), so earlier outputs affect later tasks.
• Automated rollout & evaluation pipeline
  • Agent scaffold with comprehensive tool suite in isolated workspace.
  • User simulation agent (Claude-4-Sonnet, temp=0.0) provides iterative feedback to replace human-in-the-loop.
  • Deliverables synced to a Docker remote sandbox for UI/functional execution; evaluation artifacts returned to eval-space.
  • Scoring: rule-based assertions for objective checks; LLM-as-judge (text + vision) for subjective/visual checks (Claude-4-Sonnet and Gemini-2.5-pro).
  • Human validation of user-simulator: average alignment score 4.69/5 on sampled rollouts.
• Data curation
  • Task design by 20 experts; rubrics and executable evaluators created; unanimous expert approval required for finalization.
• Main benchmark results (aggregate averages)
  • Closed-source models average: 48.4% (GPT-5.2 top at 56.5%; Grok-4.1-fast 44.3%).
  • Open-source models average: 32.1% (GLM-4.6 top at 38.6%; Qwen-3-235B-A22B-Thinking bottom at 27.0%).
  • Even leading models show substantial room for improvement on long-horizon real-world tasks.
• Behavioral findings
  • GPT-5.2: better at feedback-driven self-correction.
  • Grok-4.1-fast: higher token-use efficiency.
  • Claude-4.5-Opus: prefers shell-based tool usage.
  • Gemini-3-Pro: favors file/memory-management tools.
  • “Home-field advantage”: models perform best with their native/optimized agentic scaffolds (co-optimization matters).

Data & Methods

• Benchmark construction
  • 20 domain experts collected and wrote 32 scenarios (138 tasks). Each task includes: Query (requirements), Deliverables (expected outputs), Rubrics (scoring criteria).
  • Tasks increment in difficulty; workspace isolation prevents cross-task contamination.
• Rollout formalization
  • A scenario rollout τ is the concatenation of task rollouts τi, where each τi contains the initial query, agent reasoning and tool calls, and user-simulation feedback loops when rubrics are unmet.
• Automated evaluation
  • User simulation agent provides targeted rubric-level feedback (returns failed rubrics and reasons).
  • Docker-based remote sandbox performs UI interactions (clicks, screen recording) to generate visual artifacts.
  • Eval-space executes scripts that map deliverables + artifacts to a 0–10 score per task. Final scores average across tasks/scenarios.
  • Judges: rule-based evaluators for objective tasks; LLM judges for subjective/visual tasks (text judge + vision judge averaged where applicable).
• Validation & reliability
  • Human study validated user simulation agent feedback quality (4.69/5 agreement).
  • Evaluation scripts and LLM judges calibrated/validated against human annotations (details in paper appendices).
• Release
  • Full benchmark and toolkit available: https://github.com/GAIR-NLP/AgencyBench

Implications for AI Economics

• Measuring productive value of agents
  • AGENCYBENCH targets tasks closer to economic production (multi-hour, multi-tool, end-to-end deliverables), providing a more realistic yardstick for agent productivity than short-horizon benchmarks.
  • The benchmark can better inform estimates of labor substitution potential and the kinds of tasks agents can realistically automate today.
• Heterogeneous agent performance → heterogeneous economic impact
  • Large variance across models (and their preferred scaffolds/tools) implies uneven automation effects across industries and tasks. Some agents are better at iterative correction; others are more token/compute-efficient—this will influence adoption choices by firms.
• Cost and efficiency considerations
  • Long-horizon tasks consume large token budgets and execution time; token/compute efficiency (e.g., Grok’s behavior) materially affects per-task marginal cost. Economically, closed-source models may currently yield higher task success but can be more expensive per token and subject to vendor lock-in.
  • The “home-field advantage” suggests that total system productivity depends on co-optimizing model + agentic framework, not just model capabilities; investments should account for integration costs and platform effects.
• Investment and R&D priorities
  • Improving feedback-driven self-correction, tool-integration APIs, and long-context retention yields outsized economic value for real-world autonomous work.
  • Open-source models lag in aggregate performance but show distinct peaks—targeted engineering (frameworks, toolchains) could yield high ROI for open models.
• Policy and labor-market implications
  • Benchmarks like AGENCYBENCH enable finer-grained assessments of which roles/tasks are automatable, aiding policymakers and firms in anticipating labor displacement, retraining needs, and sectoral shifts.
  • The need for scalable, automated evaluation reduces reliance on costly human labeling, lowering the barrier to frequent economic impact assessments.
• Benchmark as an economic research tool
  • AGENCYBENCH provides reproducible, quantitative measures for agent productivity and resource usage, useful for economic modeling of automation diffusion, firm-level adoption decisions, and cost–benefit analyses of deploying autonomous agents.

Limitations to note (economic relevance) - Even top models score well below perfect on these demanding tasks, so claims about near-term full automation should be tempered. - Evaluation is shaped by the chosen scaffolds, rubrics, and simulation agents—different setups could shift measured performance and thus economic conclusions. - Running and evaluating these long-horizon scenarios is resource-intensive; widespread usage for economic measurement will require careful cost accounting.

If you want, I can (a) extract specific per-capability or per-model numbers into a table for easier economic modeling, (b) sketch a simple back-of-envelope cost model per task using token counts and current per-token prices, or (c) highlight which scenario types (e.g., full-stack dev vs research) are most/least automatable according to the paper. Which would help you most?