A new benchmark of complex, cross-application professional tasks finds leading AI agents complete under one-quarter of assignments: the best model scores 24% on Pass@1, with most competitors performing substantially worse, highlighting large remaining gaps for real-world professional productivity.

Main Finding

APEX–Agents is a new, open-source benchmark testing whether AI agents can complete realistic, long-horizon, cross-application professional tasks (investment banking, management consulting, corporate law). Across 480 expert-created tasks in 33 “worlds,” modern closed-source agent LLMs achieve nontrivial but far-from-human-level success: best Pass@1 ≈ 24% (Gemini 3 Flash), with GPT-5.2 close behind (23%). Open‑source agents score under 5%. Agents show higher ceilings with multiple attempts (Pass@8 up to 40%) but remain inconsistent (Pass^8 much lower, ≈13% for the best).

Key Points

• Dataset and release
  • 480 tasks, 33 worlds (10 investment banking, 11 consulting, 12 law), balanced across three jobs (160 tasks per job).
  • All prompts, rubrics, gold outputs, files, and Archipelago agent-execution infrastructure are open-sourced (Hugging Face + GitHub).
• Task design and realism
  • Tasks were created by industry professionals (256 contributors; mean experience ≈12.9 years). Worlds simulate real workflows with files and 9 application types (Documents, Spreadsheets, Presentations, Mail, Calendar, Code Exec, PDFs, File system, Chat); web search disabled to keep evaluations reproducible.
  • Tasks are long-horizon and data‑rich; experts estimate average time ≈1.82 hours (baselining showed ≈1.37 actual hours on sampled tasks).
  • Each task includes a rubric with 1–10 binary criteria (mean 4.06); Pass@1 requires meeting all criteria.
• Evaluation protocol
  • Eight agent models tested, each run 8 times per task → 30,720 trajectories.
  • Judge model: Gemini 3 Flash (thinking=low) automatically grades each criterion. Judge accuracy on human-labeled set: 98.5% (precision 96.7%, recall 98.1%).
  • Main metrics: Pass@1 (task-uniform mean probability an agent run fully passes), Pass@8 (any of 8 runs passes), Pass^k (consistency: pass on all k runs), mean % criteria passed.
• Performance summary (select figures)
  • Pass@1: Gemini 3 Flash 24.0% [20.7–27.3], GPT-5.2 23.0% [19.8–26.2], Claude Opus 4.5 18.4%, Gemini 3 Pro 18.4%. Open-source: GPT-OSS-120B 4.7%, Kimi K2 Thinking 4.0%.
  • Pass@8: GPT-5.2 40.0% (best), Gemini 3 Pro 37.3%, Gemini 3 Flash 36.7%.
  • Pass^8 (consistency): best 13.4% (Gemini 3 Flash).
  • Mean % criteria passed: Gemini 3 Flash 39.5%, GPT-5.2 38.7%.
• Resource use & failure modes
  • Agents differ widely in steps, tool calls, and token usage. Example: Gemini 3 Flash used ~5× tokens compared to GPT-5.2 and ~8× compared to Gemini 3 Pro, indicating efficiency differences.
  • Timeouts / loop failures occur; agents occasionally delete files (rare, 0.12% trajectories).
  • Many runs yield zero credit (≥40% of runs across agents), and partial progress is common.

Data & Methods

• Data generation
  • 33 project-worlds constructed from professional project scenarios; teams of industry experts acted in roles (partner, associate, etc.) and produced project artifacts (spreadsheets, slide decks, reports).
  • Experts then wrote 8–20 realistic agent tasks per world (mean 14.5 tasks/world) that could only be solved with in-world files/tools.
• Rubrics & gold outputs
  • Every task includes a binary-criteria rubric and a gold output; rubrics were designed to capture must-have elements only.
  • Experts manually validated gold outputs against rubrics.
• Execution & tooling
  • Archipelago framework runs agents and records multi-step trajectories (up to 250 steps per task enforced).
  • Closed-source models accessed via vendor APIs; open-source models via Baseten/LiteLLM wrapper.
  • Judge model grades per-criterion using prompt context + final artifact references (not whole agent trajectory).
• Baseline checks
  • 20% of tasks were independently executed by experts (not creators) to validate solvability and rubrics. Small prompt/rubric fixes were made where needed.
• Statistical treatment
  • Task-uniform Pass@1 reported with 95% confidence intervals via task-level bootstrapping. Pairwise model differences tested with McNemar exact tests and Benjamini–Hochberg correction.

Implications for AI Economics

• Measurable but limited near-term substitution
  • Current frontier agents can fully complete ≈1 in 4 real professional tasks in this benchmark on a single run and can reach ~40% success with multiple attempts. That suggests nontrivial automation potential for parts of high-skill knowledge work, but not wholesale replacement yet.
• Productivity gains likely to be partial and uneven
  • Agents frequently make partial progress (mean % criteria passed ~35–40% for top models). This pattern points toward augmentation/complementarity: agents may speed workflows, lower routine effort, and shift human labor to higher-value oversight, quality control, and judgment tasks rather than directly replacing expert labor.
• Heterogeneous sectoral impact
  • Performance differs by job type and task type (e.g., file-output tasks are harder). Investment banking tasks had slightly higher pass rates in this study, suggesting sectoral variance in near-term automation potential. Firms should prioritize areas where agents are most reliable.
• Economics of deployment & operational costs
  • Token/step inefficiency varies across agents (some achieve higher accuracy at much higher compute/token cost). Firms must consider per-task compute costs, engineering for robustness, and orchestration overhead when assessing ROI.
• Inconsistency and reliability are central frictions
  • Large gaps between Pass@1 and Pass@8, and low Pass^8, imply inconsistency. Economic value from agents will depend on methods to raise reliability (model improvements, ensembling, retry strategies, human-in-the-loop protocols). Reliability constraints will shape task selection and contractual liability considerations.
• Labor market dynamics & skill premiums
  • With agents handling routine or structured subtasks, skill demands may shift: premium on oversight, verification, prompt/agent-engineering, and higher-level domain judgment. Demand for complementary human roles (e.g., agent supervisors, auditors) could rise even as some task-level demand falls.
• Competitive advantage & diffusion
  • Open-source agents lag behind closed-source models on these tasks, but the open release of APEX–Agents + Archipelago lowers barrier to entry for measuring and improving agentic capabilities. Firms and labs can benchmark, iterate, and deploy more quickly—speed of capability diffusion matters for competitive dynamics and wage impacts.
• Measurement for macro/productivity accounting
  • A rigorous, task-level benchmark that simulates realistic professional workflows is valuable for micro-to-macro scaling assumptions. Economists estimating labor displacement or TFP gains should use structured task success rates, partial-credit measures, and reliability metrics (not just single-shot accuracy) when forecasting productivity or reallocation effects.
• Regulation, liability, and contracting
  • Agents making high‑stakes errors (e.g., unwanted file deletions, incorrect legal/financial outputs) highlight the need for contractual safeguards, auditing requirements, and regulatory standards before widespread deployment in professional services.
• Research & policy levers
  • Investment in agent orchestration, verification tooling, and human-in-the-loop workflows will determine how quickly firms can capture agent productivity. Policymakers and firms should track reproducible benchmarks like APEX–Agents to monitor capability progress and to design training or transition programs for affected workers.

Limitations to bear in mind for economic interpretation - No web search: worlds included all needed files and web access was disabled, which makes results reproducible but may under/over-estimate real-world agent performance when internet access is available. - Judge-model overlap: the automatic judge is itself a LLM (Gemini 3 Flash), though the paper reports high judge accuracy on human labels; potential bias should be considered. - Benchmark scope: tasks are professional but still a curated subset of possible work; broader labor-impact estimates require mapping benchmark tasks to labor-time and value across firms/sectors.

Overall, APEX–Agents provides a concrete, reproducible yardstick for agentic performance on realistic professional tasks. For economists, it offers actionable signals about the extent, heterogeneity, and reliability of agent-driven productivity improvements—and a platform to track how those signals evolve as models and deployment strategies improve.