A new expert-designed benchmark shows top AI models can complete high-value knowledge tasks at roughly 60–64% of expert-level quality, with GPT‑5 leading; however, the sizable gap to human professionals underscores continued limits to AI replacing high-skilled knowledge work.

Main Finding

APEX-v1.0 is a new, expert-designed benchmark that evaluates whether frontier LMs can perform high-value knowledge-work tasks with economic relevance. On the 200-case held-out suite (investment banking, management consulting, law, primary care), the best models achieve roughly 60–64% of rubric criteria on average (GPT‑5: 64.2%, Grok 4: 61.3%, Gemini 2.5 Flash: 60.4%), with a substantial gap to human expert performance. This indicates strong but incomplete capability for producing economically valuable outputs and highlights the need for task-aligned, economically grounded measurement of AI progress.

Key Points

• Purpose: APEX measures models on realistic, high-value knowledge-work tasks (1–8 hours of expert time, mean 3.5 hours), rather than abstract capabilities.
• Dataset:
  • 200 held-out test cases, evenly split across four domains: investment banking, management consulting, law, primary care (medicine).
  • 76 domain experts contributed; mean experience ≈ 7.25 years.
  • 5,818 rubric criteria total; mean ≈ 29 criteria per case (range 7–54).
  • Mean evidence sources per case ≈ 5.8; mean total source tokens per case ≈ 26,677 (max limited to ~100k tokens).
• Evaluation procedure:
  • 23 frontier models evaluated (13 closed-source, 10 open-source).
  • Each model run 3× per case; median score reported.
  • Responses autograded by a 3-LM judge panel (o3 low, Gemini 2.5 Pro off, Sonnet 4 off) with majority-vote Pass/Fail per rubric criterion.
  • Judge panel shows high internal consistency and ~89% agreement with human labels (on one model’s annotations).
• Summary results:
  • Top mean scores: GPT‑5 (64.2%), Grok 4 (61.3%), Gemini 2.5 Flash (60.4%).
  • Domain-average scores across models: medicine 47.5%, investment banking 47.6%, management consulting 52.6%, law 56.9%.
  • Large variation at the bottom of leaderboard; several models <50%.
• Reliability & limitations noted by authors:
  • LM judges can be biased; using a panel mitigates but does not eliminate bias.
  • Judges used are themselves models (with different “Thinking” settings), which required calibration checks (self-preference and inter-judge agreement).
  • Responses and model behavior are non-deterministic (mean range across 3 runs ≈ 11.9 percentage points).
  • APEX-v1.0 is a closed held-out dataset to preserve rigorous evaluation.

Data & Methods

• Expert sourcing and prompt/rubric creation:
  • Experts recruited and vetted (30–45 min interviews + 1–2 hour assessments); contributors produced prompts, evidence, and detailed binary rubrics decomposing “quality” into objective criteria (analogous to unit tests).
  • Prompts grounded in common high-value tasks for senior roles at top firms (e.g., Goldman Sachs, McKinsey, BigLaw, top medical centers).
  • Quality control: multi-stage human review and LM-assisted feedback; 300 prompts started, 200 accepted.
• Dataset characteristics:
  • Mean prompt tokens ≈ 430 (domain variation: medicine shorter, consulting longer).
  • Rubrics allow per-criterion Pass/Fail → percentage-of-criteria-passed scalar score per response.
• Models & inference settings:
  • 23 models (mostly 2025 releases); used recommended temperatures and “Thinking” (chain-of-thought) settings where available; no uniform system prompt except for Nova Pro.
  • Context windows capped to ensure sources fit across models.
• Grading:
  • Panel of three judge LMs grades each criterion independently; majority vote decides Pass/Fail.
  • Evaluated judge performance on consistency, inter-judge agreement (3/3 agreement ≈ 81%), judge self-preference (small biases), and judge-human agreement (≈ 89% on one model’s human labels).
• Metrics reported:
  • Mean score (percentage of rubric criteria passed).
  • Pairwise head-to-head win rates across tasks.
  • Frequency of being ranked first / last across cases.
  • Domain-specific mean scores.

Implications for AI Economics

• Measurement alignment: APEX demonstrates the value of task-aligned benchmarks for estimating economic impact. Abstract capability tests (e.g., general reasoning or NLP metrics) can miss whether outputs are actually useful for high-paid knowledge work.
• Productivity and augmentation vs replacement:
  • Best models reach ~60–64% of rubric criteria on average, indicating meaningful capability to augment expert workflows (drafting, triage, first-pass analysis), but not yet reliable autonomous replacement for complex expert tasks.
  • Heterogeneity by domain suggests partial automation potential: legal and consulting tasks appear easier on average than primary-care medical tasks (per rubric pass rates), so sectoral effects on labor demand will vary.
• Labor market and firm strategy:
  • Firms can likely deploy LMs for assisted workflows (efficiency gains, faster first drafts, decision support), but must retain human oversight for correctness, liability, and nuanced judgment.
  • Investment in complementary human capital (supervision, validation, prompt engineering, rubric design) will be economically valuable.
• Macroeconomic impact: Given the gap to human-level performance, near-term contributions to GDP via direct task automation may be limited and concentrated — large productivity gains are conditional on further model improvements, cost-effective deployment, and integration with workflows.
• Policy and governance:
  • Benchmarks like APEX enable regulators and policymakers to assess sector-specific AI readiness and risk (e.g., medical/law liability, misinformation, malpractice).
  • Standardized, economically meaningful evaluation supports targeted workforce transition policies and standards for deployment (certification, audit trails, evaluation thresholds).
• Research and investment priorities:
  • Need for broader, open, and domain-diverse benchmarks linked to economic outcomes (billing rates, time saved, error costs).
  • Improve human-ground-truth evaluations and judge calibration (reduce LM-judge bias).
  • Study cost-performance tradeoffs: model inference costs per task vs. economic value generated.
  • Track downstream outcomes (hiring, wages, firm productivity) tied to deploying models that pass APEX-style tasks.

Takeaway: APEX provides an actionable, expert-grounded way to quantify how much frontier LMs can do economically valuable knowledge work. Current frontier models show substantial capability but meaningful gaps remain; the benchmark should inform firm adoption strategies, labor-market modeling, and policy design while motivating further work in task-aligned evaluation and judge calibration.