MESSIER aggregates nearly one million agent trial outcomes across 30 benchmarks and 714 agents, revealing uneven capability gains—function-calling near saturation, programming advancing fastest, and enterprise workflows lagging—and demonstrating that common all-pass aggregation rules can obscure progress and alter leaderboards.

Main Finding

MESSIER is a unified, high-resolution corpus of interactive agent evaluations (≈957k trial records) that standardizes per-task and per-verifier outcomes across 30 benchmarks, 714 agents, 11,891 tasks, and 74,205 verifiers. By preserving verifier-level records and harmonizing model/scaffold/environment metadata (including SOC/NAICS task labels), MESSIER enables cross-benchmark capability scaling, counterfactual rescoring to expose aggregation artifacts, and fine-grained analyses of where frontier progress is (and is not) occurring.

Key Points

• Scale and scope

  • 957,253 trial outcomes consolidated from 30 benchmarks, 714 agents (345 models × 205 scaffolds → 714 distinct agents), 11,891 tasks, 74,205 verifier rows.
  • Benchmarks grouped into five work types: programming, research & reasoning, enterprise workflows, GUI navigation, and function calling.

• New contributions and sources

  • Combined public releases (METR, BRIDGE, Agent Psychometrics, General AgentBench, many benchmark author dumps) and new uniform five-agent runs on six under-represented benchmarks (HarveyAI-Lab, MedAgentBench, DABStep, QCircuitBench, ScienceAgentBench, ReplicationBench) collected via the Harbor harness (≈$10k).
  • Provides per-verifier outcomes and execution traces where available.

• Standardization and metadata

  • Normalized model, scaffold, and release date identifiers following Epoch ECI conventions.
  • Tasks labeled with SOC (occupation) and NAICS (industry) codes using a three-voter LLM ensemble (88.3% SOC majority agreement, 86.8% NAICS majority agreement), with remaining cases adjudicated.

• Core analytic findings

  • Frontier progress is uneven by domain (measured as per-task maxima among eligible agents over time):
    • Function-calling: near-saturation (frontier ≈ 0.97).
    • Programming: fastest improvement, frontier ≈ 0.82 (largest net growth over 2024–2026).
    • GUI and Research & Reasoning: intermediate (≈0.71 and 0.68 respectively).
    • Enterprise workflows: lowest frontier (≈0.57), indicating multi-step professional workflows remain most challenging.
  • Aggregation rule effects:
    • Preserving per-verifier records allows counterfactual rescoring. All-pass aggregation (task passes only if every verifier passes) can mask measurable improvements: a task can show very low all-pass pass rates while agents satisfy many criteria on average.
    • Aggregation choices can change measured capability and even alter agent rankings.

• Validation and applications

  • Derived open-data capability scales replicate Epoch ECI rankings closely (Spearman ρ = 0.81).
  • Demonstrated uses: capability scaling, benchmark auditing, predicting task difficulty prior to running agents, slicing capability by occupation, industry, action space, or verifier type.

Data & Methods

• Data sources

  • Ingested per-task outcomes from published releases (METR, BRIDGE, Agent Psychometrics, General AgentBench, multiple benchmark repositories and HF datasets).
  • Supplemented with new, uniform five-agent runs across six benchmarks via Harbor and the OpenHands scaffold.

• Reconciliation pipeline (four stages)

• Cast incoming records into a uniform data model: environment → tasks → typed verifiers (script, exact-match, LLM judge, human) with aggregation rule per task.
• Normalize model/scaffold identifiers and release dates (Epoch ECI naming conventions).
• Automated and manual validation: identifier hygiene, date consistency (agent date ≥ benchmark date rule), deduplication (benchmark × task_id), text cleaning, and spot checks—emphasis on newly contributed runs and stitched records.

• SOC/NAICS labeling via an LLM ensemble followed by adjudication for borderline cases.

• Representation choices emphasized by the paper

  • Store verifier-level binary outcomes vi(τ) rather than only aggregated task score S. Record aggregation rules (all-pass, threshold, single-verifier) so users can recompute task-level success under alternative rules.
  • Tasks annotated with action-space (e.g., shell, tool calls, code submission, web browser) and environment state (filesystem, live web, offline dataset, etc.).

• Analysis methods

  • Frontier calculation: for a quarter q, agents with release date ≤ q considered eligible; per-task frontier is max score among eligible agents; group frontier averages across benchmarks in the group.
  • Counterfactual rescoring: recompute task success under alternate aggregation rules using preserved verifier rows.
  • Capability scaling: derive item-response–style capability indices from open per-task data and compare ranks to Epoch ECI using Spearman correlation.

Implications for AI Economics

• Better measurement of automation potential across occupations and industries

  • SOC/NAICS-tagged, per-task outcomes let economists estimate occupation- and industry-level task automatability with much finer resolution than aggregate benchmark scores. This improves micro-founded estimates of substitution risk, task reallocation, and sectoral productivity impacts.
  • Ability to slice by action space (e.g., GUI navigation vs coding vs tool calls) clarifies which kinds of workplace activities are nearer-term automatable and which require longer human oversight or workflow redesign.

• Improved policy and procurement decisions

  • Preserved verifier outcomes and counterfactual rescoring expose how scoring/aggregation choices can under- or overstate capabilities. Regulators, procurement officers, and firms can use MESSIER to audit vendor claims, choose appropriate acceptance criteria (avoid misleading all-pass thresholds), and tailor evaluation rules to risk tolerance.

• More accurate forecasting and economic modeling

  • High-resolution capability indices (aligned with Epoch ECI) can be used as inputs to diffusion models, labor market equilibrium simulations, and forecasts of skill complementarities or wage impacts by occupation and industry.
  • Per-task difficulty prediction can reduce evaluation costs (fewer reruns) and support targeted investments: firms and labs can prioritize training or productization efforts on tasks where marginal capability gains yield the highest economic returns.

• Lower evaluation cost and higher reproducibility

  • Consolidation reduces redundant compute and expensive reruns (historical sweeps cost ≈$40k); MESSIER’s standardized builders lower barriers for re-evaluation and extensions, making economic studies more reproducible and less resource-intensive.

• Auditing, robustness, and market signaling

  • The corpus enables detection of evaluation pathologies (reward hacking, sandbagging, evaluation awareness) that can distort market signals about model capability and therefore investment or hiring decisions. Transparent verifier-level records support better contract design and performance-based payments.

• Caveats and limitations relevant for economists

  • Benchmarks and scaffolds matter: the same model under different scaffolds yields different outcomes—so any economic conclusion must account for deployment context (instrumentalities, APIs, tool access).
  • Coverage and representativeness: MESSIER consolidates many but not all benchmarks; some domains remain underrepresented. Inference about the broader economy requires careful extrapolation.
  • Verifier noise and aggregation bias: automated judges and human annotations have error; aggregation rules can systematically bias pass rates—economic estimates should incorporate these uncertainties.
  • Temporal and selection effects: benchmarks evolve, and frontier calculations depend on which agents and releases are included; causal claims about capability improvements and labor outcomes should avoid naive temporal aggregation.

Overall, MESSIER provides a practical, high-resolution data foundation to improve economic analysis of AI capabilities, occupational impacts, procurement rules, and policy design — while also making clear that measurement choices (scaffolds, verifiers, aggregation) materially affect the inferred capability signals economists rely on.