Agent leaderboards mislead buyers: over 97% of score variance is driven by tasks, steps and interactions, not a stable agent effect, so rankings reflect specialization rather than intrinsic capability. The author offers 'Deployment Decision Reliability' — a one-page report that converts variance decomposition into how many tasks, which difficulty strata and what cost constraints an enterprise needs to make defensible procurement choices.

Main Finding

Leaderboards for long‑horizon agents mostly reflect task specialization, not a stable, general “capability” signal. Using a four‑facet Generalizability‑Theory decomposition across three public enterprise agent trace datasets, the agent main effect (σ²_a) explains under 3% of total variance in every cell; agent-by-task interactions explain an order of magnitude more. The paper packages the measurement outputs into a one‑page Deployment Decision Reliability (DDR) report that translates variance components into actionable procurement decisions (how many tasks to sample, which difficulty bucket to target, cost constraints, holdout protocol, and how to read failure taxonomies).

Key Points

• Core empirical result: σ²_a < 3% of total variance across datasets/check types; agent×task interaction (σ²_a:t) is much larger (paper reports ~7–23% across surfaces). Leaderboards therefore rank specialization patterns, not a unified capability.
• Five practical diagnostics
• Capability ceiling: Agents don’t show large, stable main‑effect differences—differences are task‑specific.
• Hard‑task collapse: Aggregate reliability (Eρ²) can collapse on the hardest task quartile (example: τ2 action_checks Eρ² falls 0.752 → 0.000).
• Held‑out reversal: Training‑cell projections of reliability overestimate held‑out reliability (Pearson r = −0.90 on τ2 action_checks across 50 splits).
• Aggregate vs family asymmetry: Population diagnostics (e.g., capability‑gap ratio ≈ 0.35–0.40 across two datasets) can transfer, but per‑family agent rankings may invert.
• Variance ≠ frequency: Categories where agents differ (per‑category σ²_a:t) are not the same as categories where failures are most frequent (low/negative rank correlation on MAST).
• Estimation robustness: three estimators (Henderson Method‑I, REML via lme4, Bayesian binomial GLMM) agree to ~3 decimal places; Bayesian GLMM run on an NVIDIA L4 for principled binary‑outcome uncertainty.
• DDR framework: not a single new metric but a reporting discipline combining the variance‑component table, D‑study sizing frontier, cost‑adjusted reliability (RPD = Eρ² / mean inference cost), difficulty‑conditional profiles, and holdout diagnostics into a one‑page deliverable for buyers.
• Cost matters: cost‑adjusted ranking (RPD) can reorder procurement preferences (example: o4‑mini outranks GPT‑4.1 under RPD despite lower accuracy).

Data & Methods

• Object of measurement: binary per‑step success Y_atse ∈ {0,1} for agent a, task t, step s, error category e.
• Four‑facet crossed G‑study model (two‑way interactions included; three‑ and four‑way interactions omitted due to sparse/zero estimates): Y_atse = µ + ν_a + ν_t + ν_s + ν_e + ν_at + ν_as + ν_ae + ν_ts + ν_te + ν_se + ε_atse
• Reliability coefficients:
  • Relative (ranking) reliability: generalizability coefficient Eρ² = σ²_a / (σ²_a + σ²_δ) where σ²_δ sums agent‑related interaction variances scaled by sample sizes.
  • Absolute (threshold) reliability: dependability Φ (includes main effects of non‑agent facets).
  • Capability‑gap ratio: σ²_a / (σ²_a + σ²_a:t).
  • Cost‑aware Reliability (RPD): Eρ² / mean inference cost-per‑task.
• Estimators:
  • Henderson Method‑I (ANOVA closed form; floor negatives to 0),
  • REML via lme4 (canonical frequentist for unbalanced designs),
  • Bayesian binomial GLMM (bambi/numpyro/pymc) for principled binary outcome posteriors.
• Difficulty/holdout analyses:
  • Tasks stratified into quartiles by mean success (Q1 easiest … Q4 hardest), separate G‑study per quartile.
  • 50 random 70/30 (agent,task) stratified splits: fit on training cells, compare projected vs empirical held‑out Eρ².
• Datasets (public, step‑level traces, ground‑truth inspection):
  • TheAgentCompany: 17 agents × 175 tasks × 1–3 checkpoints → ~7.8k rows.
  • τ2‑bench: 3 frontier agents × 50 tasks per domain × 4 trials, multiple check types (db_check, action_checks, env_assertions, communicate_checks, nl_assertions).
  • AppWorld: 18 agent configs × ~750 tasks × ~8 unit tests → ~79.6k rows.
  • MAD/MAST: 1,242 traces with 14 failure‑mode labels (used for failure‑mode analyses).
• Reproducibility: full code, loaders, artifacts released (GitHub link in paper). REML + Bayesian posteriors agreed to 3–4 decimals; full Bayesian sweep on GPU ~70 minutes.

Implications for AI Economics

• Procurement and price‑performance decisions
  • Leaderboard ranks alone are a poor basis for procurement: they conflate specialization with general capability. Buyers should demand DDR reports that include difficulty‑conditional and holdout Eρ² and RPD.
  • Cost‑adjusted rankings (RPD) materially change vendor ordering; procurement contracts and RFPs should explicitly incorporate cost‑adjusted reliability, not just headline accuracy.
• Evaluation budgeting and risk
  • D‑study sizing (derived from the G‑theory decomposition) gives a principled way to compute how many tasks/steps/tests are needed to reach a target reliability for a given deployment mix—this informs evaluation budgets, sampling costs, and expected replication risk.
  • Hard‑task collapse signals high risk for deployments skewed to difficult cases; buyers must allocate more test resources or accept lower confidence.
• Market competition and vendor claims
  • Vendors emphasizing leaderboard position may be selling specialization tailored to benchmark mixes rather than broad capability; this affects how to value models in market comparisons and M&A due diligence.
  • The held‑out reversal (training projections optimistic) implies that vendor‑provided in‑house evaluations are likely to overstate performance replication; procurement should require held‑out replication or independent DDR certification.
• Contract design, SLAs, and insurance
  • For SLA thresholds and contractual guarantees, dependability Φ and difficulty‑conditional reliability matter more than aggregate scores. DDR gives a defensible basis to set testing requirements that align with contractual risk.
  • Insurers or risk managers valuing model performance should price the uncertainty associated with specialization vs general capability differently.
• R&D and product strategy
  • Models may be more valuable as niche specialists for specific task mixes than as universally superior generalists; firms should align model selection to task‑mix economics rather than leaderboard position.
  • Benchmark design and vendor roadmaps should incorporate G‑theory diagnostics to target hard‑task robustness if that is economically valuable.
• Policy and standards
  • Standardized DDR reporting could become a minimum disclosure in procurements of high‑risk agents, enabling comparability on reliability, not only average performance.
  • Regulators and procurement offices can require held‑out, difficulty‑conditional, and cost‑adjusted reliability in safety‑critical procurements.

Actionable takeaways for economic decision‑makers - Do not purchase on headline leaderboard rank alone—request a DDR report (variance components, difficulty‑conditional and held‑out Eρ², cost‑adjusted RPD, capability‑gap). - Use the D‑study projection to size evaluations and set test budgets for the target deployment difficulty mix. - Insist on held‑out replication or independent evaluation before accepting vendor claims for general capability. - Incorporate RPD (reliability per dollar) into procurement scoring to account for inference cost tradeoffs.

References, code, and reproducibility artifacts are publicly released by the author (GitHub link in the paper) to enable adoption of DDR in procurement/evaluation pipelines.