Human–AI collaboration roughly halves the time required to reproduce scientific code in a randomized trial, even after benchmark accuracy has saturated; improving the benchmark and adding an OOD suite reveals construct-validity and robustness issues that accuracy alone would miss.
Life After Benchmark Saturation: A Case Study of CORE-Bench
By expanding beyond accuracy, the authors show CORE-Bench can still measure efficiency, reliability, model vs scaffold contributions, and human-agent uplift, and a randomized experiment finds that human-agent collaboration roughly halves time-to-completion on scientific reproducibility tasks.
When a benchmark's accuracy saturates, it is often retired and replaced with a more challenging version. We show that this approach privileges accuracy and misses the opportunity to study six other key dimensions of agent performance: construct validity issues such as shortcuts, out-of-distribution generalizability, efficiency, reliability, the relative importance of the model versus the scaffold, and uplift from human-agent collaboration. We use CORE-Bench Hard, a benchmark for computational reproducibility of scientific code, as a case study to demonstrate that measuring agents along these dimensions yields meaningful insights into agent performance even after accuracy saturates. First, we surface threats to construct validity in CORE-Bench Hard that are difficult to anticipate with less capable agents. We introduce an improved benchmark, CORE-Bench v1.1, and an out-of-distribution task suite, CORE-Bench OOD. Second, we find that despite accuracy saturation, CORE-Bench v1.1 remains useful for measuring efficiency, reliability, model performance, and scaffold performance. Finally, we conduct a small-scale randomized experiment to measure uplift from human-agent collaboration on real-world computational reproducibility tasks. We find a statistically significant speedup by about a factor of two -- likely underestimated due to one-fifth of human-only reproductions reaching the time limit before completing -- and describe various other findings. Together, our contributions present a more rigorous alternative to the dominant accuracy-centric evaluation paradigm.
Summary
Main Finding
Even after benchmark accuracy "saturates" (top agents achieve near-ceiling, statistically indistinguishable scores), valuable, policy-relevant insights remain available by evaluating agents on additional dimensions: construct validity (benchmark bugs and shortcuts), out-of-distribution generalization, reliability (consistency and calibration), efficiency (tokens and dollar cost), the contribution of scaffolding vs the underlying model, and real-world human–agent uplift. Using CORE-Bench Hard as a case study, the authors (1) find and fix many validity flaws to produce CORE-Bench v1.1 and an OOD suite, (2) show that top agents cluster at near-ceiling accuracy but differ substantially on other axes, and (3) demonstrate a statistically significant ~2x speedup in a small randomized human–agent collaboration trial on real reproducibility tasks.
Key Points
- Accuracy saturation does not imply the benchmark is exhausted for research or deployment decisions. Other dimensions reveal meaningful differences among state-of-the-art agents.
- CORE-Bench v1.1: produced by log analysis to fix 15 task-level errors and remove 20 shortcut-exploitable tasks from CORE-Bench Hard; final suite has 39 tasks.
- CORE-Bench OOD: 19-task out-of-distribution suite (physics, engineering, economics, CS) to test transfer across field shifts; accuracy saturation transfers under this shift (top agents still cluster near-ceiling).
- Accuracy results: top agents on v1.1 and OOD cluster at ~97–100% accuracy, making accuracy alone non-discriminative.
- Reliability:
- Outcome consistency and resource consistency both correlate strongly with mean accuracy (reported correlations ~+0.94 and +0.95).
- Agents are substantially under-confident: mean empirical pass rate ≈93% while mean reported confidence ≈32% (post-hoc elicited).
- Agents struggle to discriminate correct from incorrect runs by confidence (discrimination/AUROC close to random for many agents).
- Efficiency:
- Significant differences in token usage and dollar cost between top-scoring agents. Example: GPT-5.3–Codex (medium) achieves similar accuracy as GPT-5.4 (high) at roughly 60% lower cost.
- Token counts and dollar costs can tell different stories due to provider pricing, caching behavior, and scaffold implementations.
- Model vs. scaffold:
- The choice of scaffolding (tooling, caching, repair heuristics) substantially affects outcomes. Example: with GPT-5.4 (medium), Codex CLI scaffold outperforms the CORE-Agent scaffold by ≈44 percentage points.
- Failure taxonomy:
- 56 accuracy failures analyzed across scaffolds; failure modes concentrated unevenly (e.g., wrong-metric/computation errors concentrated in CORE-Agent; timeouts and dependency failures concentrated in OpenCode).
- Human–agent collaboration RCT:
- Small randomized experiment on 20 reproducibility tasks (ML and social science papers) found that agent-assisted participants completed tasks about twice as fast as humans alone.
- Likely conservative estimate: ~20% of human-only sessions hit a 3-hour timeout without finishing, whereas all assisted sessions completed.
- The authors provide code, data, and logs for reproducibility.
Data & Methods
- Benchmarks:
- Started from CORE-Bench Hard (45 hard tasks across CS, medical science, social science; Python and R).
- CORE-Bench v1.1: 39 tasks after removing/fixing tasks flagged by automated and manual log analysis; preserved disciplines and construction pipeline.
- CORE-Bench OOD: 19 tasks from different fields (physics, engineering, economics, CS) to measure field distribution shift.
- Log analysis:
- Used Docent and LLM-based inspections (GPT-5, GPT-5.4) to flag process correctness, computation correctness, grading errors, and pre-run artifact leakage. Manual inspection augmented automated analysis.
- Agents and scaffolds:
- Evaluated multiple scaffolds (Codex CLI, Claude Code, OpenCode, CORE-Agent) and a variety of models (GPT-5 family variants, Opus variants).
- Varied model reasoning effort and scaffold parameters (e.g., concurrent subagents max_thr).
- Reliability and efficiency measures:
- Ran multiple repeated trials (five additional trials for a set of Codex CLI agents) to estimate outcome and resource consistency.
- Elicited post-hoc confidence from agents to measure calibration and discrimination (AUROC).
- Measured token usage (input + cached + output) and estimated dollar cost using model-provider pricing at run time.
- Failure analysis:
- Root-cause taxonomy constructed from logs across scaffolds, categorizing failures (wrong metric/computation, timeouts, dependency failures, vision/web fallbacks, etc.).
- Human experiment:
- Small randomized study (20 tasks) comparing human-only vs human+agent workflows on real reproducibility tasks; measured completion time and completion rate (3-hour cap).
Implications for AI Economics
- Evaluation beyond accuracy is essential for economic decision-making:
- Procurement and cost-effectiveness: buyers should evaluate cost-per-successful-task (dollars/token per solved task), not just headline accuracy. Different agents with equal accuracy may have large cost differentials (e.g., ≈60% cost gap observed).
- Total cost of ownership (TCO) includes scaffolding: scaffold design, caching, tooling, and heuristics materially affect performance and costs—so investments in scaffolding can be as economically important as improving model quality.
- Reliability-adjusted productivity: empirical consistency and calibration matter for forecasting throughput and risk. Under-confident agents that nonetheless succeed frequently complicate human supervision models; poor confidence discrimination reduces the value of confidence signals for triage and escalation policies.
- Human–AI labor substitution and augmentation: measured uplift (~2x speedup on a reproducibility task) provides an empirical multiplier for productivity models; the authors note this is likely a lower bound. Economists modeling labor impacts should incorporate such measured uplift factors and account for non-completion risk in unassisted workflows.
- Benchmark overfitting and mispricing risk: construct-validity flaws and benchmark-specific adaptation (tailoring scaffolds or prompts to benchmark idiosyncrasies) can inflate perceived capabilities and lead to misaligned pricing, procurement, or regulatory decisions if stakeholders rely solely on accuracy benchmarks.
- Market differentiation: when accuracy saturates, vendors will compete (and can be differentiated) on efficiency, reliability, scaffolding, and OOD robustness—attributes directly tied to economic value and pricing strategies.
- Policy and procurement metrics: regulators and large purchasers should require multidimensional evaluation (cost-per-success, reliability metrics, OOD performance, and validated benchmark integrity) to mitigate procurement failures driven by benchmark gaming.
- Research investment allocation: funding decisions might prioritize research into scaffolding, robustness, and human–AI workflows (high economic leverage) rather than marginal accuracy improvements once accuracy saturates.
- Recommended metrics for economic analysis and decision-making (derived from the paper):
- Cost-per-completed-task and tokens-per-completed-task (median and tail).
- Reliability-adjusted throughput (expected completed tasks per unit time accounting for non-completion risk and variability).
- Confidence discrimination value (how well agent confidence can be used to route tasks to human review).
- OOD penalty (drop in performance under relevant domain shifts).
- Scaffold maintenance/engineering cost and its contribution to performance.
- Human–AI uplift factor (empirically measured time savings and completion-rate improvements).
- Concluding note for economists: retiring benchmarks when accuracy saturates loses important signals that matter for real-world deployment and economic valuation. Extending benchmark use to multidimensional assessments yields data that is more directly tied to costs, productivity, and risk—key inputs for economic modeling, procurement, and policy.
Assessment
Paper Typerct
Evidence Strengthmedium — The randomized experiment provides credible causal evidence for uplift from human-agent collaboration on the sampled reproducibility tasks, and the benchmark analyses systematically surface construct-validity and OOD issues; however, the experiment is small-scale, subject to censoring and possible selection biases, and the results are limited to a specific domain and particular agent/scaffold configurations.
Methods Rigormedium — Strengths: careful reworking of the benchmark (CORE-Bench v1.1), an explicit OOD suite, systematic analysis of multiple performance dimensions beyond accuracy, and a randomized human-subject experiment. Limitations: small sample for the RCT, censoring of time-to-completion (one-fifth hit the time limit), limited reporting of participant/agent details in the summary, and potential unaddressed threats (non-blinding, task selection, and domain-specificity).
SampleUses CORE-Bench Hard (computational reproducibility of scientific code) as the base dataset; introduces CORE-Bench v1.1 and an out-of-distribution task suite (CORE-Bench OOD). Includes a small-scale randomized experiment on real-world reproducibility tasks with human participants comparing human-only to human+agent workflows; specific sample sizes, participant demographics, and exact agent models/scaffolds are not reported in the summary.
Themesproductivity human_ai_collab
IdentificationRandomized assignment of real-world computational reproducibility tasks to human-only versus human+agent conditions; causal inference is achieved through randomization of tasks/conditions and comparison of time-to-completion and success outcomes, though estimates may be affected by censoring (≈20% of human-only runs hit a time limit), non-blinding, and potential selection of tasks and participants.
GeneralizabilityResults confined to computational reproducibility tasks (scientific code) and may not generalize to other domains or routine office/workplace tasks, Findings depend on the particular agent(s) and scaffold(s) used and may not hold for different models or tooling, Small-scale experiment and participant selection limit external validity across skill levels, disciplines, and geographies, Time-limit censoring (≈20% of human-only runs) may bias measured effect sizes and affects interpretation for long tasks, Benchmarks and OOD suite design choices may influence measured threats to construct validity and OOD behavior
Claims (10)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| When a benchmark's accuracy saturates, it is often retired and replaced with a more challenging version; this accuracy-centric approach privileges accuracy and misses the opportunity to study six other key dimensions of agent performance (construct validity issues such as shortcuts, out-of-distribution generalizability, efficiency, reliability, the relative importance of the model versus the scaffold, and uplift from human-agent collaboration). Other | negative | coverage of evaluation dimensions (accuracy-centric vs. multidimensional evaluation) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Measuring agents along the six non-accuracy dimensions yields meaningful insights into agent performance even after accuracy saturates. Other | positive | informative value of non-accuracy evaluation metrics |
Reading fidelity
high
Study strength
medium
|
not reported
|
| We surface threats to construct validity in CORE-Bench Hard that are difficult to anticipate with less capable agents (e.g., shortcuts). Other | negative | construct validity issues (shortcuts) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| We introduce an improved benchmark, CORE-Bench v1.1, and an out-of-distribution task suite, CORE-Bench OOD. Other | positive | existence and availability of CORE-Bench v1.1 and CORE-Bench OOD |
Reading fidelity
high
Study strength
high
|
not reported
|
| Despite accuracy saturation, CORE-Bench v1.1 remains useful for measuring efficiency. Task Completion Time | positive | efficiency (time to complete reproducibility tasks) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Despite accuracy saturation, CORE-Bench v1.1 remains useful for measuring reliability and model versus scaffold performance. Other | positive | reliability / relative contribution of model vs scaffold |
Reading fidelity
high
Study strength
medium
|
not reported
|
| We conducted a small-scale randomized experiment to measure uplift from human-agent collaboration on real-world computational reproducibility tasks. Task Completion Time | neutral | human-agent collaboration uplift (measured via task completion time and success) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| The randomized experiment found a statistically significant speedup by about a factor of two for human-agent collaboration compared to human-only on computational reproducibility tasks. Task Completion Time | positive | task completion time (speedup) |
Reading fidelity
high
Study strength
medium
|
statistically significant speedup by about a factor of two
|
| The observed ~2x speedup is likely underestimated because one-fifth of human-only reproductions reached the time limit before completing. Task Completion Time | positive | proportion of human-only reproductions that timed out (censoring) and its effect on estimated speedup |
Reading fidelity
high
Study strength
medium
|
one-fifth (≈20%) of human-only reproductions reached the time limit before completing
|
| Together, these contributions present a more rigorous alternative to the dominant accuracy-centric evaluation paradigm. Other | positive | evaluation rigor (methodological robustness of multidimensional evaluation vs accuracy-centric approach) |
Reading fidelity
high
Study strength
speculative
|
not reported
|