0 cumulative citations
View corpus contextAn open-source adapter layer ports 80+ agentic benchmarks to a common framework and enables a 6,600-task evaluation revealing that choice of base model drives performance far more than harness design; the authors also publish Harbor-Index, an 82-task compact suite that remains challenging for frontier models.
Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation
Harbor Adapters standardize and scale agentic benchmark evaluation (porting 80+ benchmarks), enable a 6,627-task cross-model analysis showing base-model capability matters more than harness design and that many benchmarks are redundant, and deliver Harbor-Index, an 82-task curated, hard, high-quality evaluation suite.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Evaluating agents on the growing number of agentic benchmarks is challenging because they often require complex environments and agent integrations. We introduce Harbor Adapters, a unified evaluation infrastructure for agentic benchmarks. Our work makes three contributions. First, we develop benchmark adapters that port more than 80 benchmarks to evaluate arbitrary agents, and validate them through rigorous code review and parity experiments. Second, we conduct a large-scale evaluation of 8 models spanning capability tiers across 54 benchmarks; every model is run with Terminus-2 and with one of 3 native harnesses. This enables a broader analysis of agent capabilities and failure modes than was previously possible. Third, we introduce Harbor-Index, a curated set of 82 difficult, diverse, and high-quality tasks spanning 29 benchmarks, refined from the adapted suite through difficulty filtering, AI and human audit, and an audit-and-fix loop. Harbor-Index preserves the challenge and breadth of large-scale agentic evaluations while being affordable to run; no evaluated model-harness configuration exceeds 30% pass rate, and the strongest (GPT-5.5 with Codex) reaches 28.0%. We release the adapters, evaluation results, in-depth analysis, and Harbor-Index as open-source artifacts to support more reliable and comprehensive evaluation of language-model agents.
Summary
Main Finding
Harbor Adapters provides a unified, open-source infrastructure that dramatically reduces the integration cost of agentic benchmark evaluation (from O(mn) to O(m + n)), enabling a large-scale, multi-harness evaluation across 54 benchmarks and 6,627 tasks. From that pool the authors curate Harbor-Index, an 82-task meta-dataset that is compact, diverse, difficult, and high-quality — preserving evaluation signal while making agentic evaluation far more affordable and reliable. Key empirical findings: base-model capability matters far more than harness design; the benchmark space is low-dimensional and internally redundant; frontier models are more token-efficient but still costlier per trial; dominant failure modes remain factual/algorithmic errors and hidden-test regressions.
Key Points
- Infrastructure
- Harbor Adapters decouple agent implementations from benchmarks by mapping heterogeneous tasks into a unified Harbor schema (instruction, environment, tests, solution).
- Integration complexity reduced from O(mn) (per benchmark-agent pair) to O(m + n) (one adapter per benchmark + one integration per agent).
- Over 80 benchmarks adapted and supported, 22 agents integrated (as of May 2026).
- Large-scale evaluation
- Evaluated 8 model families under 16 model–harness configurations (each model run under Terminus-2 plus one native harness).
- Covered 54 benchmarks, 6,627 tasks, ~0.3M trajectories, 3 trials per (bench, model, harness), consuming 226B tokens and >$300K compute.
- No evaluated model–harness configuration exceeded 30% pass rate on Harbor-Index; best reached 28.0% (GPT-5.5 + Codex reported in abstract).
- Measurement findings
- Most variation in benchmark scores explained by a single shared capability axis (PCA); low effective dimensionality.
- High redundancy: 12 benchmarks can capture rankings with high correlation; 3 tasks per benchmark often recover full ranking (mean Spearman ρ ≈ 0.923).
- Models explain more variance than harnesses (model fixed-effect range ≈5.2× harness range).
- Cost vs performance
- Frontier models (top-tier) yield largest marginal gains on medium-difficulty tasks (empirical difficulty 0.3–0.7).
- Frontier models are more token-efficient (use fewer tokens per trial) but still cost 2–3× more per trial because of higher per-token prices.
- Failure modes
- Dominant failure types: wrong factual answers, algorithmic bugs, hidden-test regressions; operational mistakes (syntax, missing deliverables) also present.
- Harness design shapes behavior (e.g., native harnesses allow more iterative self-correction; Terminus-2 enforces rigid Plan-Execute-Complete flow).
- Harbor-Index
- Curated from 6,627 tasks → 1,311 difficulty-filtered → 307 AI-audited → 82 human-audited, high-quality tasks across 29 benchmarks.
- Designed to be compact, diverse, and robust to verifier/task defects while preserving challenge.
Data & Methods
- Data
- Full adapted pool: 6,627 tasks from 54 benchmarks, spanning domains (software engineering, reasoning, scientific research, multimodal, professional domains, safety).
- Harbor-Index: 82 tasks across 29 benchmarks.
- Models evaluated include multiple Google (Gemini), OpenAI (GPT-5 family), and Anthropic (Claude) variants (e.g., Gemini-3.1-Pro/Flash, Claude Opus/Sonnet/Haiku, GPT-5.4/5-mini/nano in main experiments; abstract mentions GPT-5.5 with Codex).
- Experimental scale and cost
- 16 model–harness configurations × 54 benchmarks × 3 trials ≈ 0.3M trajectories; 226B input+output tokens; >$300K compute.
- Validation & quality control
- Adapters validated with parity experiments against original benchmarks (multi-trial); strict three-stage code audit (bot → junior → senior) with extensive GitHub review (>10k comments).
- Harbor-Index constructed via automated difficulty filtering, multi-stage AI auditing, and human expert review with iterative audit-and-fix loops.
- Analyses
- PCA on model×benchmark score matrix to assess dimensionality.
- Linear mixed-effects modeling to separate model vs harness effects (benchmark as random intercept).
- Token and cost accounting per trial to analyze efficiency.
- Human annotation of failure modes: two domain-expert annotators on 200 trajectories (κ = 0.66); calibrated LLM judge extended annotation to 6,028 trajectories (varied per-rubric κ values).
- Greedy selection and correlation analyses to measure redundancy and representativeness of subsets.
Implications for AI Economics
- Reduced transaction and integration costs
- Harbor’s abstraction (O(m + n) integrations) cuts the coordination cost of evaluating agents across many benchmarks, lowering the fixed cost of comparative evaluation. This is analogous to standardized protocols in other markets that reduce bargaining and setup costs, enabling more frequent and broader benchmarking.
- Public-good infrastructure and open-source effects
- Open adapters, tooling, and Harbor-Index create shared evaluation infrastructure that reduces duplication and increases reproducibility—improving information symmetry among developers, researchers, and purchasers of models/agent platforms.
- Efficient sampling / portfolio effects in benchmarking
- Low effective dimensionality and within-benchmark redundancy suggest diminishing returns from adding many similar benchmarks. Economically, evaluators can design small, representative "evaluation portfolios" (like an index fund) to gauge model capability cheaply. Harbor-Index operationalizes this tradeoff.
- Cost-effectiveness and procurement decisions
- Frontier models yield the greatest marginal returns on medium-difficulty tasks but are 2–3× more expensive per trial despite being token-efficient. Buyers should consider marginal pass-rate-per-dollar (or cost per successful task) when choosing models for production or R&D; costly frontier upgrades may only be justified where medium-difficulty task performance is mission-critical.
- Pricing, competition, and harness differentiation
- Harness design affects agent behavior (iterative self-correction, verification loops). Platforms that provide richer harnesses (native toolchains, verification scaffolds) can extract value by improving realized performance even with the same base model — a point of competitive differentiation and potential platform lock-in.
- Market for third-party curation & audit services
- The authors’ audit-and-fix funnel highlights the value of curated, high-quality benchmarks. There is commercial value in third-party benchmark curation, verifier hardening, and quality assurance services — buyers and regulators may rely on such validators to compare claims.
- Incentives and strategic behavior
- Standardization reduces scope for cherry-picking and inconsistent reporting, lowering information asymmetry. However, compact indices (like Harbor-Index) can be targeted or gamed; continued audit and transparency are needed to mitigate overfitting or benchmark gaming.
- Measurement tools for economic evaluation
- Harbor’s outputs enable new metrics useful to economic analysis: cost-per-pass, tokens-per-pass, marginal pass-rate per dollar, and uncertainty measures from replicated trials. These allow formal ROI calculations for model upgrades, and procurement optimization.
- Externalities and scaling costs
- The evaluation itself is expensive (226B tokens, >$300K), indicating a nontrivial compute and environmental cost to large-scale agentic benchmarking. Economists and policymakers should account for these externalities when funding large evaluations or setting regulatory testing requirements.
- Policy and standard-setting implications
- A shared, curated index like Harbor-Index can serve as a reference test-suite for regulatory assessment, safety auditing, or certification — provided it continues to be updated and guarded against overfitting. Standardized benchmarks lower monitoring costs for regulators and procurement agencies.
- Research & market strategy
- Given redundancy, model developers and investors could prioritize specialization on capability axes that are underrepresented in existing indices (e.g., finance, lab automation) to capture niche advantages. Conversely, generalist frontier improvements appear to pay off broadly, which favors firms pursuing broad capability leadership.
Suggested practical metrics for economic decisions (derivable from Harbor outputs) - Cost per successful task = (dollars per trial) / (pass rate) - Incremental ROI of upgrade = (pass-rate_new − pass-rate_old) / (incremental $ cost) - Tokens-per-pass and latency-per-pass to measure operational efficiency - Concentration of score variance explained by model vs harness to value platform features
Availability - Adapters, Harbor-Index, evaluation tools, full trajectories, and interactive results are open-source (repos linked in the paper), enabling replication and downstream economic analysis.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper reports a large-scale, carefully audited empirical evaluation (6,627 tasks, 16 model–harness configurations, parity checks, human and AI audits) that supports descriptive claims about benchmarking, redundancy, and failure modes; however the results are observational and configuration-dependent (specific models, harnesses, adapters, and time snapshot), limiting causal claims or universal generalization.
Methods Rigorhigh — The authors implement systematic adapter parity checks, multi-stage code review, multi-trial experiments, statistical analyses (PCA, mixed models), and large-scale human/AI auditing of failure modes and task quality, but some analyses depend on provider-specific harnesses and a single experimental snapshot which constrains external validity.
Sample6,627 tasks drawn from 54 agentic benchmarks adapted to the Harbor schema (over 80 benchmarks ported overall); evaluation of 8 models across three providers (Google, OpenAI, Anthropic) under 16 model–harness configurations (Terminus-2 plus three native harnesses), three trials per configuration yielding ~0.3M trajectories, 226B tokens, and ~$300K compute; Harbor-Index is a curated subset of 82 tasks spanning 29 benchmarks selected via model-difficulty filtering, AI and human audits, and iterative fixes.
Themesadoption productivity innovation
GeneralizabilityResults are configuration-dependent (specific models, native harnesses, and Terminus-2) and reflect a May 2026 snapshot., Benchmarks were adapted to Harbor; adaptation choices and any residual adapter-induced artifacts may affect performance comparability with original benchmark runs., Model set covers major providers but omits other architectures/providers, so findings may not generalize to all agent implementations., Cost/token analyses depend on current pricing and usage patterns and may change as models and pricing evolve., Some benchmarks/tasks are i.i.d. subsets or have internal redundancy; conclusions about dimensionality may not hold for future, more diverse tasks.
Claims (13)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Harbor Adapters port more than 80 agentic benchmarks into a unified evaluation infrastructure. Adoption Rate | positive | Number of benchmarks supported by the infrastructure |
Reading fidelity
high
Study strength
medium
|
more than 80 benchmarks
|
| Harbor's adapter architecture reduces the integration burden for supporting m benchmarks and n agents from O(mn) benchmark-agent pairings to O(m+n) adapters and agent integrations. Organizational Efficiency | positive | Integration complexity and infrastructure scalability |
Reading fidelity
high
Study strength
medium
|
from O(mn) to O(m+n)
|
| The large-scale evaluation covered 8 models, 54 benchmarks, 6,627 tasks, 16 model-harness configurations, and three trials per configuration. Other | positive | Scale of the agent evaluation |
Reading fidelity
high
Study strength
high
|
n=6627
8 models; 54 benchmarks; 6,627 tasks; 16 configurations; 3 trials per configuration
|
| Base-model capability is more influential than harness design for benchmark performance. Output Quality | positive | Benchmark score variation attributable to model versus harness |
Reading fidelity
high
Study strength
high
|
n=16
model fixed-effect range 0.451 versus harness range 0.087; 5.2× larger
|
| Most of the 54 benchmarks provide redundant information about model rankings: after selecting 12 benchmarks, every remaining benchmark correlates with a selected benchmark at Spearman rho at least 0.7. Output Quality | negative | Uniqueness and redundancy of benchmark-based model rankings |
Reading fidelity
high
Study strength
medium
|
n=54
rho ≥ 0.7 after 12 benchmarks
|
| Within benchmarks, three representative tasks recover the overall system ranking with a mean Spearman correlation of approximately 0.923. Output Quality | negative | Redundancy of task-level benchmark rankings |
Reading fidelity
high
Study strength
medium
|
n=52
mean rho ≈ 0.923
|
| Thirteen of the 54 evaluated benchmarks are largely saturated, with state-of-the-art models exceeding a 90% score. Output Quality | negative | Benchmark pass or score rate |
Reading fidelity
high
Study strength
medium
|
n=54
13 benchmarks; scores exceeding 90%
|
| Frontier models provide their largest absolute performance gains over other models on medium-to-hard tasks, especially tasks with empirical difficulty between 0.3 and 0.7. Output Quality | positive | Difference in task pass rate between frontier and other models |
Reading fidelity
high
Study strength
medium
|
n=6627
absolute gain peaks at +29 percentage points in the 0.3–0.4 bucket
|
| Frontier models use fewer tokens than weaker models at every empirical task-difficulty level. Task Completion Time | positive | Average tokens consumed per trial |
Reading fidelity
high
Study strength
medium
|
n=6627
frontier models use 42% of weaker-model tokens in the easiest bucket; 61–86% in buckets at difficulty ≥0.5
|
| Token savings from frontier models do not offset their higher prices: other models cost 2–3 times less per trial across difficulty levels. Organizational Efficiency | negative | Dollar cost per evaluation trial |
Reading fidelity
high
Study strength
medium
|
n=6627
other models cost 2–3× less per trial; maximum absolute cost gap $0.37 per trial
|
| Wrong factual answers, algorithmic bugs, and hidden-test regressions are the dominant failure modes for frontier agents across harnesses. Error Rate | negative | Prevalence of agent failure modes |
Reading fidelity
high
Study strength
medium
|
n=6028
|
| Harbor-Index contains 82 tasks spanning 29 benchmarks and is designed so that no evaluated model-harness configuration exceeds a 30% pass rate. Output Quality | negative | Pass rate on the curated Harbor-Index tasks |
Reading fidelity
high
Study strength
medium
|
n=82
no configuration exceeds 30% pass rate; strongest configuration reaches 28.0%
|
| The Harbor-Index construction pipeline reduced an initial pool of 6,627 tasks to 1,311 difficulty-filtered candidates and then to 307 tasks after AI auditing. Training Effectiveness | negative | Number of tasks retained during dataset curation |
Reading fidelity
high
Study strength
medium
|
n=6627
6,627 → 1,311 → 307 tasks
|