Swapping the software harness that runs a coding LLM produces little average change in solve-rate, though vendor-native harnesses can outperform on contest-style problems while trailing on repository tasks; cost-per-solved-task comparisons are fragile and were affected by a telemetry semantics bug that the authors corrected.

Main Finding

Holding the underlying LLM fixed and swapping only the agent harness (vendor-native vs. neutral), the study finds no reproducible average capability advantage for vendor-native harnesses on the tested pool. Paired contrasts on two frontier models give small, statistically inconclusive average differences (claude-opus-4-8: −1.25 pp native vs neutral; gpt-5.5: +1.25 pp native vs neutral). There is, however, a strong post-hoc heterogeneity signal on Opus 4.8 (native disadvantage on repository tasks, native advantage on contest tasks) that a designed replication should test. Cost-per-solved-task estimates are sensitive to harness telemetry semantics; after correcting a telemetry bug the neutral harness appears costlier per solved task in several re-pricings, but accounting uncertainty leaves some billed-ordering unresolved.

Key Points

• Experimental question: Does a vendor-native harness outperform a neutral harness when both drive the same model (capability and cost)?
• Cells compared (paired, same-model contrasts):
  • Claude-opus-4-8: claude-agent-sdk (native, C1) vs deepagents/LangGraph (neutral, C2)
  • GPT-5.5: openai-codex SDK (native, C3) vs deepagents (neutral, C4)
  • Two side cells: gemini-3.5-flash (C5) and deepseek-v3.2 (C6) on the neutral harness
• Primary capability estimates (task-mean solve rate on frozen 80-task pool):
  • Opus 4.8: C1 (native) 48.8% vs C2 (neutral) 50.0% → paired difference −1.25 pp, task-bootstrap 95% CI [−10.0, +7.5].
  • GPT-5.5: C3 (native) 55.6% vs C4 (neutral) 54.4% → paired difference +1.25 pp, CI [−4.4, +6.9].
• Heterogeneity (post-hoc) on Opus 4.8:
  • Repository tasks (n=61): native trails by 9.0 pp (CI [−17.2, −0.8]).
  • Contest tasks (n=19): native leads by 23.7 pp (CI [+2.6, +44.7]).
  • Permutation test for interaction p = 0.003 (reported as post-hoc finding).
• Separation of endpoints: correctness vs autonomous completion — 22 of 81 runs cancelled at the wall-clock ceiling had produced a passing patch at cancellation (latency/autonomy matters).
• Cost findings (revised after fixing telemetry defect):
  • Re-priced from raw per-turn events, the neutral harness cost 1.3–1.6× per solved task on Opus 4.8 (task-bootstrap ranges and pricing bases give bands ~1.1–2.1) and ~1.2× on GPT-5.5 (1.05–1.33).
  • Telemetry defect: input-token fields had inconsistent semantics between SDKs (cache-inclusive vs cache-exclusive). Correcting this materially changed cost claims; some account-level unrecorded runs (Anthropic) leave billed-order unresolved (Opus ratio could plausibly be 0.7–2.3 depending on allocation of unrecorded spend).
• Scale and execution:
  • Private, contamination-controlled task suite: 256 tasks (179 repository, 77 post-cutoff contest); main frozen pool of 80 tasks selected mechanically and oversampled tasks on which vendor-native bundles disagreed.
  • 800 planned runs, 792 graded verdicts (380 pass, 404 fail, 7 timeout, 1 apply_error).
  • One microVM per run, append-only event ledger, Docker-isolated grading oracle; wall-clock ceiling = 1,200 s.
• Reproducibility: orchestrator, grading oracle, reanalysis code, and derived aggregates released; tasks remain private to preserve contamination control.

Data & Methods

• Suite and contamination controls:
  • Track A: 179 repository-coded FAIL_TO_PASS tasks mined from four private codebases with hidden tests and gold patches.
  • Track B: 77 contest problems published after a frozen eligibility date (cutoff registry frozen 2026-06-24; eligibility 2026-03-02).
  • Defenses: frozen cutoff registry, runtime drift gate (verifies served model identity each run), canaries and strict mirror hygiene.
• Experimental design:
  • Paired same-model contrasts holding model fixed and varying harness.
  • Main run: 80-task frozen pool (24 discordant tasks from screening + mechanically filled slots).
  • Repeats: 2 for core cells (C1–C4), 1 for side cells. Planned detectable effect ≈12 pp after amendments.
  • Outcome taxonomy: patch extracted and graded offline; verdicts ∈ {pass, fail, timeout, apply_error}; task-mean solve rate computed over scored tasks (timeouts and apply_error excluded from task denominator unless otherwise reported).
  • Ceiling and autonomy: agent runs canceled after 1,200 s; grader scores the patch present at cancellation.
• Infrastructure and telemetry:
  • One KVM microVM per run; identical repository seeds/prompts; harness code runs inside VM with vendor harness sandboxing disabled to equalize constraints.
  • Append-only per-run ledger capturing per-turn events; orchestrator writes run rows including tokens and accounted cost.
  • Post-study telemetry audit discovered SDK differences in token field semantics; re-derived cost metrics from raw per-turn events and reconciled to billed spend where possible.
• Statistical analysis:
  • Task-level bootstrap confidence intervals, permutation tests for interactions, McNemar and GEE logistic planned/used tests, BH-FDR for multiple comparisons.
  • Caveats: frozen pool oversamples tasks with vendor disagreement (conditional estimand), amendments (repeats reduced, pool reduced), and some deviations from the initial OSF preregistration (plan committed to repo; private OSF deposit exists).
• Openness: released orchestration and analysis artifacts; tasks withheld.

Implications for AI Economics

• Vendor harnesses do not guarantee an average capability advantage when the underlying LLM is held fixed. Procurement decisions that assume vendor-native harnesses are categorically superior may be misplaced; organizations should measure harness effects in their own workloads.
• Heterogeneity matters: the harness effect can differ by task type. The strong post-hoc interaction on Opus 4.8 (native worse on repository tasks but better on contest tasks) shows that workload composition can flip the sign of harness advantages. Economic evaluations and vendor comparisons must therefore be stratified by workload characteristics, not collapsed into one aggregate metric.
• Cost-per-solved-task is sensitive to (a) the harness’s run behavior (number and size of turns, truncation settings, retries, latency leading to wall-clock cancellations) and (b) telemetry semantics. Practical takeaways:
  • Use raw per-turn event logs (not SDK-aggregated counters) when computing dollar costs; reconcile to billed spend.
  • Be cautious of SDK-specific token-count conventions (cache-inclusive vs cache-exclusive) — mismatches can materially distort cost comparisons and vendor ordering.
  • Neutral harnesses can incur higher observed usage and therefore higher cost-per-solved-task in this study, but billing and telemetry uncertainty can reverse the ordering; economic decisions should account for such uncertainty and include sensitivity checks.
• Portability vs lock-in trade-off: neutral harnesses enable portability across providers but may impose capability or cost penalties that vary by task type and are not reliably predictable from vendor demos. Organizations balancing lock-in risk against performance/cost should conduct same-model harness contrasts on in-domain tasks.
• Evaluation design recommendations for economic assessments:
  • Prefer same-model harness contrasts (hold model fixed) to isolate harness effects.
  • Use private or post-cutoff tasks and a frozen cutoff registry to limit contamination bias.
  • Record raw per-turn telemetry in an append-only ledger and enforce reconciliation to billed amounts.
  • Report both capability (solve rate) and operational endpoints (time-to-solution, autonomous completion vs correctness) because latency/autonomy can change the effective cost and utility.
  • Treat post-hoc heterogeneity findings as hypotheses for targeted replication rather than as settled evidence.
• Policy and vendor claims: vendors that bundle models and harnesses should not be assumed to provide superior end-to-end performance in all workloads; neutral-harness tooling ecosystems (e.g., deepagents) merit independent evaluation on buyer-specific tasks.

Short actionable checklist for practitioners: - Log raw per-turn events and reconcile to billing; do not rely solely on SDK-aggregated token fields. - Run same-model harness contrasts on representative in-domain tasks, and stratify by task type. - Include latency/ceiling-aware metrics (autonomy completion) and cost-per-solved-task in evaluations. - Treat single-study heterogeneity signals (post-hoc splits) as replication targets before changing procurement.

References and artifacts: the paper releases orchestrator, grading oracle, reanalysis code and aggregates; tasks remain private to preserve contamination control.