Tightening the agent harness can matter more than changing model weights: under constrained context windows a closed-loop harness nearly doubled the share of failing tests fixed and raised full solutions on a 169-task benchmark from 43 to 72. The gains repeat across other benchmarks and different model designs, showing harness choices materially change what the same frozen model can accomplish.

Main Finding

Changing only the harness (the code that decides what the model sees, what tools it can call, and when to intervene) materially changed what the same frozen model weights achieved on coding benchmarks. Under context pressure (tight windows), a closed‑loop harness that (a) shortens older tool outputs as the context fills, (b) detects repeated/stalled patterns, and (c) issues fixed interventions markedly increased mean per‑task progress (fail‑to‑pass fraction, F2PF) and—on two SWE‑bench cohorts—raised the number of fully resolved tasks. The same frozen harness package also improved outcomes across several different model designs without retuning.

Key quantitative examples (Qwen3.6, 169 Verified tasks, 20,480‑token window, 480s attempt budget): - Mean per‑task F2PF: 28% (control) → 49% (treatment). - Complete solutions: 43 → 72 tasks. Other pressured comparisons: - SWE‑bench Pro: F2PF 15% → 33%; complete solutions 31 → 72. - FeatureBench: F2PF 11% → 20%; complete solutions 2 → 3.

Key Points

• Harness vs. model: The study treats the coding agent as model + harness (solver). Changes to harness alone (same weights, same tasks, same tools) produced large outcome differences.
• Treatment components:
  • Working‑view shortening: keeps a complete in‑memory record but builds a smaller per‑step view by shortening older tool results in logarithmic age tiers (activation when the estimated prompt reaches 50% of window). Newest 4 tool results stay verbatim; older results are truncated with begin/omission/end preserved.
  • Detector & interventions: pattern detector flags repeated failing commands, repeated reads without edits, and other stalls; harness emits deterministic, fixed reminders/safeguards (no model calls) to break loops or prevent predictable command problems.
  • Command safeguards: small deterministic fixes (e.g., normalizing a test command or blocking forbidden commands).
• Metrics:
  • Resolution: whether the evaluator accepts the task as fully solved (binary).
  • Fail‑to‑pass fraction (F2PF): fraction of tests that failed before patch and pass after it; captures partial repair.
• Experimental design:
  • Paired comparisons: each task run twice (control vs treatment) with identical model weights, context capacity, tools, evaluator, and run protocol; only configuration package differs.
  • Statistical tests: exact two‑sided McNemar for resolution (paired binary), exact two‑sided sign test for task‑level F2PF direction; repository‑level sensitivity via resampling.
• Context dependence:
  • Gains concentrated under context pressure (tight windows); gaps shrink at very large windows (e.g., 262,144 tokens).
  • Treatment sometimes reduced per‑turn prompt tokens in wide‑window runs (efficiency effect).
• Generalization: Without retuning, the same frozen treatment package raised endpoints for three additional models with different architectures (reported in paper), indicating harness gains are not narrowly model‑specific.
• Auditability and record: The treatment preserves a complete run record (trace files) while only altering the model-facing view; all shortening decisions are deterministic and auditable.

Data & Methods

• Models: Primary comparisons with Qwen3.6‑35B (llama‑server, Q4 K XL weights) with multiple configured context windows. Additional models tested under the same frozen treatment package (details in paper Sec. 5).
• Benchmarks:
  • SWE‑bench Verified: 169 fixed tasks used across windows and models, locally sealed images.
  • SWE‑bench Pro: 316 demand‑screened tasks (benchmark‑supplied images).
  • FeatureBench: 183 canonical tasks runnable in evaluation setup (benchmark mask used).
• Context regimes: multiple window sizes to induce/relieve context pressure. The main pressured comparisons use 20,480 tokens (Verified).
• Run protocol: fixed attempt budget (e.g., 480s), no network access, same starting repository state per task, same tool interfaces and evaluators per pair.
• Treatment specifics:
  • Shortening schedule (“half‑life”): activates at 50% estimated prompt; newest 4 tool results full; tiers cap characters as age doubles (exact caps in paper).
  • Detector: rule‑based checks on the recorded transcript (no model calls) to identify repeated command failures or no‑edit rereads.
  • Intervention: deterministic instructive message suggesting a different action; also small command normalizers and blockers.
• Metrics & scoring:
  • F2PF computed per task (numerator: target tests made to pass; denominator: target F2P tests).
  • For tasks with zero denominator in final record, operational F2PF set to zero for means.
  • Paired tests: sign test for F2PF (direction), McNemar for resolution. Repository‑level bootstrap sensitivity analyses reported.
• Coverage and robustness:
  • Shortening occurred in substantial fractions of active runs (e.g., 82.8% of Verified 20,480 runs had shortening events).
  • The treatment package was frozen prior to the primary reported campaigns (no task‑disjoint hyperparameter tuning claimed).

Implications for AI Economics

• Evaluation as a joint asset (model + harness):
  • Economic value accrues not only to model weights but also to harness engineering. Benchmark claims about model capability should explicitly name harness/configuration used: harnesses materially affect observed performance.
  • Procurement and contracting should treat harnesses as deliverables/competencies—buyers pay for both model weights and harness engineering.
• Productization and competition:
  • Firms can differentiate on harness design (view policies, loop detectors, interventions) as an economically valuable, low‑latency lever to improve agent outcomes without retraining models.
  • A market for harness optimization, best‑practice libraries, and licensed harness components (auditably deterministic interventions, view managers) is plausible and could be high ROI compared with expensive model retraining.
• Measurement, benchmarking, and incentives:
  • Benchmarks and leaderboards should standardize and disclose harness configuration and context regimes (window sizes, stopping rules). Otherwise, comparisons are confounded and incentives may push towards opaque harness tricks.
  • Partial progress metrics (like F2PF) are valuable economically: they capture incremental value that all‑or‑nothing resolution misses (useful for pricing, SLAs, and staged contracting).
  • Evaluations should report sensitivity across context pressure regimes; performance that depends heavily on harness tuning under small windows may tell a different story about general model capability than results under unconstrained contexts.
• Cost and efficiency implications:
  • Shortening older outputs and reducing per‑turn prompt tokens can lower token costs and latency—important considerations in operating budgets and pricing models for agent services.
  • Deterministic harness interventions may reduce wasted model calls (by breaking useless loops) and therefore reduce compute spend per resolved task.
• Risk, transparency, and regulatory considerations:
  • Because harness interventions can change agent behavior deterministically, regulators and auditors should treat harness code as part of the system under review (for safety, claims, reproducibility).
  • Audit trails that preserve full records (while showing a shortened view to the model) are valuable: allow post hoc analysis, compliance checks, and dispute resolution.
• Research & investment guidance:
  • Investing in harness R&D (view management, loop detectors, intervention design) can be a high‑leverage, lower‑cost strategy to improve deployed agent performance relative to model retraining.
  • Public benchmark organizers should provide harness‑neutral baselines and/or multiple standardized harness configurations so that models are compared under common solver definitions.
  • For research funding and valuation, count harness engineering as part of algorithmic capability and incorporate that into due diligence on startups and teams.

Limitations and open questions (relevant to economic valuation and deployment): - The study uses a particular mechanical shortening policy and fixed detector rules; alternative approaches (model‑written summaries, retrieval + reranker views, or learned intervention policies) were not compared here. - Gains are concentrated under context pressure; at very large windows the harness effect shrinks—investment returns depend on the deployment environment (constrained vs. unconstrained contexts). - The treatment was not retuned per model; although it generalized across several models here, firms may still gain by co‑designing harness and model for maximal performance.

Practical recommendations for stakeholders - Researchers and benchmarkers: always report the harness configuration and context windows; include F2PF or similar partial‑progress metrics. - Buyers and product managers: evaluate both models and harnesses as a combined product; budget for harness engineering and prefer solutions with auditable run records. - Regulators and auditors: require disclosure of harness actions that deterministically alter model inputs, and insist on preserved complete traces for audits.

Summary sentence A relatively small, deterministic harness package (shortening old tool outputs + pattern detector + fixed interventions) materially improved what unchanged model weights could accomplish under context pressure—implying that agent performance claims, procurement, and valuation should treat model weights and harness as a single, economically relevant solver.