Pooling rewards across multiple production harnesses during RL training concentrates improvements on the source configurations but does not produce more portable coding agents; in their experiments Cross-vs-Within grouping shows an insignificant held-out difference (Cross − Within = +0.25 pp, 95% CI [−0.48, +1.02]), while the choice of evaluation harness shifts mean solve rates fourfold (2.14% to 9.27%).

Main Finding

Pooling outcomes from multiple production harnesses during group-relative policy optimization (GRPO) makes the advantage signal carry harness identity, but that added signal produces configuration-specific adaptation rather than any detectable increase in portable capability. In controlled, same-data experiments with coding agents, the choice of evaluation harness dominates solve rates (factor of 4.3), while the Cross vs Within grouping rule produces negligible change on an unseen (“held-out”) harness: Cross − Within ≈ +0.25 percentage points (pp) [−0.48, +1.02] at 8 attempts, and pooled across seeds +0.16 pp [−0.41, +0.72]. Pooling does permit a classifier to recover the generating harness from the pooled advantage (+4.48 pp above shuffled-label null), but that recovered harness identity does not translate to better held-out performance or changed action distributions across harnesses.

Key Points

• Experimental isolation: The only difference between the two primary arms was the GRPO grouping boundary.
  • Within: group = task × harness (separate baseline per task–harness).
  • Cross: group = task (pool harnesses for the same task into one baseline).
• Dominant effect is the evaluation harness:
  • Mean solve rate across evaluation harnesses ranged 2.14% → 9.27% (factor 4.3) over 24,000 sealed evaluations.
  • Variation across training recipes was much smaller (recipe means span ≈ 5.55% → 6.46%).
• Portability (held-out harness) results:
  • No recipe, including Cross pooling, produced a reliably higher held-out score than Within or the supervised warm start.
  • Cross − Within on held-out weak-ReAct: +0.25 pp [−0.48, +1.02] at k=8; pooled seeds: +0.16 pp [−0.41, +0.72].
  • Seed variation exceeded the observed Cross vs Within difference.
• Harness identity contamination:
  • A classifier trained out-of-fold can recover the generating harness from Cross’s advantage at +4.48 pp above its shuffled-label null; Within’s advantage carries no recoverable harness signal (+0.02 pp).
  • Coverage matching and a “residualized” baseline retained part or all of this contamination respectively.
• Source vs held-out behavior:
  • Both Cross and Within concentrate most gains on the same source harness (OpenHands).
  • Cross and Within reach the same held-out score and produce the same action distribution within each harness.
• Re-collecting half the training data on-policy did not change the held-out result.

Data & Methods

• Domain and tools:
  • Repository-level coding tasks drawn from an SWE-Gym pool.
  • Four production harnesses collected frozen trajectories: Aider, OpenHands, Qwen Code, SWE-agent.
  • Supervised warm start: Qwen3-8B SFT checkpoint.
• Training design:
  • Frozen trajectory dataset: same records, tokens, binary sealed-oracle rewards per episode.
  • Two GRPO grouping rules compared while holding records, update budget, reward definitions, and SFT start fixed.
  • GRPO advantage: standardized reward within each group (subtract group mean μ, divide by σ, small ε).
  • Additional arms: coverage-matched and residualized baseline variants; one Cross run re-collected second-half of training data on-policy.
• Evaluation:
  • Sealed per-instance oracle: SWE-bench Verified on 500 tasks (four source harnesses) and a held-out minimal harness (weak-ReAct).
  • Primary campaign: 6 checkpoints × 4 source harnesses × 500 tasks × 2 attempts = 24,000 attempted episodes (99.1% graded).
  • Held-out scored at k=4 and k=8 attempts per task in different contrasts; some analyses pooled three training seeds.
  • Behavioural analysis: action labeling of assistant turns; Jensen–Shannon divergence to measure shifts across harnesses.
  • Harness identity test: out-of-fold classifier trained to recover harness from advantage vectors; null estimated by shuffling harness labels within tasks.
• Statistical reporting:
  • Confidence intervals provided for contrasts (e.g., Cross − Within).
  • Power / smallest-detectable differences reported (changing with attempts and pooling seeds).

Implications for AI Economics

• Measurement and benchmarking:
  • Model–harness interaction is a major confound: reported gains can largely reflect harness/configuration effects rather than model capability. Economists and stakeholders should treat harness as a key experimental variable when comparing agent investments.
  • Leaderboards, benchmarks, and procurement metrics should require explicit reporting of the grouping boundary (how rewards are normalized / grouped) and demonstrate results under unseen harnesses to support claims of portable capability.
• Productization and R&D prioritization:
  • Investment claims based solely on multi-harness RL that pools harnesses should be treated cautiously: pooling can amplify configuration-adaptive improvements that do not generalize to new interfaces or production contexts.
  • For product decisions (e.g., deploying an RL-updated agent), vendors should test under the target production harness or an unseen-harness proxy to estimate portability and avoid overpaying for configuration-specific gains.
• Cost of evaluation and deployment:
  • Because the evaluation harness explains far more variance than grouping rules, resources should prioritize robust, cross-harness evaluation (including held-out harnesses) over incremental recipe tweaks whose portability is uncertain.
  • Organizations should budget for harness diversity in validation sets; failing to do so risks investing in policies that only perform well under a narrow set of operational practices.
• Methodological guidance for economists and policymakers:
  • When assessing claims about RL-induced capability improvements, require (a) frozen-data ablations that isolate credit-assignment choices, and (b) held-out-harness evaluations to separate portable capability from configuration adaptation.
  • Consider mandating transparency on how credit is assigned across environments (grouping rules, baseline estimation), since those choices can embed environment-specific signals into the learning objective and mislead downstream stakeholders.
• Research & market implications:
  • Future work (and products) that aim for general-purpose agent improvements should focus on credit-assignment techniques that remove environment-provided offsets (e.g., per-harness residualization, finer-grained per-step objectives, process supervision) and validate portability explicitly.
  • Market comparisons of agent capabilities should incorporate harness-robustness metrics to better reflect real-world value across heterogeneous deployment contexts.

Limitations to bear in mind: single model family (Qwen3-8B warm start), four production harnesses, binary episode rewards, and frozen-manifest replay. Conclusions may not fully generalize to other architectures, reward structures, or to setups with richer on-policy collection across more diverse harnesses.