Optimizing the scaffolding around small enterprise agents greatly boosts their performance, but copying a stronger model's full trajectories under that tailored scaffold can backfire because the weaker model adopts plans it cannot execute. A targeted, on-policy correction that only replaces the failing turn retains the harness–model fit and recovers gains, enabling safe co-evolution of harness and weights.

Main Finding

Evolving a task-specific agent harness (prompts, tools, hooks, context management) can lift a weaker model to near-expert performance, but naive imitation of a stronger expert under that evolved harness often harms the weaker model by breaking its fit to the harness. Instead, an on‑policy, localized expert-correction pipeline—where an expert rewrites only the failing turn in the weaker model’s own rollouts—preserves model–harness fit and yields safe, incremental model gains. This enables a practical co-evolution loop that composes harness optimization with lightweight model adaptation.

Key Points

• Harness evolution is a powerful, low-cost lever: evolving the harness for a weaker model raised mean task success from 29.2% to 78.0% across seven enterprise tasks (+48.8 points).
• Evolved harnesses transfer upward: a stronger expert (gemini-3.1-pro-preview) improved further on the same evolved harness (84.4% → 93.6%).
• Naive full-trajectory imitation under an evolved harness backfires: LoRA-SFT on expert trajectories caused the weaker model (qwen3-coder-30b) to regress from 78.0% → 63.1% (−14.9 points on average), with per-task drops of −4 to −30 points (e.g., payroll auditing −29.9, website management −20.3, browser automation −16.0).
• The same imitation procedure helps under the unevolved (baseline) harness (29.2% → 35.5%), isolating the failure to the interaction between imitation and an evolved, model-specific harness.
• Diagnosis: imitation transfers expert knowledge but also transfers the expert’s planning style. The weaker model copies the expert’s planning without having the competence to execute it, producing new planning failures and breaking the harness–model fit that the evolved harness relied on.
  • Evidence: use of evolved harness edits increased (e.g., domain-computation recipe: base Qwen 30.8% → SFT-imitation 76.1%), while planning-failure share rose markedly under imitation (planning failures: 1.1% → 14.6% of failures).
• On-policy expert correction (automated by a meta-level MLE agent) fixes this: localize the failing turn in the weaker model’s own rollout, have the expert rewrite only that turn, then LoRA-SFT on those minimally edited rollouts. Results: mean success 78.0% → 79.7% (+1.7), gains on 5 of 7 tasks (notably website +5.6) and preservation of low planning-failure rates.
• Findings replicate across model families (Qwen and Gemma students; Gemini expert), showing the phenomenon is not specific to one model pairing.

Data & Methods

• Tasks: seven objectively verifiable enterprise agentic benchmarks (payroll auditing, budget approval, stock alerting, IoT anomaly detection, browser automation, website management, code refactoring) drawn from Yang et al. (2026) with environment adjustments.
• Harness evolution: GEPA-style search (meta-agent proposes failure-driven edits; keep edits that improve validation) optimizing system prompt, tools, execution hooks, context-management scaffolding, and sub-agent configs. Evolved harnesses were developed around the weaker student model.
• Models:
  • Weak students: qwen3-coder-30b-a3b-instruct (30B total / 3B active) and gemma-4-26b-a4b-it (26B / 4B active).
  • Strong expert: gemini-3.1-pro-preview.
• Model adaptation: parameter-efficient fine-tuning via LoRA-SFT on collected trajectories (converted into target model’s input/tool-call formats). Training and evaluation on held-out test splits; reported means over three runs.
• Imitation workflow: collect full successful expert trajectories under evolved harness → LoRA-SFT weaker model on these trajectories (mixed with student successes).
• On-policy expert-correction workflow:
  • Run the weaker model under evolved harness; for failed rollouts, a meta-level MLE agent automatically localizes the single failing turn (the point where the run goes wrong).
  • Expert rewrites only that failing turn; surrounding student-turns are left intact.
  • LoRA-SFT on the minimally edited rollouts.
• Evaluation: task success rates, harness-component adoption rates, taxonomy of failure modes (implicit knowledge vs. planning, etc.) to diagnose what changed after adaptation.

Implications for AI Economics

• Cost-effective capability via harnessing:
  • Evolving harnesses is an inexpensive way to get large performance boosts for weaker, cheaper-to-run models (e.g., a 30B model) instead of relying on more expensive frontier models at inference time.
  • Harness edits (tools, hooks, prompts) are a high-value engineering investment that can be reused and transferred across model families.
• Beware of inefficient or harmful transfer:
  • Naive imitation of stronger models after harness evolution can negate harness gains, producing regressions that waste compute and engineering effort (i.e., a model update intended to improve performance can reduce it substantially).
  • Economic risk arises when teams combine harness engineering and model fine-tuning without accounting for model–harness coupling.
• Better allocation of tuning budgets:
  • Use harness evolution first to unlock low-cost capacity, then use lightweight, targeted model updates (LoRA) with on-policy, localized expert signals to safely capture remaining headroom. This sequence yields better ROI on fine-tuning compute.
  • On-policy correction reduces the amount of expert supervision needed (only failing turns are rewritten), which lowers annotation/computation cost vs. collecting full expert trajectories.
• Operational design principle for agent deployment:
  • Preserve model–harness fit: when a harness is optimized for a model’s native planning style, updates to model weights should avoid wholesale shifts in planning strategy.
  • Automatable meta-agents (MLE agents) can localize corrections, reducing reliance on human experts and enabling scalable co-evolution loops.
• Strategic tradeoffs for product teams:
  • If inference cost is a major constraint (enterprise deploys), prioritize harness engineering + targeted student adaptation rather than upgrading to larger models.
  • For continual improvement pipelines, interleave harness evolution and on-policy correction: evolve harness around current model → run student rollouts and harvest on-policy corrections from expert → fine-tune student → repeat. This composes benefits while limiting destructive interference.
• Research and monitoring needs:
  • Measure and monitor model–harness fit (e.g., distributional differences in planning steps, failure-mode shifts) as a KPI in adaptation pipelines to avoid regressions.
  • Further economic analysis: quantify compute and human-in-the-loop costs of harness evolution vs. full imitation vs. on-policy correction to derive concrete cost-per-point-of-success metrics for deployment decisions.

Limitations / cautions - Experiments used seven enterprise tasks and particular models and LoRA-SFT; magnitudes may vary in other domains or for different tuning budgets. - The approach relies on an automated meta-agent to localize failures—quality of localization affects efficiency. - Future work should quantify absolute compute/human costs and explore interactions with other adaptation methods (RLHF, offline RL, etc.).