Tuning the scaffolding around frozen LLMs dramatically raises enterprise-agent performance: StarHarness’s evolved harnesses lift task success by 20–35 percentage points and cut inference costs (up to 53%), with frozen harnesses transferring across GPT and Qwen models and improving held-out tasks.

Main Finding

StarHarness is a practical framework for evolving environment-specific agent harnesses (prompts, tool interfaces, skills, subagents, execution policy, etc.) around a frozen LLM. By searching over a compact, failure-mode–stratified task subset with proposer-visible search tasks and proposer-hidden selection tasks (and held-out evaluation), StarHarness produces harness edits that (a) substantially raise end-to-end performance on stateful enterprise benchmarks, (b) reduce inference cost and trajectory length, and (c) transfer without re-evolution across model families (GPT and Qwen). Typical gains are +20–35 percentage points on full benchmarks after only 4–12 accepted edits per environment.

Key Points

• Objective: optimize the executable harness H around a fixed model M to maximize performance on held-out tasks (no model weight changes).
• Search protocol innovations:
  • Stratified evolution pool: sample K ≈ N/2 tasks by baseline failure-mode, baseline score, and verifier pass rate.
  • Partition into proposer-visible search set, proposer-hidden selection set, and held-out set for final evaluation.
  • Proposer/validator/evaluator loop with scoped git-diff patches, smoke test, proposer-selected test flip, and deterministic acceptance only if selection mean improves.
  • Guardrails prohibit task-ID branching, hard-coded answers, verifier tampering, or hidden-state leakage.
  • Two search modes: tree search (explore alternative hypotheses) and hill-climbing (exploit best frontier).
• Benchmarks used:
  • ITBench SRE (40 Kubernetes root-cause scenarios)
  • EnterpriseOps-Gym ITSM (103 ServiceNow ITSM workflows)
  • AutomationBench Finance (100 finance workflows across 47 simulated SaaS apps)
• Models and transfer:
  • Harnesses were evolved with GPT-5.4 (proposer and agent) and then evaluated, without re-evolution, on multiple GPT and Qwen models (e.g., Qwen3.6, GPT-5.5).
  • Frozen-harness transfer improved every tested model across benchmarks.
• Typical edits learned (21 accepted patches across 3 runs): three recurring categories
• Interface repair — fixing schemas/argument handling and pruning misleading fields to reduce invalid tool calls.
• Environment conventions — encoding implicit operational contracts, ordering/triage rules, and preservation of relationship fields.
• Operational knowledge / search compression — deterministic calculators, forensics overviews, and routines that move repeatable work out of open-ended reasoning.
• Measured behavioral effects:
  • Higher true-positive rates, fewer false positives.
  • Shorter trajectories (fewer turns) and fewer tool calls.
  • Lower estimated inference cost per task.
• Quantitative highlights:
  • Full-benchmark improvement (evolved Stirrup harness vs default): +13.8 to +22.3 to +17.6 pp vs a prompt-optimized Pi baseline on the three benchmarks; overall quoted improvements of 20–35 pp after evolution.
  • Cost reductions at published GPT-5.4 rates: ITBench −17%, EnterpriseOps-Gym −53%, AutomationBench −29%.
  • Specific full-run examples (GPT-5.4):
    • ITBench: score 40.0% → 75.0% (+35.0 pp), turns/task 25.2 → 22.1, false positives 0.79 → 0.33.
    • EnterpriseOps-Gym: 23.3% → 43.7% (+20.4 pp), turns/task 18.12 → 9.87, tool calls 29.53 → 16.83, verifier pass rate 34.5% → 72.8%.
    • AutomationBench: domain objective 57.1% → 83.2% (+26.1 pp), guardrail violations 33 → 4.
  • Held-out generalization (GPT-5.4): ITBench +45.0 pp on evolution set and +31.7 pp held-out; EnterpriseOps-Gym +22.0 / +15.1; AutomationBench +23.0 / +29.3.
  • Cross-model frozen-harness examples: Qwen3.5 ITBench 25.6% → 70.0% (+44.4 pp); Qwen3.6 AutomationBench 48.2% → 75.5% (+27.3 pp).

Data & Methods

• Benchmarks:
  • ITBench SRE (40 reproducible Kubernetes incident snapshots; structured JSON diagnoses).
  • EnterpriseOps-Gym ITSM (103 tasks evaluated by SQL verifiers against a ServiceNow MCP backend).
  • AutomationBench Finance (100 finance workflows scored by programmatic assertions; guardrail violations → zero).
• Partitioning / stratified sampling:
  • From reproducible tasks N', sample evolution pool K (≈N/2) matching baseline failure modes, baseline score, and verifier-pass distributions.
  • Split K into search (visible to proposer) and selection (hidden) sets; remaining N' − K is held-out for final evaluation.
• StarHarness optimizer components:
  • Proposer: reads current harness, search traces, and persistent ledger; emits a scoped git-diff patch.
  • Validator: scope/import checks and single-task smoke test; forbids forbidden edits.
  • Evaluator: proposer-selected quick flip test, then hidden selection set evaluation; acceptance only when selection mean strictly improves (or tie-breaker via verifier metric).
  • Ledger: persistent memory recording attempts, accepted patches, per-task outcomes, and candidate tree when using tree search.
• Search strategies:
  • Hill-climbing: single frontier, propose one bounded patch per iteration, accept only strict improvements.
  • Tree search: maintain candidate nodes with parents, explore multiple hypotheses; used to explore interactions before targeted hill-climb refinements.
• Guardrails:
  • No branching on task IDs, no incorporation of ground-truth, no verifier prompt changes, no hidden-state reads; ensures edits aim for reusable environment behavior.
• Experiment protocol:
  • Evolve using GPT-5.4 (medium) as both proposer and agent; evaluate frozen evolved harnesses across GPT and Qwen families.
  • Compare to unmodified Stirrup baseline, Pi, Codex harnesses, and GEPA prompt optimization.

Implications for AI Economics

• Cost-efficiency of adaptation:
  • Harness evolution can produce large end-user performance gains and substantial per-task inference cost reductions without retraining models. This suggests a high return-on-investment for engineering effort focused on harness/tooling versus expensive model fine-tuning.
• Operational scaling and labor substitution:
  • Shorter trajectories, fewer tool calls, and fewer errors (guardrail violations) reduce runtime compute costs and the human oversight/triage burden. For enterprises, harness engineering could lower operational costs and raise automation reliability faster than model upgrades.
• Transferability and economies of scale:
  • Evolved harness artifacts transfer across models without re-evolution, implying reusability across model upgrades and a one-time engineering investment that benefits multiple downstream model deployments. This increases the leverage of platform-level engineering and may shift demand from model retraining to harness/tooling marketplaces.
• Productization and market implications:
  • A market for environment-specific harnesses, schema repair libraries, and operational routines could emerge; vendors who provide high-quality harness engineering may capture disproportionate value even if the underlying LLMs are commoditized.
• Risk, governance, and maintenance costs:
  • Although cheaper than retraining, harness evolution is not free: it requires benchmarked evaluations, proposer capabilities, and software-engineering practices. Guardrails are necessary to avoid data leakage or overfitting; ongoing maintenance will be required as backends and schemas change. These recurring costs must be factored into economic models.
• Effects on compute demand:
  • Because harness improvements can lower per-task inference consumption, they can reduce incremental demand for model compute in production. However, improved task success may increase total automated throughput (more tasks automated), which could raise absolute compute consumption depending on adoption—net effects depend on substitution versus induced-demand.
• Strategic considerations for enterprises:
  • Prioritize harness/tooling investments when integrating frozen-model agents into complex, stateful environments (databases, MCPs, SaaS stacks).
  • Use stratified, held-out evaluation protocols to guard against overfitting and to quantify generalization/value of harness edits.
• Limitations and caution:
  • Efficacy depends on having verifiers, reproducible task traces, and a proposer able to reason about harness code; not all domains will permit the same level of harness repair.
  • There remain residual risks of over-specialization to particular workflows or vendor lock-in if harnesses hard-code environment conventions.

Overall, StarHarness shows that targeted, benchmark-driven harness engineering is a cost-effective lever to close model–environment gaps in enterprise automation; this shifts some of the economic focus from model training/scale to platform engineering, tooling libraries, and reusable harness artifacts.