An iterative 'Harness-of-Harness' loop lets coding agents plan, implement and independently test small increments, boosting benchmark performance by roughly 50% on average and enabling a 70+ iteration autonomous build of a playable game; gains are compelling but currently demonstrated on specific harness–model setups and engineered toolchains.
Harness-of-Harness: Multi-Day Autonomous Software Development with Continual Improvement
HoH (Harness-of-Harness) is an iterative planning–coding–testing framework that enables LLM-based coding agents to sustain continual improvement during autonomous software development, yielding large performance gains on multiple benchmarks and producing a human-playable FPS in a multi-day autonomous run.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
No provider observation is available for this paper.
Missing data, not a zero citation count.
This paper studies autonomous software development, in which LLM-based coding agents transform high-level requirements into complete, functional, and usable software systems without human intervention. We introduce Harness-of-Harness (HoH), a framework that enables coding agents to continually improve software during autonomous development. HoH operates on existing coding-agent harnesses, and organizes their executions into iterative planning-coding-testing loops. To sustain improvement across loops, HoH balances repair with capability growth, scopes development into small and verifiable increments, separates implementation-time testing from independent evaluation, and constrains verifiable outputs rather than prescribing agent workflows. It progressively exposes deliverables, role-specific tools, and skills, encourages reuse rather than recreation, and maintains versioned project histories. On GameCraft-Bench, FrontierSWE, and ProgramBench, three harness-model pairs (Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, and Pi with MiniMax-M3), HoH consistently outperforms the corresponding standalone harnesses, achieving an average relative gain of 52.25 percent and a maximum gain of 82.86 percent after three iterations. In a multi-day deployment with more than 70 iterations, HoH autonomously develops a first-person-shooter game, featuring a coherent storyline, fully implemented core mechanics, human-playable experience, polished visuals and integrated audio. Github: https://github.com/Flesymeb/HarnessOfHarness Project Page: https://flesymeb.github.io/HarnessOfHarness/
Summary
Main Finding
Harness-of-Harness (HoH) is a runtime- and workflow-level framework that wraps existing LLM-based coding harnesses into repeated planning–development–testing loops (Project Planner, Developer, QA Tester). By preserving both the evolving software artifact and a structured evidence state across iterations, and by enforcing small, verifiable increments plus independent acceptance testing, HoH enables sustained, multi-day autonomous software development. Across multiple benchmarks and model–harness pairings HoH substantially improves autonomous development outcomes (average relative gain 52.25%, max 82.86% after 3 iterations) and can autonomously produce a human-playable first-person-shooter game over 70+ iterations.
Key Points
- Core idea: organize autonomous development as iterative loops where each loop (1) selects a bounded objective (Planner), (2) implements it (Developer), and (3) independently evaluates the result (QA Tester). Repeat, carrying forward:
- Artifact state (current code, assets, config)
- Evidence state (structured test reports, validated behaviors, unresolved failures)
- Design principles:
- Balance repair with capability growth; focus on small, verifiable increments to reduce risk and localize faults.
- Separate implementation-time testing from independent evaluation to avoid self-acceptance bias.
- Constrain required outputs (structured schemas) rather than prescribing internal agent workflows.
- Progressive disclosure: persist full histories but expose concise indices; retrieve details only when relevant to avoid context-window pollution.
- Encourage reuse (assets, libraries, proven modules) and maintain versioned project histories to enable rollbacks and learning from prior attempts.
- Implementation mechanics:
- Same underlying harness and LLM are invoked three times per iteration under role-specific prompts; a deterministic Runtime enforces permissions, input access, tool availability, and required structured outputs.
- Testing combines white-box and black-box checks and returns a structured evidence bundle to the planner.
- In extended runs (e.g., game development), role-specific tools/skills (asset generation, engine APIs, git commits, MCP servers) and project-state management were added.
- Empirical results:
- Benchmarks: GameCraft-Bench, FrontierSWE, ProgramBench.
- Harness–model pairs tested: Codex + GPT-5.5, OpenCode + DeepSeek-V4-Pro, Pi + MiniMax-M3.
- After 3 iterations, HoH outperformed standalone harnesses with average relative gain 52.25% (max 82.86%).
- Absolute gains: GameCraft-Bench +16.62–22.08 pts; FrontierSWE +19–29 pts; ProgramBench +6.09–16.85 pts.
- On FrontierSWE, Codex+GPT-5.5 improved from 22% → 72.67% over 10 iterations.
- Multi-day open-ended run: >70 iterations produced a playable FPS with coherent narrative, combat systems, UI, visuals and audio; full trajectory committed on GitHub.
Data & Methods
- Evaluation settings:
- Controlled benchmarks: GameCraft-Bench (game-building tasks), FrontierSWE (from-scratch software engineering tasks with open-ended objectives), ProgramBench (program synthesis / project-level tasks).
- Open-ended: multi-day autonomous game development from a high-level PRD.
- Models and harnesses:
- Three real harness–model pairings to stress-test HoH under different base capabilities: Codex with GPT-5.5, OpenCode with DeepSeek-V4-Pro, Pi with MiniMax-M3.
- Loop mechanics and runtime:
- Each iteration invokes the same harness+LLM three times with role-specific prompts and a Runtime that:
- Freezes inputs, enforces role permissions (what files/tools can be used), binds evidence to the candidate build, and records project state/version.
- Requires structured outputs; schema violations cause retries.
- Cross-loop state: artifact (A_t) and evidence (E_t) persist between iterations; Planner uses (S + E_{t-1} + A_{t-1}) to decide the next bounded objective.
- Testing and metrics:
- Implementation-time unit/integration tests and independent QA tests (white-box + black-box + functional/usability/visual/audio checks).
- Benchmark scoring as defined by each suite; also qualitative human-playable assessments for the open-ended game.
- Reproducibility:
- Initial benchmark experiments used only the base harnesses; the multi-day open-ended run added role-specific tools/skills and committed intermediate states to a public GitHub repo (Flesymeb/HarnessOfHarness).
Implications for AI Economics
- Productivity and output scaling:
- HoH demonstrates a path to automating longer-horizon, integrative engineering work beyond one-off function generation. If generalized, this can substantially increase software output per unit of human supervision, lowering marginal cost of building software products and accelerating product development cycles.
- Labor substitution vs. complementarity:
- Likely substitution for routine, implementation-level and debugging tasks (especially junior developer work), while increasing demand for higher-level roles: product specifiers, system architects, policy/safety engineers, and people who design, audit, and orchestrate agentic workflows.
- QA shifts from manual playtesting to supervising and validating agent-produced evidence bundles; human skills move toward exception handling, specification design, and complex judgment.
- Capital concentration and returns:
- Gains accrue to organizations that control superior harnesses, model access, and tooling ecosystems (e.g., platforms that provide proven HoH-style runtimes, asset markets, or high-capacity models), potentially enhancing winner-takes-most dynamics in software services and product launches.
- Skill-biased technological change:
- Demand increases for skills in agent orchestration, tooling integration, specification engineering, verification design, and model-harness engineering; wages may rise for these specialized roles while compressing entry-level developer wages.
- Market structure and new industries:
- New markets and firms around agent-harness composition, evidence-management services, versioned agent workflows, and automated testing-as-a-service are likely to emerge.
- Reduced time-to-prototype could expand software experimentation, increasing the volume of niche/small apps but also intensifying competition.
- Measurement and economic indicators:
- Standard productivity metrics (lines of code, commit counts) will be inadequate; need new measures capturing verified functionality, delivered user value, and agentic development cost (compute + orchestration).
- Risks and policy considerations:
- Quality, safety, and liability: autonomous development can produce subtle regressions or security flaws that are hard to catch without robust independent evaluation—regulatory clarity on responsibility for deployed code may be needed.
- Distributional impacts: labor displacement risks for certain developer cohorts suggest policy responses (retraining, portable certification for agent oversight skills).
- Access inequality: compute/model costs may centralize capabilities, affecting competitive dynamics and geographic distribution of software production.
- Externalities and long-run effects:
- Faster and cheaper software production could raise consumer surplus and enable new digital goods, but may also accelerate automation-driven displacement elsewhere in the economy.
- Potential productivity growth in software-intensive sectors may amplify complementarities with other AI-enabled automation, affecting aggregate labor demand composition.
Limitations and open questions (economic angle): - Generalizability: results depend on model and harness quality; broad adoption hinges on affordable access to high-capability models and robust tooling. - Hidden costs: compute, engineering of role-specific tools, and maintenance of evidence/version systems impose capital costs that may limit decentralization. - Empirical incidence: the net labor-market effect depends on adoption speed, sectoral exposure, and policy — empirical work is needed to quantify displacement vs. task reallocation.
References and artifacts - Paper repository and artifacts (code, game, development traces) available at Flesymeb/HarnessOfHarness (GitHub) and project page linked in the paper.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper reports consistent, large relative gains across multiple benchmark suites and demonstrates an open-ended multi-day autonomous development producing a playable game, and publishes an output repository; however, evidence is limited to specific harness–model pairings and benchmark tasks, lacks randomized or statistical treatment of variability, and depends on particular (partly proprietary) model/tool configurations, limiting claims about broader causal effects or real-world productivity impacts.
Methods Rigormedium — The evaluation uses multiple benchmarks and model–harness combinations and reports iteration-wise improvements and a long-run case study, which is appropriate for a systems paper; but the manuscript (as supplied) lacks detail on metrics, run-level variability, statistical significance, ablations isolating key design choices, and sensitivity to model/harness choices, and relies on engineered role-specific tools in the open-ended run that reduce the clarity of which elements drive gains.
SampleExperimental evaluation on three benchmark suites (GameCraft-Bench, FrontierSWE, ProgramBench) with three harness–model pairings: Codex + GPT-5.5, OpenCode + DeepSeek-V4-Pro, and Pi + MiniMax-M3; results reported as iteration-wise performance gains (average relative gain 52.25% after three iterations; max 82.86%). Additionally, an open-ended multi-day autonomous development run (>70 iterations) producing a first-person-shooter game with committed artifacts and traces on a public GitHub repository. Exact dataset sizes, number of repetitions, and metric definitions are not fully specified in the supplied text.
Themesproductivity human_ai_collab
IdentificationComparative system evaluation: the authors implement HoH on top of existing coding-agent harnesses and compare performance of the HoH loop to the corresponding standalone harnesses across three benchmark suites (GameCraft-Bench, FrontierSWE, ProgramBench) and a long-running multi-day autonomous development run; there is no formal causal identification strategy (no randomized experiments, no econometric identification, no natural experiment).
GeneralizabilityResults are shown for specific harness–model pairings and may not generalize to other LLMs or future/older model versions., Benchmarks and the open-ended game development are narrow domains compared with large, heterogeneous real-world software projects (scale, team interactions, legacy code)., Open-ended run uses added role-specific tools and skills (asset generation, engine interaction, versioning) that may be necessary for success and are not guaranteed across settings., Proprietary or unreleased models/configurations (e.g., GPT-5.5, DeepSeek-V4-Pro) may limit reproducibility., Metrics and evaluation procedures on benchmarks may not map directly to firm-level productivity, developer time saved, or economic outcomes.
Claims (10)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Harness-of-Harness (HoH) outperforms the corresponding standalone coding-agent harnesses across GameCraft-Bench, FrontierSWE, and ProgramBench. Output Quality | positive | Benchmark performance on from-scratch software-development tasks |
Reading fidelity
high
Study strength
high
|
HoH consistently outperforms standalone harnesses
|
| After three iterations, HoH achieves an average relative performance gain of 52.25% over the corresponding standalone harnesses. Output Quality | positive | Relative benchmark performance after three autonomous development iterations |
Reading fidelity
high
Study strength
medium
|
52.25% average relative gain
|
| The maximum relative performance gain produced by HoH after three iterations is 82.86%. Output Quality | positive | Relative benchmark performance after three iterations |
Reading fidelity
high
Study strength
medium
|
82.86% maximum gain
|
| After three iterations, HoH produces absolute gains of 16.62–22.08 points on GameCraft-Bench. Output Quality | positive | GameCraft-Bench score |
Reading fidelity
high
Study strength
medium
|
16.62–22.08 points
|
| After three iterations, HoH produces absolute gains of 19–29 points on FrontierSWE. Output Quality | positive | FrontierSWE score |
Reading fidelity
high
Study strength
medium
|
19–29 points
|
| After three iterations, HoH produces absolute gains of 6.09–16.85 points on ProgramBench. Output Quality | positive | ProgramBench score |
Reading fidelity
high
Study strength
medium
|
6.09–16.85 points
|
| On FrontierSWE, HoH using Codex with GPT-5.5 continues improving over ten iterations, with performance increasing from 22% to 72.67%. Output Quality | positive | FrontierSWE performance across repeated autonomous-development iterations |
Reading fidelity
high
Study strength
medium
|
n=10
from 22% to 72.67%
|
| In a multi-day autonomous deployment spanning more than 70 iterations, HoH develops a first-person-shooter game with a coherent storyline, implemented core mechanics, a human-playable experience, polished visuals, and integrated audio. Output Quality | positive | Completeness and qualitative usability of an autonomously developed game |
Reading fidelity
high
Study strength
low
|
n=70
more than 70 iterations
|
| HoH organizes autonomous software development into repeated planning, implementation, and independent testing loops, carrying both the evolving software artifact and execution evidence into subsequent iterations. Organizational Efficiency | positive | Continuity and coordination of the software-development process |
Reading fidelity
high
Study strength
low
|
not reported
|
| HoH separates implementation from acceptance by assigning objective selection to a Project Planner, artifact modification to a Developer, and independent acceptance to a QA Tester. Error Rate | positive | Independence and structure of software verification |
Reading fidelity
high
Study strength
low
|
not reported
|