New benchmark maps the game-development lifecycle for coding agents and finds they can reliably create playable foundations from natural-language briefs, but routinely fail to discover hidden bugs, verify runtime behavior, and preserve functionality across iterative repairs and optimizations.

Main Finding

GameXpert-Bench (GAMEXPERT-BENCH) introduces a lifecycle-oriented benchmark for coding agents in game development and shows a consistent capability asymmetry: current coding agents are relatively good at producing playable foundations and implementing explicit requirements (initial generation), but substantially weaker at autonomously discovering defects, verifying runtime behavior, and preserving functionality across iterative changes (repair and multi-turn optimization). Initial-generation quality alone is an insufficient metric for agent competence in end-to-end game development.

Key Points

• Lifecycle framing: Through analysis of human–agent trajectories the authors define three user-facing lifecycle stages for coding-agent game development — Generation → Fix (bug diagnosis & repair) → Optimization (multi-turn refinement). GAMEXPERT-BENCH operationalizes all three as complementary tracks.
• Three tracks:
  • GAMEGEN (single-request generation): agent starts in a blank workspace and must produce a complete browser-native game from a natural-language brief.
  • GAMEFIX (diagnosis & repair): starts from human-verified “Gold Games” with reversible mutations (19–27 injected defects per level); evaluates both explicit-issue repair and agent self-discovery repair. Uses deterministic Fail-to-Pass and Pass-to-Pass checks to ensure fixes restore broken behavior without regressions.
  • GAMEOPT (human-guided optimization): multi-turn chains seeded from real human–agent development trajectories; agents must incorporate successive product-level requests while preserving earlier behavior.
• Benchmark scale and composition:
  • GAMEGEN: 97 games across 11 genres, including 44 requiring 3D rendering; three graded difficulty levels.
  • GAMEFIX: 50 confidential Gold Games; each level mutated to create 100 repair tasks per run (different reporting conditions).
  • GAMEOPT: 17 optimization chains, each with 6 turns, totaling 102 requests.
• Evaluation approach:
  • Emphasis on runtime-executable evidence (artifact behavior) rather than source-only checks.
  • GAMEGEN creates Shared Rubrics via cross-model event analysis (core vs. bonus events), then uses runtime probes and human assessment for player-facing qualities.
  • GAMEFIX uses reversible mutations with corresponding gold patches and deterministic probes to verify both that errors are fixed and no regression occurred.
  • GAMEOPT evaluates final artifacts on gameplay, level design, balance, art, interface, and audio, and checks preservation of prior behavior (evidence-grounded judge).
• Empirical takeaway: agents typically produce interactive, playable cores but often fail to (a) find hidden defects, (b) validate dynamic behaviors reliably, and (c) maintain full functionality across iterative changes — especially in multi-bug or multi-turn settings.

Data & Methods

• Corpus and task construction:
  • GAMEGEN: 97 briefs; tasks intentionally varied in complexity and mechanics (core vs bonus event taxonomy).
  • GAMEFIX: 50 human-verified Gold Games used as clean baselines; each level mutated by reversible operators producing 19–27 defects, preserving an exact gold patch for verification.
  • GAMEOPT: 17 chains derived from real human–agent interactions; experts calibrated starting snapshots, quality constraints, and rubrics.
• Evaluation mechanics:
  • Shared Rubrics: cross-model event extraction followed by human curation produces a task-level checklist distinguishing essential (core) behaviors from optional (bonus) content.
  • Runtime-grounded verification: automated behavioral probes, deterministic tests, and human evaluation (for player-facing judgments) ensure that scoring reflects the playable artifact.
  • Regression-aware verification: Fail-to-Pass (fix broken behavior) and Pass-to-Pass (preserve previously correct behavior) gates used in GAMEFIX.
  • Multi-turn evidence: GAMEOPT uses an evidence-grounded judge integrating code, rendered output, and gameplay traces to measure improvement and regression across turns.
• Agents and setup:
  • Generation experiments run from-scratch in a blank workspace with no supplied templates or assets; example agent used in protocol description: Claude Code (no task-specific scaffolding).
  • Evaluation emphasizes zero-shot and single-session capability for generation track, controlled mutation+repair for fix track, and replayed multi-turn continuity for optimization track.
• Limitations noted by authors:
  • Focus on browser-native artifacts and blank-workspace setting may not capture studio workflows tied to specific engines (e.g., Unity, Unreal).
  • Some materials (Gold Games) are confidential to support reversible mutation design and reliable gold patches.

Implications for AI Economics

• Productivity vs. quality trade-offs:
  • Coding agents can accelerate initial prototyping and reduce time-to-first-playable, lowering upfront labor costs for small teams or solo creators. However, lower reliability in defect discovery, dynamic verification, and regression control implies follow-up human QA, debugging, and iteration remain necessary — shifting rather than eliminating labor.
• Labor demand and skill complementarities:
  • Demand will likely shift from routine implementation (initial build) toward higher-skilled roles: QA engineers, gameplay/system designers, and integrators who can verify dynamic behavior and guide agents through complex multi-turn refinements. Human-in-the-loop roles gain importance.
• Cost structure and specialist service markets:
  • New service markets will emerge for agent-mediated testing, regression validation, and iterative polishing. Firms may invest in tooling and workflows that bridge agent strengths (generation) and weaknesses (verification/maintenance). Monetizable products include automated behavioral test suites, regression-proofing toolchains, and platform-specific agent adapters.
• Capital vs. labor substitution and model/provider concentration:
  • Large, high-capability model providers (or platform-integrated agents) could capture significant value if they supply both generation and robust verification stacks. Firms that internalize agent orchestration and QA pipelines may gain competitive advantages. Smaller studios may substitute certain labor costs with agent usage but still face QA overheads.
• Investment and R&D priorities:
  • Economic incentives for R&D should prioritize agent abilities for defect discovery, runtime verification, and safe iterative changes — areas with outsized downstream cost impacts. Investments into datasets and benchmarks that emphasize lifecycle evaluation (not just generation) will better align capabilities with market needs.
• Pricing, risk, and adoption dynamics:
  • Buyers will discount agent-generated artifacts that require significant human debugging, creating nuanced pricing and adoption decisions. Firms deploying agent-driven pipelines must price in verification costs and potential regressions; offering guarantees (e.g., regression-tested releases) can be a differentiator.
• Measurement and productivity accounting:
  • GDP/labor productivity measures that count lines of code or delivered artifacts may overstate actual productive value if artifact quality and maintainability are not properly measured. Lifecycle-aware metrics (time-to-stable-release, QA hours per feature) become more relevant for assessing economic impact.
• Policy and labor adjustment:
  • Workforce retraining should focus on higher-value complementary skills (QA automation design, systems thinking, human-centered design) rather than low-level coding tasks that agents can increasingly perform.

Overall, GAMEXPERT-BENCH highlights that evaluating AI’s economic impact on software (and creative) production requires lifecycle-aware metrics: initial automation reduces marginal costs of prototyping, but durable economic effects depend critically on agents’ abilities in discovery, verification, and iterative preservation — capabilities that shape ongoing labor needs, quality assurance markets, and firm-level investment strategies.