A new benchmark for long-horizon software evolution reveals a large capability gap: state-of-the-art coding agents solve just 25% of release-note-driven, multi-file tasks, underscoring that current systems struggle with sustained, cross-file engineering work despite strong single-issue performance.

Main Finding

SWE-EVO introduces a release-level benchmark for long-horizon software evolution and shows a large capability gap: state-of-the-art coding agents struggle to implement multi-file, multi-change release deltas. The best model tested (gpt-5.4) resolves only 25% of tasks; models that score ≈70% on single-issue benchmarks (e.g., gpt-5.2 on SWE-Bench Verified) drop to ≈23% on SWE-EVO. Models commonly produce syntactically-applicable patches (≈97–100% apply rate) but fail semantically to complete large, coordinated evolutions.

Key Points

• Benchmark scope and motivation
  • SWE-EVO targets release-to-release evolution rather than single-issue fixes: agents must interpret release notes/SRS and produce coordinated, multi-file changes that correspond to the behavioral delta between two tagged versions.
  • Built to reflect real maintenance/evolution work that dominates developer effort (features + multiple bug fixes + maintenance).
• Dataset and task scale
  • 48 high-confidence, human-curated instances drawn from 7 mature Python projects (conan, dask, dvc, modin, requests, pydantic, scikit-learn).
  • Per-instance averages: specification length 2,390 words; 610.5 lines edited; 20.9 files edited (paper cites “average of 21 files”); 51 functions edited; test-suite size 874 tests with ≈81 FAIL_TO_PASS tests on average.
• Evaluation metrics
  • Resolved Rate (binary): instance is resolved only if all FAIL_TO_PASS and PASS_TO_PASS tests pass after the patch.
  • Patch Apply Rate: syntactic/applicability measure.
  • Fix Rate (new soft metric): fraction of FAIL_TO_PASS tests fixed (only counted if no PASS_TO_PASS regressions); captures partial progress on long-horizon tasks.
• Experimental findings
  • Agents/Scaffolds: evaluated with OpenHands and SWE-agent frameworks (max 100 iterations / 100 LLM calls).
  • Models: 18 models across multiple providers (OpenAI GPT-5 family, GPT-4 variants, DeepSeek, Zhipu, Qwen, Moonshot, etc.).
  • Best semantic performance is low: gpt-5.4 ≈25% Resolved; gpt-5.2 drops from 72.80% on SWE-Bench Verified to 22.92% on SWE-EVO.
  • Patch Apply Rates are very high (≈97–100%), indicating many failures are semantic/coordination rather than syntax.
  • Failure modes: misinterpretation of nuanced release notes, incorrect implementations, tool-use failures, loops/stuck behavior, and regressions.
  • Release-note only vs release-note + PR context: performance declines modestly when PR context is withheld, but the main difficulty remains.
• Benchmark construction & filtering
  • Instances come from real release tags; problem defined by release-note delta between consecutive tags.
  • Execution-based filtering required at least one measurable FAIL_TO_PASS test; manual curation reduced candidates to 48 high-quality instances.

Data & Methods

• Data sources: release notes, linked PR/issue text, start-version code snapshots and gold patches from real open-source projects; seeds from SWE-Bench and SWE-gym were reused for reproducibility and environment parity.
• Task formulation: input = release-note-centered specification (+ optional linked PR/issue text) + full pre-release codebase. Output = a multi-file patch applied to the codebase.
• Filtering: automatic execution checks (ensure measurable behavioral delta and viable execution) + manual verification for SRS alignment and test validity.
• Metrics:
  • Resolved Rate (strict binary pass/fail on relevant tests).
  • Patch Apply Rate (syntactic applicability).
  • Fix Rate (soft metric; proportion of FAIL_TO_PASS tests fixed, but 0 if any PASS_TO_PASS test regresses).
• Experimental setup:
  • Two agent scaffolds: OpenHands (CodeActAgent, up to 100 iterations) and SWE-agent (up to 100 LLM calls).
  • Medium reasoning-effort inference setting balancing cost/accuracy.
  • Internet blocked during evaluation to prevent retrieval of future commits; linked PR/issue text provided when allowed.
• Analysis: per-instance test outcomes, failure-mode labeling (automated labels for common failure categories), and comparisons to SWE-Bench scores.

Implications for AI Economics

• Labor substitution vs augmentation
  • Current LLMs and agent scaffolds show strong low-level capabilities (high apply rates), but low end-to-end success on release-scale evolutions implies substantial human oversight remains necessary for release engineering and system-level changes.
  • Expect near-term economic impact to be in augmentation (productivity boosts for subsystems, test-writing, localization, scaffolding PRs) rather than full automation of release engineering.
• Product & service market implications
  • Demand growth for multi-agent orchestration, context engineering (retrieval, compression, memory), and tool-integrated workflows—these are high-leverage product areas where vendors can differentiate.
  • Enterprise buyers should not assume current “frontier” models will replace release engineers; procurement that assumes full automation risks misallocation of budget and may underinvest in integration/verification tooling.
• R&D and investment signals
  • The benchmark highlights where to invest: long-horizon planning, robust instruction-following for nuanced specs, regression-aware patch synthesis, and better tool-use primitives. Investing in retrieval/memory, hierarchical planning, and stronger environment feedback loops will likely yield outsized improvements.
  • Model scaling helps, but scaling alone is insufficient; scaffold and context-engineering innovations (e.g., hierarchical memory, GraphRAG, meta context optimization) are economically attractive targets.
• Cost structure and deployment economics
  • Long-horizon tasks require many LLM calls and iterations (benchmarks used up to 100 iterations/calls), increasing inference costs. Firms must weigh compute costs against productivity gains; pricing models for AI-assisted engineering services should reflect multi-turn compute overhead.
  • High patch-apply rates combined with frequent semantic failures imply nontrivial QA costs (test runs, human review). This raises the marginal cost of deploying model-produced changes into production.
• Measurement and procurement
  • Benchmarks that capture long-horizon evolution (like SWE-EVO) are essential procurement signals: they better align SLOs for enterprise automation than single-issue benchmarks. Buyers and policymakers should prefer evaluations that measure regression risk and partial progress (Fix Rate) when assessing readiness for production use.
• Labor market & skill demand
  • Shift in skill demand toward roles that orchestrate agents, design SRS and test suites, and build context-management infrastructure. Expect demand for engineers who can integrate LLMs robustly and design verification workflows.
• Risk and governance
  • Regression sensitivity (Fix Rate’s 0 if any PASS_TO_PASS fails) highlights a deployment risk: breaking existing behavior is economically costly. Enterprises must invest in guardrails, deployment staging, and verification to reduce risk from automated evolutions.

Takeaway: SWE-EVO exposes a clear gap between syntactic/surface competence and sustained, multi-file, specification-aligned evolution. For AI economics, that means near-term value will come from tooling, orchestration, and productivity augmentation, while full replacement of release-level engineering remains a future research and investment challenge.