AI coding agents frequently produce fix-related pull requests, but many are not merged—most often because tests fail or another PR already fixed the issue—revealing limits in correctness and situational awareness that constrain real-world integration.

Main Finding

AI coding agents can generate many viable fix-related pull requests (PRs) that get merged, but a substantial fraction (26.1%) remain closed without merging for reasons that are often not purely technical. Validation gaps (failing tests, incorrect/incomplete fixes) and process/timing dynamics (issues resolved by other PRs, inactivity, lack of review) are the dominant failure drivers. Integration success and latency vary substantially across agents, implying heterogeneous real-world effectiveness and important hidden costs (compute, review effort) from unmerged agent outputs.

Key Points

• Dataset and scope
  • 8,106 fix-related PRs from the AIDEV-POP dataset (Nov 2025) authored by five agents: OpenAI Codex (4,338), GitHub Copilot (1,993), Devin (1,249), Cursor (411), Claude Code (115).
• High-level outcomes
  • 65.0% merged (5,267 PRs), 26.1% closed without merging (2,113), 8.9% open (726).
• Agent heterogeneity (merge rates)
  • OpenAI Codex: 81.6% merged
  • GitHub Copilot: 42.4% merged
  • Devin: 42.9% merged
  • Cursor: 68.4% merged
  • Claude Code: 57.4% merged
• Latency
  • Time-to-merge varies widely; Codex shows shorter and tighter merge times, Copilot and Devin show broader distributions with heavy tails.
• Failure analysis (manual coding)
  • Random sample: 326 closed-but-unmerged PRs; manual open coding (~100 person-hours); Cohen’s kappa = 0.82.
  • Structured catalog of 12 failure reasons. Top three:
    • R1 Resolved by Another PR (RAPR): 22.1% (72/326)
    • R2 Test Case Failures (TCF): 18.1% (59/326)
    • R3 Incorrect or Incomplete Fixes (IIF): 15.3% (50/326)
  • Build failures (2.1%) and deployment failures (3.1%) are relatively rare compared to validation/test issues.
• Agent-specific failure patterns
  • Codex: dominated by test failures (54.9% of its closed-but-unmerged cases).
  • Devin: many closures due to inactivity (54% of its unmerged cases), suggesting engagement or workflow timing issues.
  • Copilot: many cases where the issue was resolved elsewhere (31.8% of its unmerged cases), indicating redundancy/timing effects.

Data & Methods

• Source: AIDEV-POP large-scale dataset of AI-authored PRs (33,596 total; 8,106 fix-related selected).
• Quantitative analysis (RQ1)
  • Classified PRs by final state: merged, closed w/o merge, open.
  • Computed submission-to-merge time as closed_at − created_at (hours); analyzed per-agent distributions.
• Qualitative analysis (RQ2)
  • From 2,113 closed-but-unmerged PRs, drew a statistically representative random sample of 326 (95% confidence, 5% margin).
  • Open coding by two authors after calibration; consensus resolving disagreements.
  • Inter-rater reliability: Cohen’s kappa = 0.82.
  • Derived 12 failure categories inductively and reported their distribution overall and per agent.
• Replication materials: authors provide an online appendix (replication package referenced).

Implications for AI Economics

• Wasted compute and marginal cost of unmerged outputs
  • Each unmerged agent PR represents compute, API, and orchestration costs with no direct codebase benefit. At scale, these make the per-merged-PR cost of AI assistance higher than naive unit prices suggest.
  • Economic evaluations of coding agents should include the rate of unmerged outputs when estimating marginal costs and ROI.
• Hidden labor and review costs
  • Maintainers incur review time and cognitive overhead for unmerged PRs (comments, CI runs, triage). These are labor costs externalized to projects and should factor into cost–benefit analyses of deploying agents.
  • High proportions of “no review conducted” or “closed due to inactivity” indicate inefficiencies in reviewer allocation and coordination.
• Quality-adjusted productivity and labor substitution
  • Aggregate merge rates (65%) suggest agents can substitute some developer effort, but heterogeneity across agents and failure modes implies limited direct substitution without additional validation/interaction investments.
  • Firms considering agent deployment must weigh productivity gains against downstream validation costs (test writing, CI tuning, reviewer time).
• Market differentiation and pricing signals
  • Agent heterogeneity in merge rates and latency is a market feature: agents that produce higher merge rates and fewer validation failures have higher effective value. This supports tiered pricing (e.g., per successfully merged PR or quality-guarantee tiers).
  • Platforms could offer integration performance metrics (merge rate, average time-to-merge, distribution of failure causes) as seller-side signals to buyers (enterprise customers).
• Externalities and coordination frictions
  • Redundancy (issues resolved by other PRs) and inactivity closures reveal coordination externalities: multiple agents or humans working in parallel can produce duplication costs. Incentive mechanisms (e.g., coordination hooks, locking issues, reputation for agent submissions) could reduce waste.
• Policy and governance implications
  • Repos and platforms may need policies to manage agent submissions (rate limits, labeling, required tests) to internalize review costs and reduce noise.
  • Standards that require automated tests, or agent-generated test cases, could raise merge-success rates and lower reviewer burden.
• Research and investment priorities
  • Investing in test-aware, context-sensitive agents (that run and pass repo CI or synthesize tests) is likely to yield the largest economic benefit—reduced unmerged rate and lower review costs.
  • Evaluation metrics for agents should include downstream integration outcomes (merge rate, validation failures) rather than only task-level or benchmark accuracy.
  • Future empirical economic work: estimate monetary costs of unmerged PRs (compute + reviewer hours), compute agent-level ROI across project types, and study how agent deployment changes labor allocation (e.g., more reviewing vs. coding).
• Business model opportunities
  • Value-added services: automated pre-merge validation (CI-run sandboxing), duplication detection, and PR prioritization could be monetized to reduce the externalized costs agents impose on maintainers.
  • Performance-based pricing (pay-per-merged-PR or subscription with SLAs on integration metrics) could align incentives and reduce negative externalities.

Suggested actionable metrics for economic assessments - Effective cost per merged PR = (compute + API + orchestration costs + average reviewer labor cost attributable to agent PRs) / number of merged PRs. - Merge yield = merged PRs / total PRs submitted by agent. - Validation failure share = proportion of unmerged PRs closed due to tests/incorrect fixes. - Duplication rate = share of PRs closed because another PR fixed the issue.

Overall, the study highlights that endorsements of AI coding agents based only on code-generation benchmarks overstate their economic value in collaborative projects; integration outcomes, validation-readiness, and workflow alignment are key determinants of real economic impact.