Autonomous coding agents now open tens of thousands of pull requests, but only some stick: documentation and CI tweaks are most likely to be merged, while larger code changes, failing builds and social frictions—duplicate, misaligned or ignored PRs—drive rejections.

Main Finding

Agentic coding tools increasingly contribute pull requests (PRs) on GitHub, but their success is highly task-dependent and many agent-authored PRs fail for socio-technical reasons. Agents succeed most on small, maintenance-style tasks (documentation, CI, build), but fail more often on larger, invasive, or semantically complex changes (performance work, bug fixes). Key drivers of failure are larger code deltas, CI/test failures, reviewer abandonment, duplicate or unwanted contributions, and agent misalignment.

Key Points

• Dataset and scale

  • 33,596 agent-authored PRs across 5 agents (OpenAI Codex, Copilot, Devin, Cursor, Claude Code) in repos with >100 stars.
  • Overall merge rate: 71.48% (24,014/33,596).
  • Agent merge rates: Codex 82.59%, Cursor 65.22%, Claude 59.04%, Devin 53.76%, Copilot 43.04%.

• Task-level outcomes

  • Highest merge rates: documentation (84%), CI (79%), build (74%).
  • Lowest merge rates: performance (55%), fix/bug-fix (64%).
  • Interpretation: agents are more successful at low-risk, maintenance tasks than at complex bug fixes or performance improvements.

• Code-change and CI signals

  • Not-merged PRs tend to be larger: more changed lines and more files.
  • Effect sizes (Cliff’s delta): #LOC changes δ = -0.17, #file changes δ = -0.10 (small-to-medium effects indicating larger changes in not-merged PRs).
  • CI failures are strongly associated with rejection (δ = -0.24, moderate effect). Each additional failed CI check reduces odds of merge by ≈15% (logistic regression coefficient ≈ -0.158).

• Review dynamics

  • Not-merged PRs receive slightly more review comments and revisions (small effects). Not-merged PRs show heavier tails: some undergo long review cycles before rejection.
  • Logistic model: increases in LOC, files, and failed CI checks are statistically significant predictors of rejection; review comment/revision counts were not significant in that model.

• Qualitative taxonomy of rejections (manual coding of 600 sampled rejected PRs; 562 accessible)

  • Reviewer-level: Abandoned / Not Reviewed — 228 PRs (38% of rejected sample). Many agent PRs are closed without human engagement.
  • Pull-request-level: 188 PRs (31%). Dominant subpattern: Duplicate PRs — 142 PRs (23%). Also unwanted feature additions, wrong branch, non-functional scaffolding.
  • Code-level: 133 PRs (22%). CI/test failures — 99 PRs (17%); incorrect or incomplete implementations.
  • Agentic-level: 13 PRs (2%). Misalignment with reviewer instructions and licensing/CLA issues.
  • Overall message: many failures stem from coordination & process mismatch (abandonment, duplicates), not only technical correctness.

Data & Methods

• Primary dataset: AIDev-pop (agent-authored PRs) — 33,596 PRs from five agents, selecting repos with >100 stars.
• Quantitative analysis (RQ1):
  • Metrics: task labels (11 categories), #LOC changes (added+removed), #file changes, #failed CI checks (via GitHub API), #review comments, #review revisions (developer edits during review).
  • Statistical approach: effect sizes (Cliff’s delta) to judge magnitude (avoiding over-reliance on p-values in large samples), kernel density estimates for distributions, and logistic regression to model merge odds.
  • Key regression findings: negative coefficients for LOC, files, failed CI checks (significant), indicating larger/higher-failure PRs are less likely to merge.
• Qualitative analysis (RQ2):
  • Random, stratified sample of 600 rejected PRs across agents (aiming for ±5% margin of error at 95% confidence).
  • Open coding, iterative codebook and hierarchical taxonomy (Reviewer, Pull Request, Code, Agentic levels).
  • Inter-rater reliability: initial round κ = 0.55; after refinement κ = 0.91.
• Limitations noted by authors (implicit/explicit):
  • Observational study on public OSS repos with >100 stars — potential selection bias.
  • Only five agents and timeframe of dataset; not necessarily representative of all agent behaviors or enterprise contexts.
  • Some PRs were deleted or archived (38 of the 600 sampled rejected PRs).

Implications for AI Economics

• Productivity vs. quality trade-offs

  • Agents can deliver high volumes of PRs and achieve good merge rates on low-risk tasks (docs, CI). Economically, this suggests immediate gains in throughput for routine maintenance work and potential labor substitution for low-skill tasks.
  • However, many PRs impose downstream costs (reviewer time, CI compute, broken builds). Productivity metrics that count “PRs authored” or “PRs merged” can misstate net value if they ignore review overhead and failure externalities. Recommend measuring net output: merged PRs adjusted by reviewer-hours, CI reruns, and rework.

• Task reallocation and comparative advantage

  • Agents have comparative advantage on small, localized, low-context tasks. Human engineers remain essential for complex bug fixes, performance tuning, and design-level decisions. Firms should reallocate labor away from routine chores (docs, trivial fixes) toward higher-value tasks where human comparative advantage persists.

• Supervision and transaction costs

  • High rate of reviewer abandonment (38% of rejected sample) signals coordination failures and potential free-riding: agentic submissions may flood maintainers with low-value PRs. This creates negative externalities for maintainers (time spent triaging, noise). Economically, adoption will depend on internalizing these costs:
    • Gatekeeping/approval processes (pre-submission checks, CLA handling, pre-run of repo CI) to reduce triage burden.
    • Pricing models for agent services could incorporate “cost per accepted contribution” or credits for failed/withdrawn attempts.

• Market differentiation for agent tools

  • Tools that specialize in safe, high-acceptance tasks (docs, CI updates, style fixes) or that integrate pre-checks (run tests, ensure CLA compliance, avoid duplicates) will have higher net economic value. Agents that better align with repo workflows (branching rules, contribution norms) will reduce friction and be more valuable.

• Risk, governance, and legal costs

  • Agentic PRs sometimes trigger license/CLA concerns. Organizations will face governance and legal transaction costs if agent outputs lack provenance or contributor agreements. This raises demand for tool features ensuring license compliance, contributor identity, and traceability — a market for governance add-ons.

• Incentives and policy for open-source maintainers

  • Maintainers' cost of handling agent PRs may discourage contributions or spur stricter contribution policies (e.g., disallow agentic PRs, require contributor sign-off). That can alter the OSS ecosystem: if maintainers impose restrictions, the externality shifts back to agent/tool producers and users. Economic models of OSS contribution need to account for these policy reactions.

• Measuring agent economic value

  • Suggested metrics for organizations and researchers: cost-per-merged-PR (including reviewer hours and CI compute), false-positive rate (PRs that fail CI), abandonment rate, duplicate-rate, and time-to-merge. These better capture the economic impact than raw merge counts.

• Research & product implications

  • Investment priorities for agent vendors: (1) pre-submission validation (run project CI/tests locally), (2) duplicate detection / awareness of project history, (3) fine-grained task selection to favor small scoped changes, (4) CLA/licensing integration, (5) better alignment with reviewer feedback loops.
  • For economists: opportunities to quantify labor reallocation effects (which tasks shift to agents), measure changes in maintainer workloads, and model equilibrium wages/roles for developers with agent adoption.

Overall: agents can reduce costs and increase throughput for specific maintenance tasks, but unchecked agentic submissions generate coordination, quality, and legal frictions that impose non-trivial economic costs. Realized economic gains will require tooling and governance that reduce CI failures, duplicates, and reviewer overhead — and careful measurement of net rather than gross productivity.