Pooled data suggest co-authored AI pull requests merge less often, but that's an illusion: stratifying by agent and adding repo and PR-structure controls erases the apparent co-authorship advantage, indicating selection and structural confounders rather than a causal productivity gain.

Main Finding

Pooled across five AI coding agents, PRs with a Co-Authored-By trailer appear to merge much less often than purely-autonomous ones (53.8% vs. 79.8%, Δ = −26.0 pp). This aggregate result is a textbook Simpson’s Paradox: stratifying by agent reverses the sign for most agents (e.g., Copilot +41.2 pp, Devin +33.5 pp). But further robustness steps expose a cascade of confounders—repository selection and PR structure largely explain away the per-agent co-authorship gaps. After within-repo controls and PR-structure (commit-count / multi-commit) adjustments, no agent retains a clear co-authorship effect. In contrast, author/committer attribution (human author/committer vs agent author/bot committer vs agent author/human committer) shows a consistent monotone association with merge rates (47.6% → 64.3% → 82.0%).

Key Points

• Pooled (misleading) statistic:
  • Co-authored PRs: 10,764 PRs, merge rate 53.8%
  • Purely-autonomous PRs: 22,832 PRs, merge rate 79.8%
  • Aggregate gap: −26.0 percentage points (χ2 ≫ 0)
• Per-agent heterogeneity (selected figures):
  • Codex (64.9% of PRs): coauth rate 1.2%; MR coauth 78.3% vs pure 82.6% (Δ = −4.4 pp)
  • Copilot (14.8%): coauth 89.4%; MR coauth 47.4% vs pure 6.2% (Δ = +41.2 pp)
  • Devin (14.4%): coauth 95.9%; MR coauth 55.1% vs pure 21.6% (Δ = +33.5 pp)
• Cascade of confounding:
  • Within-repo (repo×agent fixed effects) pooled coauth effect: +15.2 pp (SE 2.5)
  • Copilot within-repo: +36.2 pp; controlling for log(commits) → +24.4 pp
  • Copilot restricted to multi-commit PRs → +4.8 pp (p = 0.59; underpowered)
  • Devin’s within-repo gap collapses to +1.6 pp (p = 0.73)
  • Net: once repository selection and PR structure are controlled, no agent shows a clear causal co-authorship benefit based on these observational data.
• Author/committer classification (n = 33,564 PRs):
  • human_both (human author + human committer): n=23,200, MR=82.0%
  • agent_draft (bot author, human committer): n=336, MR=64.3%
  • fully_autonomous (bot author + bot committer): n=10,028, MR=47.6%
  • This graded pattern persists within commit-count bins and across agents with sufficient data.
• Multi-agent adoption (n=233 multi-agent repos): introducing a second agent is associated with a −12.1 pp change in weekly merge rates (DiD; SE=3.6), possibly due to compositional effects.

Data & Methods

• Data: AIDev dataset; 33,596 PRs from five AI coding agents (Codex, Copilot, Devin, Cursor, Claude Code) over ~218 days (2024-12 to 2025-07). Agent coverage windows differ.
• Key measurements:
  • Co-Authored-By: parsed GitHub commit trailers; PR flagged collaborative if any commit has such a trailer.
  • Author/committer classification from commit metadata to form three collaboration modes.
  • Outcome: binary merge state at collection time; open PRs treated as not merged (robustness checks on closed-only preserve main pattern).
• Statistical approaches:
  • Descriptive proportions and Pearson χ2 tests for baseline comparisons.
  • Stratification by agent identity to reveal Simpson’s Paradox.
  • Repository×agent fixed-effect regressions with cluster-robust SEs for within-repo estimates.
  • Additional controls: log(commit count); restriction to multi-commit PRs to address single-commit draft confound.
  • Difference-in-differences regression for multi-agent adoption analysis (repo fixed effects + linear time trend).
• Limitations explicitly acknowledged by authors:
  • Observational design: associations ≠ causation.
  • Measurement noise: agents auto-insert trailers; some trailers use bot emails (noreply), inflating coauth counts for some agents.
  • Dominance of Codex (65% of PRs) — pooled stats highly sensitive to agent composition.
  • No control for PR size, task type, reviewer experience; survivorship bias (only repos that adopted agents included).
  • Some analyses (e.g., multi-commit Copilot pure-autonomous group) are underpowered.

Implications for AI Economics

• Beware of pooled productivity metrics: aggregate performance measures that mix heterogeneous agents can be severely misleading (Simpson’s Paradox). Economic analysis of AI tools must stratify by agent/model and other salient covariates before drawing conclusions about value or productivity.
• Valuing human-in-the-loop interventions requires careful identification:
  • Observational co-authorship markers are confounded by repo selection and PR maturation; naïve estimates over- or understate the marginal benefit of adding human effort.
  • Author/committer signals appear more robust than coauth trailers and may better proxy for economically meaningful human involvement (e.g., effort, review time).
• Procurement and deployment decisions:
  • Firms buying or licensing coding agents should not rely on headline pooled merge rates to infer agent quality or the ROI of mandating human co-authorship. Decisions should be based on agent-stratified, within-repository comparisons and adjusted for PR structure.
  • Where merge-rate is a proxy for “value delivered,” the true marginal return to human sign-off looks smaller than pooled cross-sectional figures suggest once you control for selection and structure.
• Incentives and labor economics:
  • High coauth rates for some agents may reflect operational workflows (tools that auto-add trailers) rather than greater human effort; incentive schemes that pay or reward based on coauth markers could be gamed or misaligned.
  • The graded author/committer pattern (higher merge rates with more human involvement) implies economic value to human oversight, but causality is unresolved—economic models should treat this as selection-biased evidence until randomized or quasi-experimental estimates are available.
• Market structure and competition among agents:
  • Agent composition matters: a dominant high-merge agent (Codex) can mask poor outcomes from other agents. Economic assessments of competition, market shares, or social welfare should control for such heterogeneity.
  • The observed negative association after introducing a second agent (−12.1 pp) suggests potential quality dilution or adverse compositional effects when multiple agents are adopted; platform and firm-level policies should monitor multi-agent interactions.
• Research & policy recommendations for economists:
  • Use within-repo (or within-project) comparisons, control for PR structure (commit counts, draft markers), and prefer pre-registered randomized trials (e.g., random assignment of mandatory human review or coauth attribution) to identify causal effects of human involvement.
  • When using administrative markers (trailers, metadata) as proxies for human work, validate those proxies across agents and workflows to avoid measurement bias in cost/benefit analyses.
  • Account for power and sample-size limitations when evaluating effects for less-frequent agents or rare collaboration modes.

Bottom line: empirical evaluations of AI coding agents (and of human-AI collaboration value) must avoid pooling heterogeneous agents and must control for repository selection and PR structure; otherwise, policy, procurement, and welfare conclusions will be unreliable.