Pull requests written by different AI coding agents read and behave differently: description style predicts how quickly and positively humans respond and whether changes get merged, with sizable variation in reviewer engagement and merge rates across agents.

Main Finding

AI coding agents produce pull request (PR) descriptions with distinct communication styles, and those differences are associated with measurable variation in human reviewer behavior and final outcomes (engagement, sentiment, merge rates, and time-to-completion). Presentation (structure, verbosity, politeness) correlates with reviewer effort and processing speed, but presentation alone does not determine acceptance—code quality remains central.

Key Points

• Dataset: 33,596 AI-generated PRs from five agents (GitHub Copilot, OpenAI Codex, Claude Code, Devin, Cursor) drawn from the AIDev dataset.
• PR description features: 11 metrics covering work-style (files_changed, additions, deletions, commits), description-style (char, headers, lists, code blocks, emojis, politeness) and PR-compliance (conventional_commit). Features were Z-normalized for comparison.
• Reviewer interaction: analyzed 94,865 raw comment/review records (filtered to 28,961 human comments; 24,813 English comments for sentiment). Metrics: comments_per_pr, comment_length (median), time_to_first_comment (median), sentiment (RoBERTa fine-tuned for SE texts).
• Outcome metrics: merge_rate and time_to_completion (median). Statistical tests (Kruskal–Wallis, chi-square) show significant differences across agents (p < 0.001); medium–large effect sizes for comments_per_pr, merge_rate, and time_to_completion.
• Representative agent-level outcomes:
  • OpenAI Codex: most structured descriptions (headers/lists), highest merge rate (82.6%), shortest median completion (0.02 hours).
  • Cursor: high merge rate (65.2%) and short completion (0.90 h) despite receiving relatively negative sentiment.
  • Claude Code: large textual descriptions, many positive sentiments and long reviewer comments, merge rate ≈59%, completion ≈1.95 h.
  • GitHub Copilot: highest comments per PR (heavy reviewer engagement) but low merge rate (43%) and long completion (13 h).
  • Devin: moderate behavior, merge rate ≈53.8%, completion ≈8.9 h.
• Interpretation: Structured, readable PR descriptions are associated with faster reviewer responses and shorter review cycles; however, merges depend on code quality and other factors (repo norms, task complexity). Sentiment is contextual and should not be equated with technical correctness.

Data & Methods

• Data source: AIDev dataset (pull_request, pr_commit_details, pr_commits, pr_comments, pr_reviews, pr_review_comments_v2).
• Agents studied: GitHub Copilot, OpenAI Codex, Claude Code, Devin, Cursor.
• Feature extraction:
  • Work-style metrics from commits/tables (files_changed, additions, deletions, commits).
  • Description-style metrics via regex on PR body (character count, header/list/code block densities, emoji count, politeness phrase density).
  • PR compliance via Conventional Commits pattern detection.
• Normalization: Z-scores across all PRs to compare heterogeneous features.
• Reviewer analysis:
  • Filtered out bot accounts and empty comments; restricted sentiment to English comments.
  • Sentiment classification via RoBERTa fine-tuned on software engineering corpora (positive/neutral/negative).
  • Engagement/depth/timing metrics: comments_per_pr mean, comment_length median, time_to_first_comment median.
• Outcomes: merge_rate (percentage merged) and time_to_completion median for closed/merged PRs.
• Statistical analysis: Kruskal–Wallis tests for continuous metrics, chi-square tests for categorical distributions; reported p-values and effect sizes.

Limitations noted by authors: - Observational design—no causal claims; potential confounders (task type, repo norms, code complexity). - Automated parsing may mischaracterize description elements. - Sentiment analysis limited to English, possibly biasing agent-level sentiment distributions. - Dataset limited to AIDev / GitHub context; external validity beyond these settings is uncertain.

Implications for AI Economics

• Measuring AI productivity should include communication effectiveness, not only code output:
  • Merge rate and time-to-completion are meaningful economic signals of value; agents that produce clearer PRs can reduce transaction costs (reviewer time).
  • Productivity metrics that ignore reviewer burden (e.g., lines of code or number of PRs) may overstate the economic value of agents that force heavy human review.
• Strategic design choices by agent developers have market effects:
  • Investing in PR-explanation/structure features (headers, lists, concise summaries) can increase acceptance and reduce review time, improving the effective productivity of adopters.
  • Agents optimized only for code generation may externalize costs to human reviewers (longer review cycles, lower merge rates), creating negative platform externalities.
• Labor and task allocation:
  • Better communication from agents can substitute for portions of human review effort, shifting labor toward higher-value verification tasks; conversely, poor communication increases reviewer workload and may raise labor costs.
  • Firms adopting AI agents should account for both direct time savings and indirect costs (increased reviewer attention, potential rework).
• Platform governance and incentives:
  • Repositories / platforms may want to standardize PR templates or require structured metadata to lower reviewer cognitive load and harmonize signals across agents.
  • Incentive schemes (e.g., scoring or badges for agent-generated PRs that meet description-quality standards) could align agent developer incentives with social welfare on code-hosting platforms.
• Market competition and product design:
  • Merge-rate-adjusted performance metrics could become a competitive differentiator among AI-assistant providers; pricing/licensing might reflect not just raw code capability but integration into human workflows.
• Policy and regulation:
  • When assessing AI contributions (for attribution, licensing, or compensation), regulators and firms should consider reviewer costs and acceptance rates, not only generation counts.
• Recommendations for economic evaluation and product strategy:
  • Use merge-rate and reviewer-time-adjusted productivity measures when estimating ROI from AI coding tools.
  • Encourage agent developers to optimize for structured explanations and PR compliance as part of product-market fit.
  • Platforms should collect standardized signals (e.g., time-to-completion, comments per PR) to inform adopter decision-making and to monitor systemic reviewer burden.

Suggested next steps for researchers and practitioners: - Incorporate task and repository controls to identify causal effects of PR presentation on acceptance. - Estimate monetary value of reduced reviewer time from better-structured PRs to inform cost–benefit analyses. - Explore incentive mechanisms on platforms to reward agents that lower overall review costs.