AI-generated PR descriptions that misrepresent code are rare but costly: 1.7% of agentic PRs showed high inconsistency and were accepted far less often and merged 3.5× more slowly, undermining reviewer trust and slowing collaboration.

Main Finding

AI-authored pull request (Agentic-PR) descriptions sometimes misalign with the actual code changes. In a 23,247-PR sample, 406 PRs (1.7%) exhibited high message–code inconsistency (PR-MCI). High-MCI PRs have materially worse review outcomes: 51.7% lower acceptance (28.3% vs. 80.0%) and 3.5× longer time-to-merge (55.8 vs. 16.0 hours). The paper identifies common inconsistency types (dominantly “Phantom Changes”) and shows variation across agents and task types, arguing this undermines trust and motivates automated verification and better PR-generation.

Key Points

• Dataset & artifacts
  • 23,247 closed Agentic-PRs (filtered subset of AIDev-pop).
  • 974 PRs manually annotated (600 for validation + 374 additional), with 432 partial/misaligned cases released.
  • Code and supplementary materials published: https://github.com/gjz78910/PR-MCI.
• Prevalence & heterogeneity
  • 406 high-MCI PRs = 1.7% overall (similarity score < 0.61).
  • Large agent heterogeneity: GitHub Copilot 8.7% high-MCI vs. Devin 0.4% (≈20× difference).
  • Task heterogeneity: chore (4.0%), refactor (3.5%), bug fix (2.1%), feature (1.5%), docs/test (≈1.0%).
• Taxonomy of PR-MCI (8 categories)
  • Most common: Phantom Changes (45.4%) — descriptions claim unimplemented changes.
  • Other frequent types: Scope Understated (22.0%), Placeholder/Incomplete (18.8%).
  • Distribution varies by agent and task (e.g., Copilot dominated by Phantom Changes).
• Outcome associations (statistical evidence)
  • Acceptance: 28.3% for high-MCI PRs vs. 80.0% for low-MCI (Δ = 51.7 percentage points, p < 0.001; small-to-medium effect).
  • Time-to-merge: mean 55.8h (high-MCI) vs. 16.0h (low-MCI), p < 0.001 (Cliff’s δ = 0.310).
  • Results hold after regression controls for code churn, files changed, task type, and agent.
• Practical recommendations
  • For users: verify commits, avoid templated prompts, refine prompts.
  • For reviewers: quick checks (diff non-empty, verbosity vs. churn, template markers).
  • For tool-builders: integrate automated PR-MCI detection into agent pipelines to flag/avoid problematic PR descriptions.
• Limitations noted by authors
  • PR-MCI measured by a heuristic proxy; threshold calibrated on validation set.
  • Observational design — associations, not causal inference.
  • Dataset filtering (license, repo selection) and agent sample-size imbalance may limit generalizability.

Data & Methods

• PR-MCI measurement
  • Heuristic similarity score s ∈ [0,1] combining three signals:
    • scope adequacy ss (0.3 weight): match between description verbosity and code churn,
    • file-type consistency sf (0.4 weight): whether mentioned file types are modified,
    • task-type alignment st (0.3 weight): match between description language and labeled task type.
  • Aggregation weights: s = 0.3·ss + 0.4·sf + 0.3·st. Threshold θ = 0.61 for high-MCI selected via F1 optimization on annotated validation data.
  • Validation: two annotators labeled 600 stratified PRs (κ = 0.892). Heuristic method F1 = 0.63 (precision 0.742, recall 0.548). Alternative embedding-based method performed poorly (F1 = 0.15).
• Taxonomy derivation
  • GPT-5.2 generated initial codebook; two annotators refined it; 432 partial/misaligned PRs were manually categorized into 8 types.
• Statistical analysis
  • Comparisons: chi-square tests (acceptance) with Cramér’s V; Mann–Whitney U tests with Cliff’s δ (time, counts).
  • Regression models (log-transformed code churn, files changed, task type, agent) to control for confounders; PR-MCI remained significantly associated with outcomes (p < 0.001).
• Dataset scope
  • Source: AIDev-pop subset; final filtered set: 23,247 PRs authored by five agents (GitHub Copilot, Cursor, Claude Code, OpenAI Codex, Devin); closed PRs only; permissive licenses (MIT/Apache 2.0); confined to six common task types (~95.2% of PRs).

Implications for AI Economics

• Direct labor cost and productivity effects
  • High-MCI PRs take ~40 additional hours to resolve on average (55.8h vs 16.0h); at scale even a small high-MCI rate creates non-trivial reviewer overhead. For example, 1.7% of PR volume with ~40 extra reviewer-hours each implies substantial hidden labor costs across organizations using agentic PRs at scale.
  • Lower acceptance rates for high-MCI PRs (28.3% vs. 80.0%) mean more rework and manual intervention, reducing the net productivity gains expected from automating PR creation.
• Adoption, trust, and transaction costs
  • Misleading PR descriptions undermine trust in AI agents, increasing monitoring and verification requirements (higher supervision costs), which raises the effective price of deploying agentic workflows.
  • Unreliable agent outputs create policy and governance needs (e.g., enforced verification in CI/CD), which introduces fixed costs to adoption and may segment the market toward agents/tools that provide provable/verifiable description fidelity.
• Market opportunities and productization
  • Demand for PR-MCI detection and verification tools: firms can monetize automated verification services (integrated checks in CI, PR linters, or agent plug-ins). The ROI argument: preventing a single high-MCI PR could save tens of reviewer-hours, so verification can be cost-effective in medium-to-large teams.
  • Reputation and competition among agent providers: measurable differences across agents (20× variation) suggest market differentiation based on description fidelity; agents that reduce PR-MCI can command higher adoption or premium pricing.
  • Services for improving agent reliability (finetuning, RLHF focused on faithful descriptions, prompt engineering toolkits) become valuable; companies may invest in internal teams or procure third-party services.
• Externalities and downstream risk
  • Phantom or misstated changes can increase technical debt, introduce latent bugs, or cause security regressions if descriptions mislead reviewers into accepting incorrect assumptions — leading to downstream remediation costs larger than immediate review overheads.
  • Regulatory/compliance implications in some domains (e.g., safety-critical or highly regulated codebases) where correct documentation of changes is required — higher standards may be enforced for agentic contributions, raising compliance costs.
• Research and measurement needs for economic evaluation
  • Standardized metrics for PR-description fidelity (like PR-MCI) enable tracking of agent improvement over time and economic valuation of reliability gains.
  • Empirical cost–benefit analyses: compare labor savings from automated code generation against added verification/supervision costs under varying PR-MCI rates to guide procurement and integration strategies.
  • Incentive-aligned optimization: agent developers could optimize for “net economic value” (reduction in total human-hours per accepted PR) rather than raw generation metrics.
• Policy and organizational recommendations
  • Organizations should instrument their pipelines to measure PR-MCI-like metrics and include them in vendor evaluations and internal KPIs.
  • Short-term practice: mandate lightweight automatic checks for PR-body/code alignment to limit the economic drag; long-term: favor agent platforms that demonstrate low PR-MCI in standard benchmarks.

Summary: Even with a modest prevalence (1.7%), message–code inconsistency in agent-authored PRs has outsized economic implications via lost reviewer time, reduced acceptance, and trust erosion. These effects create clear market opportunities (verification tools, reliability-focused agent improvements) and justify measuring PR-fidelity as a core economic performance metric for AI coding agents.