Nearly half of AI-generated code fixes are rejected, creating substantial review and validation waste; a qualitative analysis of 306 non-merged PRs identifies 14 failure reasons and recommends better guidance, constraints, and CI-aware validation to curb wasted effort.
Understanding the Rejection of Fixes Generated by Agentic Pull Requests -- Insights from the AIDev Dataset
An analysis of 306 AI-authored non-merged pull requests finds that 46.41% of agent-proposed fixes are rejected, with 14 distinct failure reasons grouped into incorrect implementations, CI/test failures, agent incapacity, and low priority changes.
AI coding agents are increasingly used to generate pull requests (PRs) that propose code fixes in software projects. From a first exploration of the AIDev dataset, we find that 46.41\% of the fixes proposed by the agents Copilot, Devin, Cursor, and Claude are rejected. This represents a significant amount of wasted resources that require human reviews, verifications, and running tests and validations for fixes that are merely discarded. Our goal in this paper is to understand the failure modes of AI-agents, an understanding that is crucial for better integrating AI-agents as efficient teammates. In this paper, we conduct a qualitative study on a representative sample of 306 non-merged pull requests created or co-authored by the agents mentioned earlier, followed by a quantitative analysis of the reasons for rejection. Our qualitative findings identify 14 reasons divided into four high-level categories for rejecting AI-agent fixes. We observe that developers can reject fixes due to fixes whose implementation is incorrect (e.g., incomplete, wrong approach), fixes that do not pass the continuous integration (CI) pipelines and fail tests, fixes for which the agent is unable to perform the implementation (e.g., no code generated, sessions lost), and fixes whose priority is low. Our results shed light on the importance of better guiding the model at these levels: (1) proposing hints about the approach to follow for fixing an issue, (2) outlining constraints or limitations regarding the approaches that should not be taken, and (3) instructing the agent on how to validate the implementation through CI pipelines and without introducing a breaking change. Our results suggest the need for good prioritization of tasks so that generated fixes do not lead to wasted human review efforts or wasted agent resources (e.g., tokens, compute, or allowed number of requests).
Summary
Main Finding
46.41% of AI-agent–generated bug-fix pull requests (PRs) from Copilot, Devin, Cursor and Claude in the AIDev dataset are rejected. The authors qualitatively identify 14 rejection reasons grouped into four high‑level categories (Relevance, Implementation, Technical, Provider). Rejected fixes impose substantial wasted effort and agent resource use: median code churn per rejected PR ranges from 81 to 293 lines (depending on rejection category) and discussion comment medians range 1–4.5.
Key Points
- Scope and headline statistic
- From AIDev (projects with >100 stars), 3,225 fix PRs by the four agents; 1,497 (46.41%) were rejected.
- Study uses a representative random sample of 306 rejected agentic fix PRs (95% confidence, 5% margin).
- Four high‑level rejection categories and 14 sub‑themes (counts out of 306):
- Relevance of the fix (most common)
- Inactivity: 53 (17.3%)
- Superseded: 18 (5.9%)
- Low priority: 3 (1.0%)
- Architecture change: 1 (0.3%)
- Test PR (created for testing): 1 (0.3%)
- Implementation issues
- Incorrect fix (functionally wrong/incomplete): 17 (5.6%)
- Wrong approach (different solution chosen): 8 (2.6%)
- Ambiguity (misunderstood requirements): 2 (0.7%)
- Insufficient (partial fix): 2 (0.7%)
- Wrong repository: 2 (0.7%)
- Provider-related issues
- Agent failure/unreachable/no commits: 23 (7.5%)
- Rate limits/API restrictions: 3 (1.0%)
- Technical-related issues
- CI failures / failing tests: 21 (6.9%)
- Breaking changes introduced: 1 (0.3%)
- Unclear / others: 151 (49.3%) had no clear reason labelled from discussion
- Relevance of the fix (most common)
- Quantitative waste indicators
- Code churn medians by category: Technical 293, Relevance 131, Implementation 103, Others 81, Provider 0.
- Median # of comments by category: Technical 4.5, Implementation 3, Relevance 3, Others 2, Provider 1.
- Labeling process and reliability
- Two authors independently labeled PRs; Cohen’s kappa = 0.605 (substantial agreement). 93 conflicts resolved by a third author.
Data & Methods
- Data source: AIDev dataset (curated PRs across GitHub projects; authors used subset with >100 stars).
- Inclusion criteria for PRs:
- Closed and not merged (rejected),
- Created or co‑authored by Copilot, Devin, Cursor, or Claude,
- Classified as fixes (bug‑fixing PRs). Codex PRs excluded due to labeling ambiguity.
- Sample: From 1,497 rejected fixes, drew representative random sample of 306 PRs (95% confidence, 5% margin).
- Qualitative analysis:
- Two raters read PR threads (comments, CI output, artifacts), assigned initial labels, discussed to form final label set of 14 reasons grouped in 4 themes.
- Inter‑rater reliability: Cohen’s κ = 0.605. Disagreements adjudicated by third author.
- Quantitative analysis:
- Extracted code churn (lines changed), number of commits, and number of discussion comments per PR from AIDev to measure “wasted effort”.
- Reported medians and compared across rejection categories.
- Limitations noted by authors:
- Large share (49.3% of sample) could not be clearly labeled from PR thread context.
- Dataset limited to public GitHub projects and four agents; Codex excluded.
Implications for AI Economics
- Direct resource waste and cost channels
- Agent compute/tokens and premium query costs are wasted when many generated fixes are rejected (esp. for inactive or low‑priority tasks).
- Human review time and CI runs consumed on rejected PRs create measurable labor and infrastructure cost.
- Provider rate limits (and associated pricing tiers) materially affect agent productivity and rejection likelihood.
- Platform and operational recommendations to improve ROI
- Task prioritization/triage before dispatching to agents: avoid assigning low‑priority or duplicate issues to agents to reduce wasted spend.
- Quota, scheduling and billing policies: platforms/teams should align agent usage quotas to value (e.g., higher priority issues get more agent budget).
- Cost‑aware agent orchestration: incorporate estimated rejection risk into execution decisions (e.g., run lightweight checks first, gate full runs).
- Product and market implications
- Pricing models for agent services should reflect rejection/waste risk (e.g., bill by successful landed fixes or provide volume discounts conditional on acceptance).
- Enterprise buyers may demand CI‑aware agents and stronger SLAs (uptime, request limits, determinism) to reduce provider‑related failures.
- Tools that certify or predict likelihood of PR acceptance (based on issue clarity, tests included, repo conventions) could have economic value.
- Research and measurement agenda for economics of AI agents
- Quantify monetary cost of rejected agent PRs (compute + CI + human review) across settings and agents.
- Design economic incentives (cost‑sharing, quotas, rewards for high‑precision agent runs) to align developer behavior and reduce waste.
- Evaluate ROI of interventions: instruction files, prompt templates, CI‑aware validation, and pre‑execution triage—measure reductions in rejection rate and cost per accepted fix.
- Model market effects of improving acceptance rates (e.g., lower marginal costs per accepted change, higher adoption, different pricing equilibria).
- Short term operational levers
- Require minimal metadata before agent runs (priority, relevant tests, prohibited approaches) to reduce low‑value generation.
- Use staged/cheap validation (static checks, unit-test selection) before full CI runs to avoid expensive wasted pipeline executions.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper provides systematic descriptive evidence based on a moderate-sized, mixed-methods sample (306 non-merged PRs) and quantifies a clear rejection rate (46.41%). This yields useful, actionable insights about failure modes, but it is not causal and may be affected by selection biases in the AIDev dataset, coding subjectivity, and limited coverage of agents/projects.
Methods Rigormedium — The study combines qualitative coding with quantitative counts, which is appropriate for classifying failure modes; however, the rigor depends on details not provided here (e.g., sampling frame, coder training, inter-rater reliability, codebook transparency). The dataset covers four agents and 306 PRs—sufficient for exploratory classification but limited for inference across platforms, languages, or time.
SampleRepresentative sample of 306 non-merged pull requests from the AIDev dataset that were created or co-authored by four AI coding agents (Copilot, Devin, Cursor, Claude); includes metadata on PR outcomes, CI/test results, and textual/code diffs used for qualitative coding and quantitative aggregation (reported overall 46.41% rejection rate).
Themesproductivity human_ai_collab org_design
GeneralizabilityLimited to the AIDev dataset and the four specific agents studied (Copilot, Devin, Cursor, Claude), Focuses on non-merged PRs only—does not analyze merged/accepted PRs or overall agent success rates in broader populations, Potential language, repository, and CI-configuration selection bias (may over- or under-represent certain project types), Time-bound snapshot; agent behavior and tooling evolve rapidly, Qualitative coding may reflect researcher subjectivity and project-specific norms, limiting cross-project comparability
Claims (8)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| AI coding agents are increasingly used to generate pull requests (PRs) that propose code fixes in software projects. Adoption Rate | positive | use/adoption of AI coding agents to generate PRs |
Reading fidelity
high
Study strength
low
|
not reported
|
| From a first exploration of the AIDev dataset, 46.41% of the fixes proposed by the agents Copilot, Devin, Cursor, and Claude are rejected. Output Quality | negative | proportion of proposed fixes that are rejected |
Reading fidelity
high
Study strength
medium
|
46.41% rejected
|
| The high rejection rate represents a significant amount of wasted resources that require human reviews, verifications, and running tests and validations for fixes that are merely discarded. Organizational Efficiency | negative | resource waste from human review, verification, and CI for rejected fixes |
Reading fidelity
medium
Study strength
speculative
|
not reported
|
| We conduct a qualitative study on a representative sample of 306 non-merged pull requests created or co-authored by the agents mentioned earlier, followed by a quantitative analysis of the reasons for rejection. Other | null_result | qualitative and quantitative characterization of non-merged PRs |
Reading fidelity
high
Study strength
high
|
n=306
|
| The qualitative findings identify 14 reasons divided into four high-level categories for rejecting AI-agent fixes. Output Quality | negative | number and categorization of reasons for rejection |
Reading fidelity
high
Study strength
medium
|
n=306
14 reasons in 4 high-level categories
|
| Developers reject fixes for (a) incorrect implementation (e.g., incomplete, wrong approach), (b) fixes that do not pass CI pipelines and fail tests, (c) fixes for which the agent is unable to perform the implementation (e.g., no code generated, sessions lost), and (d) fixes whose priority is low. Output Quality | negative | reasons for rejection of agent-generated fixes (implementation correctness, CI/test failures, agent inability, low priority) |
Reading fidelity
high
Study strength
medium
|
n=306
|
| The results suggest improving model guidance by (1) proposing hints about the approach to follow for fixing an issue, (2) outlining constraints or limitations regarding approaches that should not be taken, and (3) instructing the agent on how to validate the implementation through CI pipelines and without introducing a breaking change. Developer Productivity | positive | effectiveness of proposed guidance interventions for AI agents |
Reading fidelity
high
Study strength
speculative
|
n=306
|
| There is a need for good prioritization of tasks so that generated fixes do not lead to wasted human review efforts or wasted agent resources (e.g., tokens, compute, or allowed number of requests). Task Allocation | positive | task prioritization to reduce wasted human and agent resources |
Reading fidelity
high
Study strength
speculative
|
n=306
|