3100 Opinions on Code Review in an AI World: Building Causal Theory from Practitioner Discourse

Coding agents now author entire pull requests, and practitioners sharply disagree about what this does to code review: whether it becomes the bottleneck, whether human review is still necessary, and whether it quietly erodes the understanding that it once built. Repository-mining studies measure surface trends but seldom explain the mechanisms beneath them, and the trends themselves prove unstable. A motivating observational analysis of public GITHUB activity finds that agent-authored pull requests are reviewed less often, merged several times faster, and discussed less than human-authored ones, yet the direction of these trends flips under different but equally defensible analysis choices, so the traces establish what is changing without explaining why. To recover the mechanisms, we synthesize practitioner discourse at scale into an explanatory theory: we collect 38,709 grey-literature documents (engineering blogs and Reddit threads), filter to those substantively about code review, and code a stratified random sample of 3,100 with an LLM-assisted pipeline, from which we build a causal model of 26 constructs and 67 relationships (64 directed, 3 contested). Its organizing claim is that review is the control point through which a coding agent's effect on software is decided, and that AI does not fix the sign of that effect: the team sets it, through the expertise its humans bring and how it structures the review process. The theory makes the competing positions explicit and turns "AI is changing code review" into falsifiable propositions with named constructs and moderators. As a secondary contribution, we offer the underlying LLM-assisted, grey-literature theory-building method as a scalable template for software-engineering research, with a public implementation.

Summary

Main Finding

The authors synthesize practitioner discourse at scale into an explanatory causal theory of how coding agents (AI tools that author code) reshape code review. Their core claim: code review is the control point that determines whether AI improves or harms software outcomes — the sign of AI’s effect is set by team-level choices (human expertise, how review is structured), not by the technology alone. They produce a causal model of 26 constructs and 67 relationships (64 directed, 3 contested) and demonstrate a scalable, LLM-assisted grey-literature method for building such theory. A motivating GitHub analysis shows surface trends (agent-authored PRs often merged faster and receive less discussion), but these trends are fragile to defensible operational choices, motivating the need for causal mechanisms rather than surface metrics.

Key Points

Motivating observation: agent-authored pull requests (PRs) on GitHub tend to be merged faster, discussed less, and receive fewer independent reviewers, but these directions can reverse under different reasonable operationalizations — so surface trends are ambiguous.
Theory claim: review acts as the mediating/control mechanism that translates agent output into software outcomes. Whether effects are positive (productivity, quality) or negative (comprehension debt, lower quality) depends on moderators like reviewer skill, review thoroughness, incentives, and team practices.
Practitioner evidence: assembled from 38,709 grey-literature documents (7,630 web articles and 31,079 Reddit threads), filtered and coded to create a grounded theory reflecting diverse practitioner views (including optimistic and critical perspectives).
Resulting model: 26 named constructs (e.g., review load, review depth/effectiveness, code quality, comprehension debt, reviewer skill, agent quality) and 67 hypothesized relationships, including feedback loops that can produce virtuous or vicious cycles.
Methodological contribution: an LLM-assisted grounded-theory pipeline (Thematic-LM for open coding; Gemini 2.5 Flash as judge/coder) that scales qualitative theory building to thousands of practitioner documents; codebook of ~4,838 codes grounded in ~110k quotes; final theory built by human authors with LLM tools for retrieval and organization.
Limitations acknowledged: grey literature can contain biased, vendor-driven, or LLM-generated content; LLM coding can err; axial/selective coding decisions are interpretive and contingent; findings are a proposed, falsifiable theory to be empirically tested, not a validated causal fact set.
Practical outcome: the theory turns contested claims about “AI changing code review” into explicit, testable propositions (named constructs and moderators) to guide future empirical work.

Data & Methods

Corpus construction
- Two registers: Reddit developer discussion (public Arctic archive) and edited long-form web writing (engineering blogs, vendor writeups, tech press).
- Reddit: scanned 43 subreddits across seven strata (including r/codereview in full, language communities, AI-coding tool forums, broader AI forums), 2020–2026; inclusion by presence of review-related words in title/body → 31,079 threads.
- Web: semantic retrieval with 68 paired queries across multiple source passes (open web, Feedspot blogs, targeted institutional domains) → 7,630 articles.
- Full corpus: 38,709 documents spanning 2020–2026.
Relevance filtering
- Two-stage filters: Reddit keyword gate (keeps threads with code+review terms outside r/codereview), then an LLM judge (Gemini 2.5 Flash, temp=0) applying a reasoning-first rubric to 23,631 candidates; retained 13,469 (57%); validated judge agreement (κ ≈ 0.75 vs. stronger model).
- Coding restricted to 2025–2026 subset to focus on agent-related discussion (~9,100 docs).
Sampling & coding
- Stratified random sample: 3,100 documents (1,263 web articles, 1,837 Reddit threads).
- Open coding with Thematic-LM: documents segmented (web) or kept whole (threads); three independent LLM coder agents per segment producing 1–3 short codes anchored to verbatim quotes. Three coder “lenses” used: neutral, critical (erosion-focused), appreciative (strengthening-focused).
- Aggregation: near-duplicate codes merged; a reviewer maintained a versioned adaptive codebook. Final codebook: 4,838 codes grounded in 109,951 quotes from 2,669 sampled docs.
From codebook to theory
- Authors performed axial and selective coding manually, using LLMs for themed proposals and retrieval of supporting/counter quotes; iterative human-led refinement produced the final causal model (26 constructs, 67 relationships).
Reproducibility
- Collection frames, inclusion rules, judge rubric, and a manifest tracing each quote to its source are versioned; the LLM-assisted pipeline and replication package are publicly released.
Limitations
- Practitioner discourse is not verified behavior; possible LLM-generated content in corpus; LLM coding and human interpretation introduce errors and subjectivity; overrepresentation of vocal adopters and vendor perspectives.

Implications for AI Economics

This paper has multiple implications for researchers, policymakers, and firms concerned with the economics of AI in software production.

Labor reallocation and task shifting
- Coding agents do not simply substitute for developers; they reallocate work from code authoring to review, integration, and governance. Whether this raises or lowers labor demand depends on how teams respond (e.g., hiring more reviewers, retraining engineers, or automating review).
- Important economic margin: tasks where humans have comparative advantage (review depth, architectural judgment, comprehension) may see higher returns; routine patch-writing may be automated.
Bottlenecks and productivity measurement
- If review becomes the delivery bottleneck, aggregate productivity gains from faster code generation can be limited or reversed. Economists should measure not just code output but throughput constrained by review capacity.
- Productivity metrics based on lines/PRs or merge counts will be misleading without accounting for review quality, independent oversight, and comprehension effects.
Human capital, returns to expertise, and complementarity
- The model predicts increased returns to reviewer skill and team practices that preserve deep understanding. This implies complementarity between AI tools and skilled reviewers: firms investing in reviewer expertise can capture more value from agents.
- Conversely, low-investment firms that neglect review may experience negative productivity/quality externalities.
Externalities and distribution of costs
- Open-source and multi-repo ecosystems face externalities: proliferation of low-quality AI-generated contributions can shift review costs to maintainers, potentially reducing welfare and increasing volunteer burden.
- Platforms and firms may have incentives to flood projects with agent outputs (e.g., to train models, to capture market share), producing negative externalities that call for governance or pricing mechanisms.
Incentives, governance, and market structure
- Teams set the sign of AI’s effect via incentives (rewards for thorough review vs. speed), contractual clauses (who is liable for agent-authored code), and tooling (automated reviewers, policy enforcement). These institutional choices shape labor demand and bargaining.
- New markets may emerge for third-party review services, certification of agent outputs, or specialized auditing roles; platform providers could capture rents by providing built-in review infrastructure.
Policy and regulation
- Regulators and platform owners should consider interventions to internalize externalities (e.g., requiring provenance/labels for agent-authored code, rate-limiting automated contributions, or supporting maintainers).
- Disclosure, liability rules, and incentives for high-quality review can influence equilibrium outcomes and distribution of surplus.
Measurement guidance for economists
- The paper shows surface metrics are fragile: small, defensible operational choices can flip conclusions. Economists must:
  - Distinguish machine vs. human comments and separate CI/bot noise.
  - Track reviewer independence (is the agent-invoking developer the sole reviewer?).
  - Measure mediators (review load, depth/effectiveness, reviewer skill, comprehension debt) not only outcomes (merge time, bug counts).
  - Account for moderators (team practices, incentives, agent quality).
- Proposed measurable variables (drawn from the theory): proportion of PRs with no human review; time-to-merge; number and independence of reviewers; comments per line changed; review depth/effectiveness proxies; downstream defect/fault rates; indicators of comprehension debt (e.g., regression in ownership metrics, onboarding times); reviewer workload and turnover.
Empirical strategies suggested
- Use causal designs that leverage mediators/moderators (e.g., mediation analysis) rather than only reduced-form correlations.
- Exploit natural experiments or staggered rollouts of agent tools across teams/repos for difference-in-differences; instrument adoption using exogenous variation (tool availability, licensing changes).
- Run RCTs within firms (e.g., assignment to review-support tooling, incentives for deeper review) to measure causal effects on quality and throughput.
- Combine repository mining with surveys/interviews to observe organizational practices that mediate outcomes.
Long-run distributional and market implications
- If teams can capture productivity gains by investing in review skill and governance, inequality across firms may widen (leaders with good governance scale better). If review capacity is constrained, overall output growth could be muted.
- Markets for developer labor may bifurcate: greater premium on review/architectural skills, downward pressure on routine coding wages.

Concluding note for economists: this paper provides a practitioner-grounded causal vocabulary and specifies measurable constructs and moderators you can use to build economically-testable models. Its methodological warning — that surface repo metrics are sensitive to operationalization — is critical: build your empirical designs around the mediating role of review, not only around agent adoption status or raw outputs.

Assessment

Paper Typedescriptive Evidence Strengthmedium — The paper combines a large-scale corpus (38,709 grey-literature documents) with a stratified, LLM-assisted coded sample of 3,100 documents and descriptive analysis of public GitHub activity, giving substantial empirical grounding for its mechanism hypotheses; however, the GitHub trends are unstable under plausible analysis choices and the causal model is derived from practitioner reports rather than causal inference using exogenous variation, limiting strength for definitive causal claims. Methods Rigormedium — The authors use transparent, scalable methods (large corpus, stratified random sampling, manual coding supported by LLMs, and public implementation) and produce a structured causal model (26 constructs, 67 relationships), which indicates careful methodology; but potential threats remain—selection bias in grey literature, reliance on practitioner self-report, possible LLM-assisted coding errors or coder subjectivity, and lack of robustness/identification in the observational GitHub analysis. SampleTwo-part empirical base: (1) observational analysis of public GitHub activity comparing agent-authored vs human-authored pull requests (measures: review frequency, merge latency, discussion volume); (2) a corpus of 38,709 grey-literature documents (engineering blogs, Reddit threads) filtered for substantive discussion of code review, from which a stratified random sample of 3,100 documents was coded using an LLM-assisted pipeline to build a causal model of 26 constructs and 67 relationships. Themeshuman_ai_collab org_design productivity adoption IdentificationNo formal causal identification from the observational repository analysis (trends are described but shown to be sensitive to analytic choices); causal claims are generated by synthesizing practitioner discourse into a causal model—i.e., inference by large-scale qualitative coding and theory-building (LLM-assisted coding of a stratified random sample) rather than experimental or quasi-experimental identification. GeneralizabilityPublic GitHub repositories only — may not reflect private/enterprise codebases or internal review processes, Grey literature (blogs, Reddit) is self-selected, informal, and likely biased toward vocal practitioners and English-speaking contexts, Findings reflect an adoption-era snapshot; practices and agent capabilities evolve rapidly, Focus on PR-based review workflows — results may not generalize to non-PR development models or different development cultures, Mechanisms inferred from discourse may not map directly onto measurable productivity or labor-market outcomes

Claims (12)

Claim	Direction	Outcome	Confidence & Evidence	Details
Agent-authored pull requests are reviewed less often than human-authored ones. Task Allocation	negative	review frequency	Reading fidelity high Study strength medium	reviewed less often 0.18
Agent-authored pull requests are merged several times faster than human-authored ones. Task Completion Time	positive	time to merge (merge latency)	Reading fidelity high Study strength medium	several times faster 0.18
Agent-authored pull requests are discussed less than human-authored ones. Team Performance	negative	discussion volume in pull request threads	Reading fidelity high Study strength medium	discussed less 0.18
The direction of these observed trends (review frequency, merge speed, discussion) flips under different but equally defensible analysis choices. Organizational Efficiency	mixed	direction/stability of observational trends	Reading fidelity high Study strength medium	direction flips under different analysis choices 0.18
Practitioners sharply disagree about how coding agents change code review: whether review becomes the bottleneck, whether human review remains necessary, and whether agents erode the understanding that review once built. Worker Satisfaction	mixed	practitioner opinions about code review effects	Reading fidelity high Study strength medium	n=38709 sharp disagreement across practitioner discourse 0.18
Repository-mining studies measure surface trends but seldom explain the mechanisms beneath them, and the trends themselves prove unstable. Research Productivity	negative	explanatory power and stability of repository-mining findings	Reading fidelity high Study strength low	seldom explain mechanisms; trends unstable 0.09
We collected 38,709 grey-literature documents (engineering blogs and Reddit threads) and filtered to those substantively about code review. Other	null_result	grey-literature corpus size	Reading fidelity high Study strength high	n=38709 38,709 documents collected 0.3
The authors filtered that corpus and coded a stratified random sample of 3,100 documents with an LLM-assisted pipeline. Training Effectiveness	null_result	coded sample size using LLM-assisted pipeline	Reading fidelity high Study strength high	n=3100 3,100 documents coded 0.3
From that coded sample the authors built a causal model of 26 constructs and 67 relationships (64 directed, 3 contested). Research Productivity	null_result	causal model complexity (constructs and relationships)	Reading fidelity high Study strength high	n=3100 26 constructs and 67 relationships (64 directed, 3 contested) 0.3
The organizing claim of the theory is that review is the control point through which a coding agent's effect on software is decided, and that AI does not fix the sign of that effect: the team sets it, through the expertise its humans bring and how it structures the review process. Organizational Efficiency	mixed	net effect of coding agents on software (mediated by review process and team expertise)	Reading fidelity high Study strength medium	n=3100 AI does not fix the sign of effect; team/review determine direction 0.18
The theory makes competing positions explicit and turns 'AI is changing code review' into falsifiable propositions with named constructs and moderators. Research Productivity	positive	theoretical clarity and falsifiability	Reading fidelity high Study strength medium	n=3100 creates falsifiable propositions with named constructs and moderators 0.18
As a secondary contribution, the authors offer the underlying LLM-assisted, grey-literature theory-building method as a scalable template for software-engineering research, with a public implementation. Training Effectiveness	positive	method availability and scalability	Reading fidelity high Study strength high	n=3100 LLM-assisted grey-literature theory-building method provided with public implementation 0.3

AI coding agents change the dynamics of code review but do not unilaterally improve outcomes; whether agent-written code helps or harms depends on how teams structure and staff review, according to a large-scale analysis of GitHub activity and 3,100 coded practitioner discussions.