GenAI coding assistants deliver modest productivity gains—about a third of a standard deviation—mostly in controlled experiments, but they do not reliably boost learning or skill acquisition in educational settings.

Main Finding

Generative-AI coding assistants produce a small-to-moderate, statistically significant increase in short-term developer productivity (Hedges’ g = 0.33, 95% CI [0.09, 0.58], p = 0.008), but effects are highly heterogeneous across contexts (I2 = 99%). In contrast, there is no reliable pooled effect on learning outcomes for students (Hedges’ g = 0.14, 95% CI [−0.18, 0.47], p = 0.389; I2 = 86%). Productivity gains are large in controlled lab experiments (g ≈ 0.73) but negligible in open‑source and enterprise settings.

Key Points

• Scope: Pre-registered meta-analysis covering studies published 2019–2025. Aggregate: n = 23 studies, k = 27 effect sizes (productivity: 14 studies, 16 effect sizes; learning: 10 studies, 11 effect sizes).
• Productivity result: pooled g = 0.33 (moderate positive); substantial between-study variance (τ2 = 0.22; I2 = 99%).
• Learning result: pooled g = 0.14 (not statistically different from zero); τ2 = 0.25; I2 = 86%.
• Heterogeneity driver: study setting is the only statistically significant moderator for productivity (QM(2)=10.51, p=0.005). Laboratory studies show large positive effects; enterprise and open-source studies show no significant effect.
• Other moderators (GenAI interface, programming language, participant experience level, randomization, experimental design) were not significant predictors of effect-size variation.
• Time trend: cumulative meta-analysis shows a stable, modest productivity gain over time (despite rapid model improvements).
• Robustness checks:
  • Leave-one-out: pooled g ranged 0.24–0.38; two studies (P002, P004) had disproportionate influence but did not reverse the overall finding.
  • Risk-of-bias (RoB2 / ROBINS‑I): not a significant moderator, though lower-risk studies had smaller point estimates (g ≈ 0.08) than higher-risk ones (g ≈ 0.50).
  • Publication-bias diagnostics: Egger’s test flagged funnel asymmetry (z = 4.52, p < 0.001); trim-and-fill imputed no missing studies, suggesting asymmetry may reflect heterogeneity rather than suppressed nulls.
• Measurement: Productivity proxied by task completion time, commit counts, and lines of code; learning measured by exam performance. Primary tools studied included GitHub Copilot and other LLM-based assistants; most productivity studies used Python or multi-language tasks.

Data & Methods

• Literature search: ACM, arXiv, Scopus, Web of Science; inclusion required quantitative comparison of GenAI-assisted vs. unassisted programming on productivity or learning outcomes.
• Sample sizes: productivity studies included m ≈ 3,535 participants and r ≈ 6,355 repositories; learning studies included m ≈ 1,069 participants.
• Effect-size metric: standardized mean differences computed as Hedges’ g.
• Meta-analytic approach: random-effects models (REML estimator) and mixed-effects meta-regressions for moderator analysis.
• Risk-of-bias assessment: RoB2 for RCTs and ROBINS‑I for non-randomized studies.
• Sensitivity and bias checks: leave-one-out influence analysis, Cook’s d and studentized residuals for influential studies, Egger’s regression for funnel asymmetry, and trim-and-fill method.

Implications for AI Economics

• Productivity and short-run labor effects
  • Firms may realize modest short-term productivity gains from GenAI assistants, but the gains are context-dependent and appear concentrated in controlled/task-limited work. Expect heterogeneous returns to adoption across teams, codebases, and task types.
  • Large measured effects in lab settings likely overstate real-world firm-level productivity impacts. Economic models and cost‑benefit analyses should treat lab-based estimates as upper bounds.
  • Heterogeneity (I2 ≈ 99%) implies uneven returns across workers — potential for reallocation of tasks (toward review, integration, debugging) and changes in skill complementarities rather than uniform labor displacement.
• Human capital and education
  • No consistent evidence that GenAI improves core programming learning outcomes. Widespread classroom adoption could alter skill acquisition processes (cognitive offloading), with potential long-run effects on worker ability and credential signaling.
  • Policymakers and educators should be cautious: credential validity, exam design, and curricula may need revision to ensure skill persistence and to measure true understanding rather than tool-assisted output.
• Wages, task composition, and inequality
  • If GenAI mainly augments productivity for some tasks/workers (e.g., those doing short, well-specified programming tasks), we may see differential wage gains and task reallocation rather than uniform labor-cost reductions.
  • Differential adoption and effectiveness could exacerbate within-occupation inequality (those who integrate tools effectively capture more gains).
• Firm strategy and investments
  • To capture real-world productivity gains, firms likely need complementary investments: integration into workflows, quality assurance, guidelines to control technical debt, and training on effective human-AI collaboration.
  • Potential hidden costs—higher maintenance, security vulnerabilities, and technical debt from AI-generated code—should be incorporated into ROI estimates.
• Research and policy needs
  • Economic models should incorporate contextual heterogeneity (task type, codebase familiarity, collaboration overhead) and longer-term outcomes (maintenance costs, skill formation).
  • Priority empirical work: field experiments and longitudinal studies measuring durable productivity, code quality, maintenance costs, and worker learning trajectories.
  • Education policy: trials of pedagogical interventions (e.g., structured AI use, forced unaided assessments, tool-aware curricula) to preserve foundational skills while leveraging AI benefits.

Short summary: GenAI coding assistants can raise short-term programmer productivity modestly, but effects depend crucially on context and do not reliably translate into improved learning. Economic assessments of AI in software development should account for strong heterogeneity, potential long‑run human-capital effects, and costs associated with code quality and maintenance.