AI coding assistants can raise short-term output for novices who fully delegate code, but that comes at a measurable cost to learning: novices using AI show weaker conceptual understanding, reading, and debugging skills, while only cognitively engaged interaction patterns preserve skill formation.

Main Finding

Using AI to complete unfamiliar coding tasks reduced participants’ acquisition of library-specific skills (conceptual understanding, code reading, debugging) by roughly 17% (≈ two grade points; Cohen’s d = 0.738, p = 0.010). AI assistance did not produce a statistically significant average speedup in task completion time. Some users who fully delegated coding to the AI were faster, but this came at the cost of learning the library.

Key Points

• Experiment: randomized between-subjects study with n = 52 (26 treatment with AI, 26 control without AI). Participants were experienced Python users who had never used the Trio async library.
• Primary outcome: a 14-question, 27-point quiz focused on conceptual, code-reading, and debugging skills for the Trio library. Participants were not allowed AI on the quiz.
• Main quantitative result: treatment (AI) group scored 17% lower on the library-specific evaluation; no significant overall reduction in completion time.
• Qualitative findings: many AI users spent substantial time interacting with the assistant (some >30% of available time; up to 15 questions). This interaction time partly explains the lack of average speed gains.
• Interaction patterns: authors coded six common AI-usage patterns:
  • AI Delegation (full delegation)
  • Conceptual Inquiry (ask conceptual questions / explanations)
  • Iterative AI Debugging (use AI for stepwise debugging)
  • Hybrid Code–Explanation (mix of generated code and explanatory queries)
  • Progressive AI Reliance (increasing reliance over time)
  • Generation-Then-Comprehension (generate code first, then try to understand it) Three of these patterns (those that maintain active cognitive engagement — e.g., Conceptual Inquiry, Iterative AI Debugging, and Hybrid Code–Explanation) preserved learning outcomes even when AI was used.
• Mechanism: control participants benefited from encountering and resolving errors themselves; heavy delegation short-circuited this experiential learning process.
• Safety/overreliance concern: because LLM-generated code can be incorrect or hallucinated, weaker human code-understanding undermines the ability to supervise and catch AI errors.

Data & Methods

• Design: randomized controlled trial; between-subjects assignment to AI-access vs. no-AI.
• Task domain: two short coding tasks (10–20 min each in pilots; max 35 minutes in main study) teaching/use-testing the Trio async Python library (structured concurrency, nurseries, error handling, channels).
• Participants: recruited with criteria: >1 year Python experience, weekly Python use, some prior AI coding exposure, no prior Trio experience.
• Treatment: chat-based AI coding assistant (GPT-4o) integrated with the coding environment; assistant could access current code and generate complete solutions when prompted.
• Measures:
  • Primary: quiz score (14 questions, covering debugging, code reading, conceptual understanding; pre-registered rubric).
  • Secondary: task completion time, keystroke logs, screen recordings, AI-chat transcripts.
• Analysis: quantitative comparison of outcomes between groups; qualitative/coding analysis of screen recordings and chat transcripts to identify interaction patterns.
• Pilots: revealed non-compliance risks (participants using AI in no-AI arms), prompting study design safeguards (screen recordings, stricter instructions).

Implications for AI Economics

• Human capital formation: AI assistance can raise short-run task performance for some users but may reduce the accumulation of domain-specific skills that are necessary for supervising automated systems. This suggests a tradeoff between immediate productivity and durable skill acquisition.
• Labor-market dynamics: if novices increasingly rely on AI and fail to develop supervisory skills, firms may face a future shortage of skilled supervisors/experienced workers who can detect AI errors — potentially increasing demand (and wages) for those who do develop such skills or for more experienced staff.
• Wage and productivity implications: productivity metrics that measure only output (e.g., lines of code, tasks completed) may overstate sustainable gains if they ignore declines in worker competence. Short-term gains from delegation could increase reliance on external AI, producing fragile productivity that collapses when AI is unavailable or inapplicable.
• On-the-job training and onboarding: firms should design AI-integrated workflows and training that preserve active engagement (e.g., require explanations, prompt debugging steps, scaffolded use) rather than pure delegation. Hiring/training investments may need to shift to emphasize human oversight skills.
• Tool and policy design: AI tools that encourage conceptual questioning, require justification of generated outputs, or guide users through debugging are more likely to preserve learning. Policymakers and firms in safety-critical sectors should be cautious about unfettered deployment of AI assistants without mechanisms that sustain skill formation.
• Research and evaluation metrics: economists and organizations should measure not just immediate output gains but also impacts on human capital accumulation and longer-term retention of skills. Longitudinal and cross-domain studies are needed to estimate aggregate effects on labor supply, wage distributions, and task allocation.

Caveats and open questions - External validity: results come from a short-term lab-style experiment in software engineering (one specific library); impacts may differ across domains and over longer horizons. - Sample size and population: modest n and participants were self-selected, experienced Python users; novices or other occupations might show different patterns. - Long-run effects: the study measures immediate post-task knowledge; it does not measure long-term retention or learning trajectories with repeated AI use. - AI model and interface: findings may vary with different model capabilities, UI designs, or organizational incentives.

Suggested next steps for researchers and firms - Run longitudinal field experiments measuring skill retention and on-the-job performance over months. - Test AI interfaces that nudge active learning (explain-then-generate, forced-debug steps, graded hints). - Evaluate industry outcomes: impacts on training costs, promotion paths, and demand for supervisory roles.