AI coding agents speed up initial development but hollow out understanding: a randomized experiment of 54 CS students shows agent-editing boosts early task completion and time-to-submit yet measurably lowers code comprehension and fails to aid later extension without AI.

Main Finding

Coding agents that directly edit users’ code (vs. chatbots that force users to write/adapt code themselves) raise short-term task completion and speed but substantially reduce programmers’ understanding of the code they produce. Reduced understanding is linked to worse ability to reason about code and offers no reliable advantage when users later must extend code without agent help. Users nonetheless prefer agents for ease and speed. The paper concludes that task-completion metrics alone mischaracterize the value of coding agents because they hide losses in human capital and oversight capacity.

Key Points

• Experimental setup: between-subjects user study (n = 54 CS students) producing a small web app (“zic-zac-zoe”) with one of two AI interfaces:
  • Agent: an editor-style agent (AIDER-like) that proposes diffs and edits to the user’s code.
  • Chatbot: a chatbot that provides high-level syntax/generic code snippets; users must write/adapt code themselves.
• Main quantitative outcomes:
  • Initial task accuracy and time-to-submission: agent users were faster and substantially more accurate initially (p ≪ 0.01; Cohen’s d ≈ 1.2–1.4).
  • Comprehension: agent users scored much lower on tailored comprehension tests (recall + reasoning about their own code) (p ≪ 0.01; d ≈ 0.9).
  • Extension task (users must extend their submitted code without agent help): no significant advantage for agent users (p ≈ 0.45); when controlling for initial accuracy, gains disappear.
• Comprehension measures:
  • Tailored, LLM-generated multiple-choice questions (12 total: recall and reasoning across HTML/CSS/JS).
  • A post-hoc extension task with a separate rubric (adding a win condition, styling symbols, adding a reset).
• Background skill effects:
  • Background ability (measured by a 12-item HTML/CSS/JS quiz) predicts comprehension across both groups.
  • For the initial task, background skill predicts performance in the chatbot group but is largely masked by the agent (agents equalize initial task performance across skill levels).
• Interaction patterns explain part of the comprehension gap:
  • Low-effort behaviors (copy+paste prompts, auto-accepting edits) correlate with lower comprehension.
  • Simpler, more readable edits by agents correlate with higher user comprehension.
• Preferences: despite weaker comprehension, users prefer the agent interfaces because they save time and effort.
• Deliverables: authors release a dataset (54 user websites, prompts, AI traces, background ability, understanding scores).

Data & Methods

• Participants: 59 recruited CS students; after attention checks, 54 included (17 B.S., 31 M.S., 11 Ph.D.). Reported mean 5.4 years coding experience and 16.2 months coding with AI tools.
• Task design:
  • Initial website task (50 minutes): implement a 5×5-board web game with specified structural, logic, and styling requirements (15-rule rubric).
  • Extension task (20 minutes): extend submitted code to add features (12-rule rubric); performed with the chatbot (so all had active coding required).
• AI implementations:
  • Agent group: AIDER-style agent that edits users’ files and produces execution summaries.
  • Chatbot group: LLM restricted to giving short, generic snippets and syntax guidance; no direct access to user files.
  • Underlying models: prompts and tooling used GPT-4.1 for the interactive agents; GPT-5.2 used to generate tailored comprehension questions; Gemini-3.1 Pro used to auto-score rubrics (validated by human scoring).
• Metrics and scoring:
  • Initial and extension task accuracy: proportion of rubric rules met (automated rubric adjudication, validated with human raters; Cohen’s κ ≈ 0.96).
  • Comprehension: 12 tailored multiple-choice items split into recall and reasoning questions; users answered without viewing their full code.
  • Behavioral logging: prompt counts (mean prompts—initial: 5.8 agent vs. 11.6 chatbot), edit acceptance behavior, time-to-submission.
• Analysis:
  • Between-group comparisons (t-tests, effect sizes reported).
  • Linear regressions predicting outcomes from background ability, group, and their interaction. Adjusted R² reported (e.g., initial accuracy model adj R² ≈ 0.37; comprehension model adj R² ≈ 0.17; extension model adj R² ≈ 0.05).
  • Qualitative coding of interaction types to connect behaviors (copy-paste, auto-accept) with comprehension.

Implications for AI Economics

This study has several implications for economic analysis of AI deployment, labor markets, and firm productivity:

1. Productivity measurements that rely on short-term task completion overstate value

   • Standard productivity metrics (tasks completed / time) show clear gains with agents, but they do not capture losses in human capital (understanding). Economic evaluations that use only these metrics will overestimate the long-term productivity benefit and understate hidden costs.

2. Human capital depreciation and retraining costs

   • If agents become widely used, routine displacement of active coding by prompt/review workflows can erode programmers’ tacit knowledge and ability to extend or maintain code without AI. Firms may face higher future maintenance costs, more frequent debugging incidents, and greater retraining needs—economic frictions not visible in immediate productivity gains.

3. Labor market signaling and hiring/credentialing

   • Because agents “equalize” initial task performance across skill levels, standard skill assessments or short coding tests may no longer distinguish worker ability reliably. This can disrupt hiring markets (adverse selection, noisy signals) and shift demand toward new tests that measure understanding or higher-order skills. Employers may increase investment in longer evaluations or supervised probationary periods.

4. Task reallocation, complementarities, and substitution

   • Agents shift the division of labor: they substitute for code-writing but complement higher-level tasks (design, architecture, specification, oversight). However, if human oversight capacity declines with agent use, complementarities weaken. Economic models of automation should incorporate effects on both production and human capital that alter future comparative advantage.

5. Maintenance, externalities, and systemic risk

   • Reduced human understanding increases the probability of undetected errors, security vulnerabilities, and poor maintainability. These negative externalities (e.g., outages, data loss, cascading bugs) generate social and firm-level costs that standard productivity accounting misses.

6. Diffusion and adoption incentives

   • Firms and individual developers will adopt agents for immediate gains, even when anticipating longer-term understanding losses—this is a present-bias/short-horizon adoption problem. Policy or procurement choices (e.g., certification of AI tools that preserve learning, procurement incentives) can alter adoption paths to internalize future costs.

7. Pricing, compensation, and returns to skill

   • If agents compress observable task performance across skill levels, wage differentials tied to routine coding tasks may decline while premiums rise for skills that are complementary to agents (system design, oversight, verification). This suggests a reallocation of returns toward higher-level cognitive tasks.

8. Metrics and evaluation for organizations and policymakers

   • Organizations and regulators should broaden evaluation metrics for coding agents to include measures of human understanding, maintainability, and long-run costs. Procurement and ROI calculations should discount short-run productivity gains by expected future maintenance and retraining costs.

Suggestions for empirical follow-ups and policy-relevant research - Longitudinal field experiments: measure how continued agent use affects skill retention, maintenance costs, and on-the-job learning over months/years. - Firm-level cost–benefit analyses: incorporate expected maintenance, incident, and retraining costs into adoption ROI. - Market signaling studies: test how agent-assisted production affects hiring accuracy and wage dynamics; design assessment protocols robust to agent assistance. - Regulation and certification: explore certification regimes or labelling for agents that quantify “understanding-preservation” features. - Mechanism design: evaluate interfaces and incentive schemes (e.g., mandatory review steps, assessment tasks, pedagogical modes) that maintain human capital while preserving productivity gains.

Practical recommendations for stakeholders - For AI developers and purchasers: incorporate UI features that discourage low-effort prompting and mass auto-accept; optimize agents to produce readable, modular code; provide learning modes (explain-and-quiz) that promote active engagement. - For firms and managers: measure downstream maintenance and extension performance, not just initial delivery speed; require rotation of tasks between humans and agents to maintain skills. - For economists and policymakers: adopt broader welfare accounting when assessing AI tools—include dynamic human capital effects, signaling distortions, and systemic externalities.

Limitations to consider - Sample: university CS students — may not generalize to experienced industry programmers. - Task domain: single web development task; other domains (backend, data pipelines) might show different dynamics. - Short-term experiment: longer-term learning and adaptation effects are not observed. - Simulated agent/chatbot setups and particular model choices may affect external validity.

Overall, the paper highlights an important economic externality of productivity-improving AI: immediate output gains can coexist with erosion of worker understanding and downstream costs. Economic evaluation and policy should widen the objective from raw task completion to include human capital and system-level risks.