Developers with AI coding assistants write substantially more — but also delete more — indicating faster iteration rather than straightforwardly higher-quality output; users report productivity gains but few perceived changes in other workflow areas.

Main Finding

Longitudinal telemetry and survey evidence shows that sustained use of IDE-integrated AI coding assistants is associated with higher gross code production but also with more code deletion, greater reliance on external code sources, and increased context switching. Developers report perceived productivity gains, but are less aware of or report fewer changes in code quality, editing, reuse, or switching — suggesting AI reshapes the structure and cognitive demands of work more than it simply reduces effort.

Key Points

• Dataset & participants

  • 151,904,543 anonymized IDE interaction events collected over two years.
  • 800 professional developers in telemetry analysis: 400 AI users and 400 non-users.
  • Complementary survey of 62 practitioners and five follow-up interviews.
  • Most-used AI tools reported: ChatGPT (n=42), GitHub Copilot (n=29), JetBrains AI Assistant (n=23).

• Behavioral findings (telemetry)

  • AI users write and modify substantially more code (higher lines added / edits).
  • AI users also delete significantly more code, indicating more churn or trial-and-error.
  • AI users rely more on external code snippets / copying from web or external sources.
  • AI users exhibit higher rates of context switching / fragmented workflows.

• Perception findings (survey & interviews)

  • Majority of respondents report productivity gains after adopting AI assistance.
  • Respondents generally perceive minimal changes in code quality, editing frequency, reuse habits, and context switching — a mismatch with telemetry evidence.
  • Interviews indicate shifts from generation tasks to verification, integration, and review of AI output.

• Conceptual insight

  • AI does not simply reduce developer effort; it redistributes effort toward more fragmented, reactive, and cognitively demanding activities (e.g., verifying and integrating generated code).
  • Self-reports can understate structural workflow changes; telemetry reveals "silent" restructuring.

Data & Methods

• Mixed-method design combining:
  • Fine-grained, anonymized IDE telemetry (two years; 151.9M events) aggregated monthly across developers.
  • Survey with Likert-scale items on five workflow dimensions: productivity, code quality, code editing, code reuse, context switching.
  • Five semi-structured follow-up interviews for qualitative context.
• Sample
  • Telemetry cohort: 800 developers (balanced AI user vs non-user groups).
  • Survey: 62 AI-using respondents (diverse roles and experience levels).
• Measures & operationalization
  • Productivity: code output metrics (e.g., lines added) from telemetry; self-reported productivity/time coding.
  • Code quality: deletion/rewrite rates and telemetry proxies; self-reported code quality/readability.
  • Code editing & reuse: frequency of edits, use of external snippets, copy/paste events.
  • Context switching: event sequences, session fragmentation measures.
• Analysis
  • Longitudinal comparison between AI users and non-users over two years.
  • Descriptive statistics and inferential comparisons across the five workflow dimensions.
  • Triangulation between behavioral logs and self-reports/interviews.
• Limitations noted by authors
  • Observational (non-experimental) design — causality cannot be definitively established.
  • Potential selection bias in telemetry and survey samples (who opts into AI tools and to be surveyed).
  • Telemetry proxies may not capture all aspects of code quality or cognitive load.

Implications for AI Economics

• Productivity measurement and valuation
  • Common numeric measures (e.g., lines added) may overstate “productive” output because AI-assisted workflows increase churn (more deletions) and shift work toward review and integration. Economists should develop richer productivity metrics that incorporate verification, rework, and net value delivered.
• Labor demand and task reallocation
  • AI shifts effort from generation toward verification, integration, and higher-order judgment. Demand for skills may move toward code review, debugging, security auditing, and system-level reasoning. This implies complementary investments in worker training and potential upskilling needs.
• Wage and task-price effects
  • If AI reduces time spent on boilerplate but increases time on cognitively demanding verification, wages and billing models may adjust to reflect greater emphasis on higher-skill tasks rather than simple output volume. Firms may re-evaluate pricing or productivity targets.
• Fragmentation, attention costs, and productivity paradox
  • Increased context switching and fragmentation produce attention costs that can offset some automation benefits. Macro-level productivity gains from AI adoption may be smaller or slower than naive calculations predict — relevant to forecasting sectoral productivity growth.
• Code reuse, external markets, and ecosystem effects
  • Greater reliance on external snippets and web-sourced code could reduce demand for some forms of knowledge reuse (e.g., fewer StackOverflow posts) but increase spillovers and copyright/compliance risks. Markets for curated, high-quality components and provenance-tracking tools may grow.
• Measurement and policy implications
  • Observed mismatch between self-reported and actual behavioral change cautions economic studies that rely solely on surveys. Policy assessments (e.g., on labor displacement) should use longitudinal behavioral data where possible.
• Firm strategy and tooling investments
  • Firms adopting AI should invest not only in model access but in tooling that reduces verification overhead, supports continuity across fragmented tasks, and tracks provenance — these can affect realized productivity and cost savings.
• Research directions for economists
  • Quantify net value added: integrate telemetry-based measures of churn, verification time, and defect rates to estimate true productivity change.
  • Model labor reallocation: estimate shifts in task bundles and consequent wage/profile changes across developer types.
  • Welfare and externalities: study ecosystem-wide impacts (open-source contributions, knowledge markets, IP risks) driven by AI-mediated code reuse.

Short caveats - The study is observational and focused on professional IDE users tied to JetBrains tooling; generalization to all developers, tools, or firm contexts should be cautious. - Telemetry-based proxies imperfectly capture code quality, cognitive load, or business value; further work should combine richer outcome measures (e.g., defect rates, delivery times).

If you want, I can (a) extract concrete telemetry effect sizes reported in the paper (e.g., percent increases in lines added/deleted, context-switching rates) if you’d like numeric estimates, or (b) draft a short policy brief targeting managers about how to measure AI-driven productivity in engineering teams.