Generative LLMs speed programmers' idea generation but appear to reduce moments of human creativity; they still produce code that is at least as idea-rich and more often correct.

Main Finding

Using an LLM-based coding assistant (GitHub Copilot) speeds up implementation and produces more functionally correct code, but shortens idea-generation time and reduces the number of observed creative “aha” moments during programming. Despite the process-level change, the final solutions contain roughly the same number of distinct ideas as unassisted solutions. How the LLM is used (collaboration mode) strongly mediates its effect on creativity.

Key Points

• Study design: within-subject, counterbalanced lab study with N=20 programmers; each completed four Python tasks under two conditions (LLM-assisted vs. unassisted).
• Main quantitative results:
  • Idea-generation periods were significantly shorter with LLM assistance (p = 0.0004).
  • Participants experienced significantly fewer creative moments in LLM-assisted sessions (p = 0.002).
  • LLM-assisted code was functionally and syntactically better overall, but contained roughly the same number of unique ideas as participant-written solutions.
• Four empirically observed human–LLM collaboration modes:
  • Brainstormer: LLM used to generate ideas.
  • Implementer: LLM used to implement a participant’s idea.
  • Verifier: LLM used to check/refine solutions.
  • Co-pilot: active back-and-forth assistance where LLM shares thinking agency.
• Creative outcomes depend on mode: modes that preserve user ideation agency (implementer, verifier) produced more creative moments and diversity than modes where the user ceded thinking to the LLM (co-pilot/brainstormer).
• Behavioral observation: participants often followed “the path of least resistance” — delegating ideation to LLMs rather than using freed time to explore alternatives.
• Design note from pilots: aggressive autocompletion (Copilot) can bias or shortcut the user’s creative process and was controlled in the experiment.

Data & Methods

• Participants: 20 (after excluding 3 for drop/data loss), recruited from undergrad/graduate students with Python and LLM exposure; varied experience levels.
• Tasks: 4 programming tasks per participant (two algorithmic tasks with complexity constraints and two system-design tasks), implemented in Python inside VS Code.
• Conditions:
  • Unassisted: no external resources allowed; only provided IDE.
  • LLM-assisted: GitHub Copilot accessible; no other external resources allowed.
• Procedure:
  • Within-subject, counterbalanced order of conditions and tasks.
  • Think-aloud protocol, screen/audio recording.
  • Pop-up self-report prompt every 5 minutes asking about “aha” moments to triangulate observed creative moments.
  • Post-session retrospective interviews probing strategy, LLM effects, and triggers for creative moments.
• Analysis:
  • Mixed-methods: quantitative comparisons of idea-generation duration, number of creative moments, code quality (functional/syntactic correctness), and number of unique ideas in solutions; qualitative coding of interaction episodes and interviews to identify collaboration modes.
• Limitations noted by authors:
  • Lab setting, limited sample size (N=20), participant pool skewed to students/researchers.
  • Single LLM tool (GitHub Copilot) and single language (Python) — limits generalizability.
  • Task lengths were short; autocompletion effects were controlled but may differ in real-world workflows.

Implications for AI Economics

• Productivity vs. Creative Process trade-off
  • Short-term productivity gains: LLMs raise implementation speed and reduce syntactic/functional errors, implying higher measured output per coder hour for routine implementation tasks.
  • Process externality: reduced ideation time and fewer creative moments imply potential long-run declines in individual creative skill maintenance and exploration efforts, which may lower the rate of novel innovations absent countervailing actions.
• Labor demand and skill composition
  • Reallocation of tasks: demand likely shifts away from routine implementation toward higher-level design, evaluation, and orchestration tasks. Workers who can preserve or augment ideation capability (critical thinking, design framing) capture more value.
  • Human capital depreciation risk: widespread reliance on LLMs for ideation/creative thinking could erode developers’ creative problem-solving skills, lowering long-run human capital in creativity-intensive tasks and affecting wages for those who fail to adapt.
• Product differentiation, market structure, and aggregate innovation
  • Homogenization risk: LLMs’ tendency to propose modal/common patterns can push outputs toward similar solutions across developers and firms, potentially reducing product differentiation and increasing competitive pressures based on speed/cost rather than novelty.
  • Innovation growth: if reduced process-level creativity translates into fewer truly novel solutions, aggregate innovation rates could slow — particularly in domains where creativity drives breakthrough products.
• Returns to complementary investments
  • Complementarity with organization design and interfaces: firms that invest in workflows, tooling, and training that preserve human ideation (e.g., using LLMs as implementers/verifiers, not ghostwriters) will likely realize better long-term innovation returns.
  • Platform design matters: LLM providers and IDE vendors can influence economic outcomes by offering interaction modes that encourage user agency (e.g., delayed/conditional suggestions, scaffolded ideation tools) — affecting productivity, skill retention, and product diversity across the economy.
• Policy and measurement suggestions for economists and firms
  • Track idea diversity and creative-process indicators, not just output quantity—e.g., measures of solution heterogeneity across teams, time spent in ideation, and frequency of novel approaches.
  • Monitor skill trajectories of workers exposed to LLMs longitudinally to detect human-capital depreciation in creative skills.
  • Consider incentives for maintaining creative engagement (training, rotation to ideation tasks, design reviews) to offset cognitive offloading externalities.
• Practical recommendations
  • For firms: integrate LLMs as tools that augment implementation and verification while institutionalizing practices (code review, design workshops) that preserve developer ideation.
  • For platform designers: provide collaboration-mode controls (brainstorm vs. implement vs. verify), adjustable suggestion aggressiveness, and explicit scaffolds that nudge users to explore alternatives before accepting completions.
  • For policymakers: encourage transparency/reporting about how LLMs are used in creative work and support workforce upskilling programs focused on higher-order creative skills.

Overall, the paper shows LLMs improve implementation efficiency but can shorten the creative process unless interaction design and organizational practices deliberately preserve human ideation agency. Those dynamics have measurable implications for productivity, skill demand, innovation, and market outcomes in the AI-augmented economy.