Scientific programmers — especially students and junior researchers — are adopting general-purpose conversational code tools like ChatGPT, and report higher perceived productivity when they accept larger blocks of AI-generated code; limited use of formal development practices and less experience are linked to greater perceived gains, suggesting productivity may be measured more by code generation than by validation.

Main Finding

Scientists who program report substantial adoption of generative AI (genAI) for research coding, with highest uptake among students and less experienced programmers. Users overwhelmingly favor general-purpose conversational interfaces (e.g., ChatGPT) over developer-specific tools (e.g., GitHub Copilot). Self-reported productivity gains are larger for less experienced programmers and for those who accept larger blocks of generated code at once; limited use of formal development practices (testing, code review, version control) is also associated with higher perceived productivity, although these practices appear to moderate the relationship between inexperience and perceived gains. The strongest single predictor of perceived productivity is how many lines of generated code a user typically accepts at once, suggesting productivity is often judged by generation throughput rather than by validation.

Key Points

• Sample: 868 survey respondents who program as part of scientific research (from 1,272 collected; exclusions for non-completion, screening).
• Adoption patterns:
  • Highest use among students and less experienced programmers.
  • Variation across research fields (details in full paper).
  • Preference for general-purpose conversational tools (ChatGPT-style) over developer-specific assistants.
• Perceived productivity:
  • Less experienced programmers report larger perceived gains.
  • Users who accept more lines of generated code in a single acceptance report higher perceived productivity.
  • Lower adoption of formal dev practices (testing, review, version control) correlates with higher perceived productivity; but interaction effects indicate that formal practices can partially offset inexperience.
• Risks and context:
  • Prior experimental work (cited) shows conversational assistants can impair conceptual understanding and debugging without consistent efficiency gains.
  • Over-reliance and “automation bias” are concerns, especially given historically inconsistent code validation practices in research settings.
• Non-adoption reasons (qualitative): perceived lack of need, distrust of outputs, concerns about code correctness or reproducibility, tooling or access barriers.

Data & Methods

• Design: Cross-sectional online survey (Qualtrics), reviewed by IRB; targeted scientists who code as part of research.
• Recruitment: Mailing lists and communities (US-RSE, pyOpenSci, Sloan Foundation newsletter, internal university lists, etc.); survey link shareable (non-probability, convenience sample).
• Sample processing: 1,272 responses collected (July 10–Aug 25, 2025); 868 met inclusion criteria after removing incomplete responses and screening.
• Instrument highlights:
  • Demographics and programming background (years experience, languages, frequency).
  • Software development practice questions (version control, testing, code review, CI, publishing/reuse).
  • GenAI usage questions: tools tried, primary tool, access mode, how many lines of generated code typically accepted at once.
  • Perceived productivity: adapted SPACE questionnaire (9 items, 5-point Likert), adjusted to general genAI tools and augmented for activity and code-understanding use cases.
  • Open-ended item on reasons for non-adoption; qualitative coding with a two-stage codebook and consensus annotation.
• Analysis:
  • Quantitative processing and analysis in R; log-transform of experience variables; ordered factors for Likert/frequency items.
  • Correlations (Pearson and polychoric where appropriate), linear models for predictors of perceived productivity.
  • Qualitative thematic analysis of 210 non-adopter free-text responses with axial codebook.
• Limitations noted by authors:
  • Cross-sectional and self-reported—no direct measurement of code quality or actual productivity.
  • Non-representative sample (convenience recruitment), potential selection bias toward those engaged in RSE communities.
  • Some survey items optional, leading to item-specific missingness.

Implications for AI Economics

• Labor supply and task allocation
  • Skill-biased effects: genAI appears to deliver larger perceived gains for less experienced programmers, which could change the effective productivity gradient by experience. This may:
    • Lower the marginal productivity premium for junior programming labor or reallocate junior effort toward higher-level tasks.
    • Alter training incentives and demand for formal software engineering skills among scientists.
  • Task composition: Increased use of code-generation tools may shift scientists’ time from coding to evaluation, experimentation design, or higher-level analysis—if verification is actually performed.
• Complementarity vs substitution
  • GenAI may substitute for some routine coding tasks but be complementary to developers who implement robust validation and software engineering practices. The survey’s interaction result (formal practices partially compensate for inexperience) implies complementarities between genAI and institutionalized quality practices.
• Measurement and productivity accounting
  • Economists should be cautious using self-reported or throughput-based metrics (e.g., lines of code produced) as proxies for productivity: respondents appear to equate accepted generated code with productivity even though validation and correctness may lag.
  • Objective productivity studies should track both generation throughput and downstream verification/bug/failure rates, reproducibility outcomes, and rework costs.
• Quality externalities and social costs
  • If acceptance of generated code increases without corresponding verification, negative externalities may arise (erroneous scientific findings, retractions, reputational damage) with welfare implications for the broader research ecosystem.
  • Estimation of these externalities is important to evaluate net social gains from genAI adoption in science.
• Market structure & product design
  • Strong user preference for general-purpose conversational tools signals product-market demand: purchasers and platform makers may lean toward conversational UX and generalist models rather than developer-oriented integrations—affecting competition, pricing, and feature investment.
  • Trust (explainability, provenance, testing support) will be a key differentiator for tools aiming to penetrate more experienced or safety-critical scientific use cases.
• Policy and institutional responses
  • Funding or policy incentives that promote adoption of development practices (testing infrastructure, code review norms, reproducible workflows) could change the realized productivity/quality tradeoff of genAI adoption.
  • Research institutions may need to invest in training, code auditing, and monitoring to internalize quality externalities.
• Suggested empirical next steps for economists
  • Use quasi-experimental or experimental designs to estimate causal effects of genAI tool adoption on researcher productivity, output quality, and labor supply (e.g., randomized access, staged rollouts).
  • Link tool usage to objective downstream outcomes (publication corrections/retractions, reproducibility checks, time-to-result, code maintenance burden).
  • Study wage and hiring dynamics in academic and research labs to detect shifts in demand for junior vs senior programmer skills.
  • Model market dynamics for genAI tools in research: pricing, bundling with verification/testing features, and competition between generalist and developer-specialized offerings.
  • Quantify externalities by assessing incidence and costs of faulty research attributable to automated code generation.
• Practical takeaways for policymakers and institutions
  • Encourage and possibly subsidize the adoption of basic software engineering practices in research settings to mitigate quality risks.
  • Promote transparency and logging of genAI-assisted code generation in research workflows to enable auditing and replication.

Concluding note: this survey provides timely descriptive evidence that genAI is already reshaping scientific programming practices and perceptions, particularly among less experienced researchers. For AI economists, the central research tasks are to move from self-reported perceived productivity to causal, outcome-based measurement (productivity, quality, and labor market effects) and to account for institutional complementarities and externalities that determine the net social value of genAI in science.