Moderate use of GitHub Copilot speeds up developers and eases their workload, but heavy or mixed use blunts the benefit; chat interactions also improve task completion. In everyday work, developers report both higher cognitive load and greater productivity when interacting with AI.

Main Finding

Moderate, targeted use of a generative AI coding assistant (GitHub Copilot) in a real-world firm setting improves developer efficiency (shorter task duration) and reduces perceived workload compared to not using Copilot; chat-based interaction additionally improves task accuracy (completion). Excessive or combined use of both in-code suggestions and chat reduces or erodes these benefits. Subjective workload ratings generally align with behavioral indicators. Multimodal, mixed-method field studies (physiology + telemetry + questionnaires) are feasible and yield complementary insights into developers’ experience with GenAI.

Key Points

• Study focus: developer-centered, multimodal, mixed-methods field study of professional software developers’ interaction with GitHub Copilot in their natural work environment.
• Primary outcomes analyzed: efficiency (task duration), accuracy (task completion/success), and perceived workload (NASA-TLX).
• Interaction types compared: no Copilot, in-code suggestions, chat prompts, and combined in-code + chat use.
• Main behavioral pattern:
  • Moderate use of either in-code suggestions or chat → improved efficiency and lower perceived workload vs. no Copilot.
  • Chat interaction → higher accuracy (better task completion) than in-code-only or no Copilot.
  • Excessive or combined use → diminished gains or neutral/negative effects.
• During the uncontrolled (everyday work) period, participants reported both higher perceived productivity and higher cognitive load when using GenAI.
• Subjective measures (NASA-TLX and self-report) were broadly consistent with objective behaviors recorded in the controlled sessions.

Data & Methods

• Participants: professional software developers at SAP (two US locations); each participant completed a four-day protocol (two supervised controlled sessions framing a three-day uncontrolled work period).
• Study design:
  • A/B design: Group A — no Copilot in first controlled session, Copilot in second; Group B — Copilot in both controlled sessions. This enabled between- and within-subject comparisons.
  • Controlled sessions: pre-questionnaire, n-back cognitive task (1–3 back) with startle event, relaxation videos, three baseline coding tasks, and six randomized main tasks per session spanning six categories (coding, debugging, documentation, unit tests, summaries, brainstorming). NASA-TLX filled after each task.
  • Uncontrolled period: three normal workdays where participants logged tasks, start/end times, perceived cognitive load and productivity, and whether GenAI was used.
• Multimodal measurement:
  • Physiological data: continuous recording via EmbracePlus wristband (for cognitive load / stress proxies and startle-event verification).
  • Behavioral telemetry: screen, mouse, and keyboard recordings during controlled sessions.
  • Self-report questionnaires: pre-questionnaire (job/experience/personality/devex), Copilot evaluation, NASA-TLX after tasks, daily work logs, end-of-day questionnaires, post-questionnaire.
• Task selection and control: main tasks chosen from HumanEval-X; Java language; tasks balanced for cognitive demand using code metrics (cognitive complexity, cyclomatic, Halstead metrics, LOC/NCLOC/CLOC, nested block depth). Task order randomized per participant.
• Copilot interaction: participants could use in-code suggestions, inline chat, and separate chat; free to switch models and modes (ask, edit, agent). Analyses focused on interaction type and intensity effects on efficiency, accuracy, and workload.
• Analysis highlights: controlled-session analyses emphasize behavioral + subjective alignment; uncontrolled period provided ecological validation (self-reported higher productivity and cognitive load with AI use).

Implications for AI Economics

• Conditional productivity gains and diminishing returns:
  • Economic models and ROI estimates for GenAI adoption should account for nonlinearity: moderate, targeted usage yields gains, while excessive or unfocused use can reduce benefits. Subscription/usage pricing and value projections must consider optimal usage intensity rather than linear per-hour gains.
• Interaction-type value differentiation:
  • Chat-style GenAI interactions produce higher accuracy (quality). For tasks where correctness and quality matter (e.g., security-critical code, reviews, compliance), investment in chat-enabled GenAI workflows may yield higher economic value than in-code autocompletions alone.
• Labor allocation and skill complementarity:
  • Findings support a nuanced view of human-AI complementarities: firms should direct GenAI towards tasks and developer profiles that benefit most (e.g., experienced developers using moderate, task-specific AI assistance). For novices, risk of undetected incorrect outputs implies potential costs (rework, defects) that must be incorporated into training and onboarding investments.
• Measuring productivity for cost–benefit analysis:
  • Traditional productivity proxies (lines of code, acceptance rate) are insufficient. Economic assessments should include task completion quality and cognitive/workload metrics, since subjective workload aligns with behavioral data and can predict long-run productivity and error costs.
• Implementation and policy design:
  • Firms should create guidelines and training to encourage effective interaction patterns (moderate, task-appropriate use; prefer chat for accuracy-sensitive tasks) and to avoid overreliance that erodes benefits.
  • Monitoring and governance: multimodal telemetry (IDE logs + self-report) can inform adaptive policies, targeted training, and dynamic licensing strategies (e.g., role- or task-based access tiers).
• Recommendation for evaluation investments:
  • Because mixed-method field studies are feasible and informative, organizations should invest in pilot evaluations inside their workflows to estimate real-world productivity effects before wide rollout. These evaluations reduce uncertainty during the post-hype “trough of disillusionment” and help calibrate expected returns.
• Modeling suggestion:
  • Incorporate an interaction-intensity term and interaction-type term into producer-function style models of developer output (output = f(labor, capital, GenAI_use_intensity, GenAI_mode)), allowing for diminishing marginal returns and quality-adjusted output measures.

Overall, the study suggests that GenAI can produce economically meaningful gains when used selectively and with the right interaction mode, but naive scaling of usage may produce limited or negative returns unless accompanied by guidance, training, and monitoring.