Access to generative AI during study boosts short-term learning by 0.27 SD and yields persistent gains after one week; improvements in writing quality emerge only once students write unaided and are largest for those who use AI to augment understanding rather than to automate drafting.
Experimental Evidence on the Learning Impact of Generative AI
In an RCT with undergraduates, access to generative AI during learning raised immediate test scores by 0.27 standard deviations with gains persisting one week later, while essay-quality improvements appeared only after AI was removed and were concentrated among students who used AI to explain concepts (augmentation) rather than to generate text (automation).
We study how generative AI affects student learning in a randomized experiment. In proctored, in-person sessions, undergraduates learn about an unfamiliar topic and write an analytical essay with or without access to off-the-shelf generative AI, then complete unaided assessments immediately and one week later. We measure learning with knowledge tests (factual and conceptual understanding) and open-ended essays (higher-order skills). AI access raises immediate test scores by 0.27 standard deviations. These gains persist one week later. Essay quality, by contrast, changes little while students have AI access but improves in style and relevance one week later, when students write unaided. These delayed gains are larger among augmentation users-who use AI to explain concepts rather than generate text-whereas automation users' short-run quality gains vanish once AI is removed. We find evidence for two mechanisms behind the learning gains: students shift time away from drafting text and toward reading and searching for information, and they report greater learning enjoyment.
Summary
Main Finding
Access to off-the-shelf generative AI (ChatGPT GPT-4o) during a time-limited learning-and-writing task raised undergraduates’ immediate knowledge-test performance by 6.7 percentage points (0.27 standard deviations) relative to no-AI, and about 76% of that gain persisted one week later. Essay-quality gains emerged only after AI was removed, and persistence depends on how students use AI: durable gains for “augmentation” users who use AI to explain concepts, but short-lived or absent gains for “automation” users who use AI to draft essays.
Key Points
-
Sample and uptake
- Randomized in-person experiment with N = 211 undergraduates, two sessions ~1 week apart.
- In the AI-allowed arm, 68% of students used generative AI during the learning phase; near-zero usage in control.
- Treated students overwhelmingly reported AI as helpful (88%).
-
Main effects
- Immediate knowledge-test: +6.7 pp (baseline 56.3%), = 0.27 SD.
- One-week retention test: +5.1 pp (≈76% of immediate effect persists).
- Session-One essays: small/imprecise overall quality change; stylistic signatures of AI (longer essays, simpler sentences, higher share flagged as AI-generated).
- Session-Two essays (unaided): improved style, clarity, and relevance among treated students; AI-detection/stylistic differences fade.
-
Heterogeneity by usage mode
- Using ChatGPT logs, users classified as:
- Augmentation (49% of AI users): prompt for explanations, tutoring; showed persistent test gains and improved unaided essays.
- Automation (32%): prompt to draft essays; had higher AI-detection in submissions and any short-run quality gains disappeared once AI was removed.
- Gains concentrated more in higher-ability students (upper quartiles by GPA/SAT), suggesting potential widening of gaps.
- Using ChatGPT logs, users classified as:
-
Mechanisms
- Time allocation shifted (no change in total time): −12% time on drafting/writing, +10% time reading/searching—consistent with AI reducing mechanical drafting and freeing effort for information processing.
- Treated students reported 13% higher enjoyment of learning.
- Some increase in rule violations on the first test, but violations explain at most a modest share of gains.
-
Measurement and robustness
- Multi-dimensional outcomes: factual/conceptual multiple-choice tests, human and AI essay grades, linguistic features (readability, lexical diversity), AI-detector flags, ChatGPT conversation logs, self-reports, and proctor observations.
- Treatment was off-the-shelf AI access (logged-in ChatGPT provided) — not a heavily scaffolded/tailored tutor.
Data & Methods
-
Design
- Between-subjects RCT: AI-allowed vs AI-forbidden during Session One learning phase; all students unaided in Session Two.
- Session One: 60-minute lab with baseline test, up-to-35-minute learning + ~500-word analytical essay, post-learning test (5 multiple-choice items).
- Session Two (~1 week later): 10-item knowledge test and 20-minute analytical essay on the same topic (different prompt), both without external resources or AI.
-
Sample and administration
- 211 undergraduates; topics assigned at random (block randomization over 3 technical topics: blockchain, carbon capture, CRISPR).
- Proctored in-person labs, privacy dividers, platform logging (Qualtrics), ChatGPT conversation logs for treated computers, proctor compliance checks, and exit surveys.
-
Treatment and compliance
- AI-allowed: explicit permission + an already-logged-in ChatGPT (GPT-4o) tab.
- AI-forbidden: explicit prohibition; comparable browser tabs without ChatGPT.
- Compliance monitored via proctors, self-reports, platform snapshots, and ChatGPT logs.
-
Outcomes and classification
- Knowledge tests target factual recall and conceptual understanding.
- Essays scored by human graders and AI graders along multiple dimensions; linguistic and AI-detection metrics computed.
- Users’ ChatGPT logs coded to classify usage into augmentation vs automation, and to record purposes (explain concepts, draft, summarize, etc.).
-
Additional analyses
- Heterogeneity by baseline ability (GPA/SAT), usage mode, and self-reported beliefs.
- Checks for cheating/rule violations and their contribution to test-score differences.
Implications for AI Economics
-
Human-capital accumulation vs task productivity
- Off-the-shelf generative AI can enhance durable learning (not just immediate task performance) when students engage it as an explanatory/tutoring tool. This suggests AI may raise long-run worker productivity both directly (task augmentation) and indirectly (skill accumulation).
-
Importance of use mode: augmentation vs automation
- Economic returns to AI depend on how people use it. Policies, instructional design, and product features that steer users toward augmentation (explanations, worked examples, prompting for reasoning) should increase human-capital gains; unchecked automation use risks short-lived gains or deskilling on unaided tasks.
-
Distributional consequences
- Larger gains among higher-ability students imply potential to widen achievement gaps. Economists and educators should consider equity: differential access, AI literacy, and incentives may amplify inequality in skill accumulation.
-
Assessment, incentives, and academic integrity
- AI changes the production of written work (stylistic signatures while AI is available) and complicates assessments that assume unaided outputs. Institutions may need to redesign assessments (in-person/closed-book, process-based evaluation, tasks requiring idiosyncratic reasoning) and invest in AI-literacy and honor-system enforcement.
-
Policy and product design levers
- Interventions likely to amplify benefits:
- Teaching students effective prompting/use for explanation and self-explanation.
- Integrating AI as a tutor (scaffolded prompts, reflection prompts) rather than permitting end-to-end automation.
- Building feedback loops that favor learner-generated outputs (e.g., requiring students to show work or partial attempts).
- Regulators and institutions should weigh trade-offs: productivity and learning gains versus integrity and inequality risks.
- Interventions likely to amplify benefits:
-
Directions for further research (economics-focused)
- Longer-run follow-up: do gains persist beyond one week and translate into course performance or labor-market skills?
- External validity: replication in varied populations (younger students, professional learners), settings (online courses), and with different AI models or institutional scaffolds.
- Causal mechanisms: experimentally manipulate prompts/scaffolds to test augmentation vs automation incentives and measure learning processes (metacognition, retrieval practice).
- Welfare and distributional analyses: quantify long-run returns to education investment when generative AI is widely adopted, and model equilibrium labor-market effects if AI boosts human capital heterogeneously.
Limitations to keep in mind: proctored lab setting with undergraduates and a one-week retention window; treatment used an off-the-shelf model without bespoke scaffolding, so effects may differ for tailored AI tutors or other contexts.
Assessment
Paper Typerct
Evidence Strengthhigh — A randomized controlled trial provides strong internal validity for causal claims; outcomes include objective knowledge tests and blind-scored essays measured immediately and after one week, and the paper tests plausible mechanisms (time reallocation, enjoyment). Remaining threats are typical RCT concerns (attrition, measurement of usage by self-report, and short follow-up), but they do not undermine the core randomization-based identification.
Methods Rigorhigh — Proctored, in-person administration reduces contamination/cheating; multiple outcome types (factual/conceptual tests and open-ended essays) and delayed follow-up strengthen inference; subgroup and mechanism analyses add credibility. Details not provided here on sample size, pre-registration, balance checks, or coder blinding for essays, which would further strengthen the assessment.
SampleUndergraduate students participating in proctored, in-person learning sessions where they learned an unfamiliar topic and wrote analytical essays; randomly assigned to have or not have access to an off-the-shelf generative AI during the learning/writing period; outcomes include immediate and one-week delayed knowledge tests (factual and conceptual) and open-ended essays; usage patterns (augmentation vs automation) and time-use/enjoyment self-reports were collected.
Themesskills_training human_ai_collab
IdentificationRandomized assignment of undergraduate students to in-person, proctored sessions with or without access to an off-the-shelf generative AI tool; causal effects estimated by comparing immediate and one-week outcomes between treatment and control groups, with subgroup (augmentation vs automation users) and mechanism checks using time-use and self-reported enjoyment measures.
GeneralizabilityUndergraduate student sample (limits transferability to older workers or professional contexts), Lab-style, proctored in-person sessions (may differ from real-world, on-the-job learning environments), Short follow-up (one week) — limited evidence on long-term retention and labor-market outcomes, Single task domain: learning an unfamiliar academic topic and analytical essay writing (may not generalize to other tasks or disciplines), Results tied to the specific off-the-shelf AI tools used (other models or interfaces could differ), Potential novelty effects for first-time AI users that may attenuate over time
Claims (7)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| AI access raises immediate test scores by 0.27 standard deviations. Skill Acquisition | positive | immediate knowledge test scores (factual and conceptual understanding) |
Reading fidelity
high
Study strength
high
|
0.27 standard deviations
|
| The immediate test-score gains from AI access persist one week later. Skill Acquisition | positive | delayed knowledge test scores (one-week follow-up) |
Reading fidelity
high
Study strength
high
|
not reported
|
| Essay quality changes little while students have AI access but improves in style and relevance one week later when students write unaided. Output Quality | mixed | essay quality (immediate and one-week delayed), specifically style and relevance |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Delayed learning gains are larger among 'augmentation' users who use AI to explain concepts rather than generate text. Skill Acquisition | positive | delayed learning gains / one-week knowledge test scores (heterogeneous treatment effect by AI usage type) |
Reading fidelity
medium
Study strength
medium
|
not reported
|
| Automation users' short-run essay quality gains vanish once AI is removed. Output Quality | null_result | essay quality (short-run gains and one-week follow-up) for automation users |
Reading fidelity
medium
Study strength
medium
|
not reported
|
| Students shift time away from drafting text and toward reading and searching for information when they have AI access. Task Allocation | positive | time allocation between drafting text and reading/searching for information |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Students with AI access report greater learning enjoyment. Worker Satisfaction | positive | self-reported learning enjoyment |
Reading fidelity
high
Study strength
medium
|
not reported
|