A ten-minute onboarding session substantially raised voluntary use of an LLM and produced measurable performance gains on a law-school exam: training increased usage from 26% to 41% and improved scores by 0.27 grade points, whereas untrained access delivered no score benefit.

Main Finding

A brief, targeted training intervention substantially increased voluntary adoption of a generative LLM and improved objective performance on a law issue‑spotting exam. Training raised reported LLM use from 26% to 41% and yielded a 0.27 grade‑point improvement versus untrained LLM access (p = 0.027, roughly one‑third of a letter grade). LLM access without training did not improve scores (and was associated with shorter answers). Principal‑stratification estimates are consistent with training working mainly by inducing additional users (extensive margin) rather than materially improving effectiveness among those who would have used the LLM anyway, though confidence intervals are wide.

Key Points

• Design: Randomized controlled trial with three arms:
  • Group 1: No GenAI access (Westlaw only).
  • Group 2: Optional LLM access (DeepSeek), no guidance.
  • Group 3: Optional LLM access + ~9.5 minute training video + 5‑question quiz.
• Sample: 213 signed up; 164 completed the study and were analyzed (Group sizes in analysis: 49, 57, 58).
• Task: Open‑book, timed (75 min) issue‑spotting examination in contract law (identification + analysis of major and sub‑issues).
• Adoption effect: Self‑reported LLM use rose from 26% (Group 2) to 41% (Group 3).
• Performance effect:
  • Trained access (Group 3) vs untrained access (Group 2): +0.27 grade points, p = 0.027.
  • Untrained LLM access vs no access: no statistically significant improvement; answers tended to be shorter.
• Mechanism evidence: Principal stratification decomposition suggests training primarily expands who chooses to use GenAI (induced users) rather than substantially increasing the per‑user effectiveness of GenAI use. Estimates consistent with this mechanism but imprecise.
• Training content emphasized prompting strategies, decomposing tasks, multiple prompts/iterations, giving feedback, and strong cautions about hallucinations and the need for human verification.
• Pre‑registration: hypotheses pre‑registered; some analyses declared when not pre‑registered.

Data & Methods

• Randomization: between‑subjects assignment to three conditions to isolate the marginal effect of training while holding LLM access constant between Groups 2 and 3.
• Participants: LL.B. and J.D. students enrolled in contract law at the University of Hong Kong; invited ~2 months before the official exam; compensation/gamification (gift card + feedback).
• Interventions:
  • DeepSeek LLM made available to Groups 2 and 3.
  • Training: 9m30s instructional video demonstrating best practices, warnings about reliability, and a short quiz (delivered only to Group 3).
• Outcome measures:
  • Primary: exam score (graded against rubric listing major issues and sub‑issues).
  • Secondary: self‑reported DeepSeek use (binary), answer length, and post‑exam survey items on prior training and attitudes.
• Estimation:
  • Mean comparisons across groups for adoption and scores (p‑values reported).
  • Principal stratification used to decompose effects into (a) adoption/extensive margin and (b) effectiveness/intensive margin among would‑be users.
• Attrition and protocol:
  • 164 of 213 completed; groups run in separate classrooms without prior disclosure of differing LLM access to minimize selection and spillovers.
• Limitations noted by authors: student sample (not practicing lawyers), single task context (issue‑spotting), self‑reported usage, short training and one LLM, and wide confidence intervals in mechanistic decomposition.

Implications for AI Economics

• Complementary investments matter: Technology access alone may not translate into productivity gains in judgment‑intensive, error‑sensitive tasks. Small, low‑cost training can materially raise adoption and realized gains.
• Adoption frictions are central: Training appears to lower psychological or perceived risk costs (k_T in the authors’ model), expanding the set of users and use‑cases—this highlights the importance of non‑pecuniary frictions in diffusion models for high‑skill occupations.
• Distributional effects & heterogeneity:
  • Theoretical model and some empirical patterns align with the idea that lower‑ability users tend to gain more from effective GenAI use, but training may shift who adopts rather than raising per‑user effectiveness; this can produce leveling effects or alter occupational task allocation depending on how training is targeted.
• Organizational policy and human capital:
  • Firms and professional organizations should treat user training, guidance, and verification protocols as complements to GenAI deployment—investments in instruction, workflows, and oversight may unlock productivity that raw access does not.
• Measurement and evaluation of AI productivity:
  • Lab/field differences matter: controlled studies showing GenAI capabilities are necessary but insufficient for estimating real‑world productivity gains; adoption and interaction patterns conditional on instruction, incentives, and accountability are critical mediators.
• Research agenda:
  • Broader field experiments with practitioners, longer or iterative training, integration of retrieval‑augmented systems, and task‑varied settings will be needed to quantify persistence, external validity, and the balance between extensive vs intensive margin effects.
• Caution for regulators and firms:
  • Rapid rollout of GenAI without investment in user literacy and verification can yield non‑trivial risks in high‑stakes professions (hallucinations, malpractice, liability). Policy should encourage training, transparent disclosure, and monitoring as part of safe adoption.