AI diagnostic predictions improve chest X‑ray interpretation most for lower‑ability clinicians and for those whose confidence better matches accuracy; a replication using 68 radiologists and 11,420 paired reads confirms larger incremental value from AI for these groups.

Main Finding

This paper replicates the core result from Caplin et al. (2025, "ABCs") in a professional medical setting: when radiologists read chest X‑rays with access to state‑of‑the‑art AI predictions, lower baseline ability predicts larger incremental gains from AI and better belief calibration predicts larger gains. The qualitative pattern (largest gains for low‑ability, well‑calibrated users) holds, though the average treatment effect and coefficient magnitudes are smaller in the pooled radiology sample than in the original lab experiment.

Key Points

• Replication target: Caplin et al. (2025b) finding that ability and belief calibration (the “ABCs”) predict heterogeneous returns to AI assistance.
• Setting and sample: uses the Collab‑CXR / AMRS radiology experiment (repeated‑case designs), 68 radiologists, 11,420 paired radiologist–case observations, aggregated to radiologist × treatment outcomes (136 observations).
• Main quantitative results (pooled Designs 2–3, standardized traits):
  • Average AI effect on outcome‑block accuracy: +1.79 percentage points (s.e. 0.67). Original CDLMMWY reported +6.16 pp in comparable specification.
  • Ability × AI interaction (β1): −1.45 (s.e. 0.93) — lower baseline ability associated with larger AI gains.
  • Calibration × AI interaction (β2): +1.86 (s.e. 0.95) — better‑calibrated users gain more from AI.
• Heterogeneous fitted AI gains at ±1 SD of traits (pooled):
  • Low ability / high calibration: +5.10 pp
  • Low ability / low calibration: +1.38 pp
  • High ability / high calibration: +2.20 pp
  • High ability / low calibration: −1.53 pp
• Robustness:
  • Design‑specific estimates (Design 2 and Design 3) have the same sign; Design 3 yields larger magnitudes (ability × AI = −5.57, calibration × AI = +3.22).
  • Wild cluster bootstrap p‑values: pooled calibration interaction significant (p≈0.043); pooled ability interaction not significant at conventional levels (p≈0.124); Design 3 interactions are statistically significant.
• Interpretation: behavioral mechanism (users’ ability and their self‑calibration affect when they defer to AI) appears to generalize from a lab age‑guessing task to professional diagnostic work.

Data & Methods

• Data source: Collab‑CXR repository (Moehring et al. 2025); uses AMRS radiology experiment repeated‑case designs (Designs 2 & 3) where the same radiologist reads the same cases with/without AI in controlled sequences.
• Unit and construction:
  • Start with repeated radiologist–patient–pathology readings; create a patient‑level split into a “skill block” and an “outcome block” (global split so each patient’s readings stay in one block).
  • Estimate baseline ability (Ah) and calibration (Ch) from no‑AI cases in the skill block.
  • Measure outcomes Yh(0) and Yh(1) as average no‑AI and AI‑assisted accuracy in the outcome block.
  • Aggregate to radiologist × treatment observations (two per radiologist).
• Measures:
  • Correctness defined using a 50% decision threshold on reported diagnostic probability; accuracy Ah = mean correctness.
  • Confidence = max{Bhi, 1−Bhi}; net confidence = confidence − correctness; calibration Ch = −|average net confidence| (so closer to zero = better calibration, sign flipped).
• Regression specification (analogue to CDLMMWY col.3): Yht = α0 + α1 Ah + α2 Ch + β0 AI_t + β1 (Ah × AI_t) + β2 (Ch × AI_t) + εht
  • Ah and Ch standardized within sample; AI_t indicator for AI access.
  • Standard errors clustered by radiologist with finite‑sample correction; wild cluster bootstrap used for robustness.
• Design choices to mirror original study: paired patient sorting and splitting to balance skill/outcome blocks and minimize calibration/accuracy differences across blocks.

Implications for AI Economics

• External validity: The ABCs (ability & calibration) mechanism survives a move from a low‑stakes crowdworker task to high‑expertise, medically meaningful decisions. This strengthens claims that heterogeneous returns to AI are a general feature of human–AI collaboration.
• Policy and management implications:
  • Allocation of AI tools should consider both user ability and calibration. Targeting AI to low‑ability but well‑calibrated users may yield the largest productivity improvements.
  • Training programs should address not only skill (accuracy) but also metacognitive calibration (alignment of confidence with accuracy).
  • Interface and workflow design should surface uncertainty and support appropriate deference decisions (e.g., confidence displays, decision thresholds, recommended overrides).
  • Labor market effects: heterogeneity in returns implies redistribution of productive tasks and complementarity patterns that depend on both skill and self‑knowledge, affecting who benefits from AI adoption and how wages/roles evolve.
• Research implications and cautions:
  • Effect sizes vary by domain and design: pooled AI gain in radiology is smaller than in the original lab task, and precision is limited by expert sample size. Sector‑specific evaluation is necessary.
  • Future work should test interventions that improve calibration (feedback, training, interface cues) and measure downstream welfare/clinical outcomes, not just diagnostic accuracy.
  • Measuring beliefs/confidence in field deployments is valuable for predicting heterogeneous impacts and designing allocation rules or subsidies for AI deployment.

Overall, this replication provides supportive external‑validity evidence that ability and belief calibration shape who benefits from AI, and it highlights practical levers (training, interfaces, allocation) relevant for policymakers and firms deploying AI in skilled labor markets.