Short, introductory AI courses for clinicians improve confidence, attitudes and basic skills but leave clinical workflows and patient outcomes unchanged; most programs are brief, academic, and aimed at students or early‑career clinicians, limiting their likely productivity payoff.

Main Finding

A systematic review of 27 evaluated AI education/training programs for the healthcare workforce found consistent short‑term improvements in AI literacy mapped to Kirkpatrick‑Barr levels 1–3 (learner reaction, attitudes/perceptions, knowledge/skills, behavior change). No programs demonstrated level‑4 outcomes (organizational change, patient outcomes, or metacognitive mastery). Most programs were introductory, short, and delivered in academic settings, suggesting early-stage progress but insufficient depth to build advanced clinical AI competencies.

Key Points

• Scope and search: PRISMA‑guided review, PROSPERO‑registered; five databases searched (PubMed, Scopus, CINAHL, Embase, ERIC) on 20 Aug 2024. Inclusion: any evaluated AI training/education intervention for the healthcare workforce.
• Studies included: 27 evaluated programs.
• Outcomes: Improvements reported at Kirkpatrick‑Barr levels 1–3 (satisfaction/reaction; shifts in attitudes/perceptions; gains in knowledge/skills; behavior change). No evaluated program reported level‑4 outcomes (organizational impact, patient benefit, or sustained metacognitive learning).
• Program characteristics:
  • Short duration: 44% categorized as short courses.
  • Setting: 56% delivered in academic settings.
  • Participants: 44% targeted doctors, 44% targeted medical students (overlap possible).
  • Career stage: 56% at entry‑to‑practice level.
  • Content: 67% taught introductory AI; technical/advanced AI skills were less frequent.
• Overall interpretation: Programs produce measurable early learning gains but frequently lack depth, duration, and design features to produce advanced competencies or measurable system/patient impacts.

Data & Methods

• Review design: Systematic review following PRISMA guidelines; protocol registered in PROSPERO.
• Databases searched: PubMed, Scopus, CINAHL, Embase, ERIC; search date 20 Aug 2024.
• Eligibility: Any study design reporting an evaluation of an AI education or training intervention aimed at the healthcare workforce.
• Included studies: 27; extracted data on program length, setting, target audience, content focus, and reported evaluation outcomes mapped to the Kirkpatrick‑Barr training evaluation hierarchy (levels 1–4).
• Outcome mapping: Evaluations typically used learner surveys, knowledge/skill tests, or self‑reported behavior change measures to classify outcomes into Kirkpatrick‑Barr levels 1–3; no studies provided evidence of organizational change or patient‑level benefits (level 4).

Implications for AI Economics

• Human capital and productivity
  • Short, introductory programs likely yield modest increases in individual AI literacy but may not generate the deeper competencies required to materially change clinical task allocation or workflow productivity.
  • For economic models of AI adoption, current training investments may produce limited short‑run returns; deeper, longer, and practice‑embedded training could be necessary to unlock larger productivity gains.
• Labor market effects
  • Predominant focus on entry‑level trainees (students/early practitioners) may increase future workforce supply of basic AI literacy but leaves current mid‑career clinicians undertrained—potentially slowing adoption or creating heterogeneous skill premiums.
  • Limited advanced technical training reduces the near‑term supply of clinician‑AI integrators (roles that bridge clinical and technical domains), which could keep wages/premia for such specialists high.
• Policy, accreditation, and standardization
  • The absence of level‑4 evidence (organizational/patient outcomes) hampers cost‑benefit and return‑on‑investment analyses needed by health systems and payers to justify larger upskilling expenditures.
  • Standardized curricula, competency frameworks, and validated assessment tools would improve comparability and enable economic evaluation across programs.
• Market and scaling considerations
  • Growing demand for AI education creates market opportunities (commercial courses, continuing medical education, institutional programs), but heterogeneous quality and outcome measurement limit purchasers’ ability to identify high‑value offerings.
  • Economies of scale (online, modular, credentialed programs) could lower per‑learner costs, but evidence is required that scaled programs achieve higher‑order outcomes.
• Research and evaluation priorities for economic analysis
  • Conduct cost‑effectiveness and cost‑benefit studies linking training to measurable organizational outcomes (workflow changes, error rates, throughput, patient outcomes).
  • Evaluate training targeted at mid‑career clinicians and interdisciplinary teams to estimate marginal returns relative to student/entry‑level programs.
  • Model long‑term impacts on labor demand, task reallocation, wage effects, and healthcare spending under alternative upskilling investments.
• Practical recommendations for stakeholders
  • Fund and prioritize longer, practice‑embedded programs that teach beyond introductory concepts (technical literacy, model interpretation, human‑AI teaming), and include robust outcome measurement tied to organizational and patient metrics.
  • Develop standardized competency frameworks and assessments to enable credentialing and rigorous economic evaluation.
  • Health systems should pilot and evaluate scaled upskilling programs with embedded economic evaluation to guide investment decisions.

If you want, I can draft a short outline for an economic evaluation protocol that links AI training inputs to organizational and patient outcomes (costs, productivity, and QALYs) to illustrate how to measure level‑4 impacts.