A new evaluation framework warns that helpful AI fact‑checking tools may not leave users better able to judge new claims: Trattner defines ‘epistemic transfer’ and prescribes randomized tests and two metrics—ETE and TRC—to distinguish capability building from mere dependence.

Main Finding

The paper defines and formalizes "epistemic transfer": the effect that prior use of an AI-assisted verification tool has on a user’s later unassisted ability to evaluate novel claims. It introduces two complementary estimands — the Epistemic Transfer Effect (ETE) (delayed unassisted performance compared across conditions) and the Tool-Removal Cost (TRC) (immediate within-person performance drop when the tool is removed) — and provides a concrete, preregisterable experimental protocol to measure them. The key insight is that evaluating AI verification tools requires measuring not only immediate assisted gains but also what the tool leaves behind (positive learning, inertness, or de-skilling).

Key Points

• Definition
  • Epistemic transfer: change in later unassisted performance on novel claims after prior interaction with an AI verification system. Must use a delayed, unassisted, held-out test.
• Two core estimands
  • ETE(c, k, d, b): difference in delayed unassisted performance for AI condition c versus comparator k at transfer distance d and assessment regime b.
  • TRC(c): within-subject performance gap on matched novel items when the tool is available vs removed.
• Diagnostic logic
  • Read ETE and TRC jointly to distinguish outcomes: capability building, capability + tool advantage, verification on loan (high TRC, low/no ETE), or de-skilling (negative ETE).
• Core experimental design
  • Between-subjects: four conditions — answer-first AI (verdict + explanation), evidence-first AI (sources + structured prompts), active practice (unassisted verification), no-practice (matched-duration filler).
  • Within-subjects: immediate removal probe for AI conditions (matched items with/without tool) to estimate TRC.
  • Timing: baseline unassisted set, practice phase (repeated trials), immediate removal probe, delayed unassisted test after retention interval (e.g., 7–14 days), optional later follow-up.
• Outcomes and measurement
  • Primary: accuracy / discernment on novel claims.
  • Secondary: confidence calibration, search/verification behaviors (sources, lateral reading), effort/persistence, trust/reliance measures.
• Analysis recommendations
  • Use mixed-effects models (participants and items random effects); report marginal effects and CIs; equivalence tests where appropriate.
  • Power via simulation accounting for nesting; indicative scale: several hundred participants per condition (total ~1,200–2,000) for small (3–5 pp) delayed differences with 20–30 items.
• Practical design cautions
  • Include an active-practice control — without it you cannot tell if the tool builds or displaces capability.
  • Design transfer distance (near/mid/far) and avoid leakage (paraphrases) between practice and delayed-test items.
  • Counterbalance probe order and match probe exposure across all arms.

Data & Methods

• Proposed experimental protocol (preregisterable):
• Baseline: unassisted judgments on a claim set; collect confidence and behavioral logs.
• Random assignment to one of four practice conditions (answer-first AI, evidence-first AI, active unassisted practice, no-practice filler).
• Immediate removal probe (AI arms only): within-subject matched novel items with vs without tool — randomized/counterbalanced — to estimate TRC. Active and no-practice arms should complete matched probe blocks to keep exposure equivalent.
• Delayed unassisted test after retention interval (e.g., 7–14 days) on novel claims, stratified by transfer distance.
• Debrief and provide correct source info.
• Outcomes: binary/ordinal/continuous veracity ratings; calibration measures; behavior logs (search actions, sources opened, time); effort/persistence metrics; attitudinal measures.
• Statistical model example (binary accuracy):
  • Mixed logit: logit Pr(Yij=1) = β0 + β1 Ci + β2 Dj + β3 (Ci×Dj) + γ'Xi + ui + vj
    • Ci: condition, Dj: transfer distance, Xi: baseline covariates, ui/vj: participant/item random intercepts.
• Formal estimands:
  • ETE(c,k,d,b) = E[Y_delay | c,d,b,X0] − E[Y_delay | k,d,b,X0]
  • TRC(c) = E[Y_probe | tool available, c] − E[Y_probe | tool removed, c]
• Feasibility notes:
  • A two-wave online panel experiment is feasible; typical attrition manageable with oversampling (~25%).
  • Use pilot items to calibrate difficulty and avoid ceiling/floor effects.

Implications for AI Economics

• Valuing AI verification tools requires dynamic, not just instantaneous, accounting
  • Short-run gains (assisted accuracy) can overstate social value if tools produce weak ETE or negative ETE (de-skilling). Conversely, large TRC with strong ETE suggests tools are valuable as persistent complements to skill formation.
  • Cost–benefit analyses (procurement, licensing, subscription pricing) should incorporate ETE as a persistence parameter influencing long-run productivity and required training investments.
• Human capital and labor-market effects
  • Tools that lower later unassisted capability (negative ETE) imply depreciation of worker skill and higher dependency on tool access — affecting labor demand for verification-trained roles, hiring/training strategies, and the value of tool access for firms vs freelancers.
  • Heterogeneous ETE/TRC across skill levels or domains implies distributional effects: less-skilled users may become dependent, increasing inequality in epistemic capabilities.
• Platform and information economics
  • Platforms that supply or surface verification AIs generate externalities: improved in-the-moment accuracy may reduce downstream individual capacity to judge future information, affecting misinformation dynamics, advertising markets, and content moderation costs. Regulators and platforms should measure both TRC and ETE when assessing interventions.
  • Network effects: platform-level adoption that produces strong positive ETE can increase overall information reliability over time; if ETE is negative, reliance can amplify fragility (system outages produce large performance drops).
• Product design and market differentiation
  • Evidence-first designs are hypothesized to produce higher ETE (by keeping users engaged and exposing strategies) than answer-first designs — this can be a basis for product claims, pricing segmentation, and investment in interface design.
  • Firms should consider the tradeoff between frictionless assistance (higher immediate uptake, larger TRC risk) and pedagogical support (possibly lower immediate TRC and higher ETE).
• Policy and regulation
  • Evaluations for certification or procurement (government, schools, health agencies) should require ETE/TRC-style evidence to avoid buying tools that "lock in" dependence.
  • Disclosure and labeling policies could mandate information about likely persistence/transfer (ETE estimates) similar to claims about algorithmic performance.
• Empirical research agenda for economists
  • Incorporate ETE as a persistence parameter in dynamic adoption models (e.g., replacing or augmenting productivity multipliers with retained-capability multipliers).
  • Estimate welfare impacts of AI tools that affect learning/skill formation (e.g., incorporate ETE into human capital accumulation models).
  • Study complementarities between tools and formal training investments (are tools substitutes for training or complements that magnify long-run returns?).
  • Measure heterogeneity by baseline skill, domain, and institutional context to assess distributional consequences.
• Practical recommendation
  • When evaluating or procuring AI verification tools, economists, evaluators, and policymakers should require experiments or field studies that report both ETE and TRC (and their joint diagnostic interpretation) rather than only immediate assisted accuracy or user satisfaction.

Summary takeaway: to judge the economic and social value of AI verification tools you must measure both what they do now (TRC/assisted gains) and what they leave behind (ETE). The paper provides a concrete, implementable protocol and statistical framework to do that.