Prompting users to spot assumptions in AI plans curbs overreliance without extra cognitive burden; hypothetical 'what-if' prompts feel more useful to users but are less effective at reducing blind trust.

Main Finding

Assumption-focused cognitive forcing functions (CFFs) applied to AI-generated execution plans for writing tasks most effectively reduced users’ overreliance on AI outputs without increasing cognitive load. Targeting argument-analysis (identifying the AI’s implicit assumptions) outperformed targeting hypothesis-testing (WhatIf) or combining both. Users nevertheless perceived the WhatIf prompt as more helpful, revealing a divergence between subjective preference and objective effectiveness. Participant traits (cognitive disposition, GenAI familiarity) influenced whether users revised their judgments, but CFF design primarily drove overreliance behavior.

Key Points

• Research question: Which plan-focused CFFs help people critically evaluate AI plans and prevent overreliance in AI-assisted writing?
• Interventions tested (between-subjects):
  • Assumptions: prompts to identify implicit assumptions in the AI plan (targets argument analysis).
  • WhatIf: prompts to reason about the impact of changing a key plan step (targets hypothesis testing).
  • Both: Assumptions then WhatIf.
  • None: control (no CFF).
• Studies:
  • Controlled experiment: n = 214 participants completing AI-assisted writing tasks with plan review.
  • Qualitative think-aloud + interviews: n = 12 participants comparing conditions.
• Primary outcomes:
  • Revision of initial readiness assessment of AI output (did users change their readiness ratings after CFF engagement).
  • Behavioral overreliance (tendency to accept AI output without critical revision).
  • Cognitive load and perceived helpfulness/trust of CFFs.
• Main behavioral result: Assumptions reduced overreliance most effectively while not increasing measured cognitive load. Combining CFFs (Both) was not additive and could be counterproductive.
• Perception gap: users tended to rate WhatIf as more helpful, despite poorer objective performance relative to Assumptions.
• Individual heterogeneity: cognitive disposition and GenAI familiarity shaped revision behavior, suggesting CFF effects interact with user traits.

Data & Methods

• Experimental design:
  • Realistic AI-assisted writing scenarios where the system generated an execution plan and a draft.
  • Participants reviewed the AI plan, completed the assigned CFF (or none), then assessed readiness of AI output and could request changes.
  • Plan-centered CFFs were implemented as short interactive prompts/quizzes tied to the AI plan (e.g., select assumptions, identify the most critical step and its impact).
• Sample:
  • Large-scale controlled experiment: n = 214 (between-subjects across four conditions).
  • Qualitative follow-up: n = 12 think-aloud interviews covering all conditions for comparative subjective feedback.
• Measures:
  • Behavioral measures of overreliance (acceptance vs revision of draft/plan).
  • Changes in readiness judgments (did CFF prompt reassessment).
  • Cognitive load (measured comparatively across conditions).
  • Subjective ratings of helpfulness and trust.
  • Recorded participant traits (cognitive disposition, GenAI familiarity) to assess moderation effects.
• Analysis:
  • Between-group comparisons to identify which CFFs changed behavior and cognitive load.
  • Qualitative coding of think-aloud data to understand perceived usefulness and friction.

Implications for AI Economics

• Productivity vs quality trade-off at the interface level:
  • Plan-focused CFFs can reduce costly errors from overreliance without imposing measurable extra cognitive load (Assumptions CFF), implying a way to preserve gains in productivity while improving output quality.
  • However, perceived helpfulness (WhatIf) does not equal effectiveness; product teams that optimize for user satisfaction alone may underdeliver on error reduction and quality control.
• Labor quality, deskilling, and human capital:
  • Effective CFFs that preserve critical evaluation (argument analysis) can slow or prevent deskilling that accompanies passive use of generative tools, supporting longer-term maintenance of worker skills and human capital.
  • Firms that deploy such CFFs may avoid downstream costs tied to degraded judgment (rework, reputational risk, regulatory fines).
• Adoption dynamics and product design incentives:
  • Designers face a trade-off: some CFFs reduce overreliance but may be perceived as less useful. Adoption and continued use depend on both objective benefits and subjective user experience. A/B testing and education may be needed to align perception with effectiveness.
  • The ineffectiveness (or harm) of combining multiple CFFs cautions against stacking friction indiscriminately; incremental, well-targeted interventions are preferable.
• Heterogeneity and targeted deployment:
  • Effects vary by worker traits (cognitive disposition, GenAI familiarity). Economic evaluations of GenAI deployment should account for heterogeneous returns—CFFs may be more valuable for less experienced users or in contexts where critical thinking has higher payoff.
  • Customizable or adaptive CFFs (e.g., enabled based on user profile, task criticality) could maximize net benefits.
• Organizational policy and regulation:
  • CFFs are a light-touch, design-based safeguard that firms could adopt to improve trust calibration and reduce automation bias. Regulators or industry standards for high-risk domains might recommend plan-focused CFFs (especially argument-analysis prompts) as part of human-in-the-loop UI requirements.
• Measurement and valuation in AI economics:
  • Economic impact assessments of GenAI should include behavioral-intervention effects (e.g., CFFs) because interface design materially alters realized productivity and error rates.
  • Cost–benefit analyses should quantify both time costs (perceived friction) and error-avoidance gains; this paper suggests some CFFs can yield net quality gains without measurable time/cognitive penalties.
• Product-market signals and mismatch risk:
  • The divergence between perceived helpfulness and objective effectiveness can distort market signals—user ratings may favor interfaces that feel helpful but deliver worse calibration. Firms and policymakers should rely on outcome-based metrics (error rates, revision frequency) when evaluating tool safety and effectiveness.

Limitations & open questions relevant to economics - Domain and task scope: experiment focused on writing tasks; generalization to other knowledge work (e.g., coding, medical synthesis, legal drafting) requires validation. - Short-term vs long-term effects: experiment measured immediate behavior; long-term impacts on skill retention, training needs, and labor productivity are not yet known. - Field deployment costs and behavioral adaptation: real-world adoption dynamics, compliance, and possible gaming or superficial responses to prompts need study. - Optimal targeting and personalization: how to cost-effectively adapt CFFs to user types and task criticality remains an open design/economic optimization problem.

Practical takeaways for economists and practitioners - When evaluating or deploying GenAI in organizations, measure both subjective satisfaction and objective calibration/quality; prefer outcome-based metrics. - Consider integrating light, plan-focused CFFs that target argument analysis to reduce overreliance without detectable extra cognitive load. - Account for heterogeneity in workforce skill and tool familiarity; allocate training and UI personalization resources where they yield the largest return in quality. - Use controlled A/B tests that track error rates and revision behavior (not just engagement) to decide which CFF designs to scale.