High-quality LLM suggestions markedly improve caseworkers’ SNAP accuracy—by about 27 percentage points—while incorrect suggestions substantially reduce accuracy; gains level off above roughly 80% chatbot accuracy, revealing an 'AI underreliance plateau' that complicates real-world deployment.

Main Finding

LLM-based chatbot suggestions can substantially improve nonprofit caseworkers' accuracy in giving benefits eligibility guidance, but gains depend strongly on chatbot quality. High-quality chatbots (≈96–100% accurate) raised caseworker accuracy by about 27 percentage points, while incorrect chatbot suggestions can substantially harm performance. As chatbot accuracy rises, additional caseworker gains diminish — an "AI underreliance plateau" that limits returns from further model improvements unless human-AI interaction is addressed.

Key Points

• Benchmark: The authors built a 770-question multiple-choice benchmark of realistic, difficult questions a caseworker might receive about programs like SNAP.
• Experiment design: Randomized controlled trial with nonprofit caseworkers in Los Angeles. Ground truth answers were expert-verified.
• Baseline: Control (no chatbot suggestions) mean accuracy = 49%.
• Treatment: Chatbot suggestions were shown to caseworkers; the suggestions were artificially varied to have aggregate accuracies between 53% (low) and 100% (high).
• Positive effect: High-quality chatbots (96–100% accurate) increased caseworker accuracy by ~27 percentage points relative to control.
• Negative effect: Incorrect chatbot suggestions reduced caseworker accuracy substantially; on "easy" questions (where control performance was best), incorrect suggestions caused about a two-thirds reduction in accuracy — evidence of anchoring/automation bias.
• Diminishing returns: Improvements in caseworker accuracy level off as chatbot accuracy increases ("AI underreliance plateau"), indicating that even very accurate models may not produce proportional human performance gains.
• Implicit tradeoff: Both the probability of model error and human susceptibility to wrong suggestions matter; average model accuracy alone is not sufficient to predict net benefit.

Data & Methods

• Dataset: 770 multiple-choice items designed to reflect difficult, realistic eligibility questions for social safety-net programs (e.g., SNAP).
• Ground truth: Answers verified by subject-matter experts.
• Participants: Caseworkers recruited from nonprofit outreach organizations in Los Angeles (field-relevant users).
• Intervention: Randomized assignment to control (no suggestions) or treatment (saw chatbot suggestions). Treatment chatbot outputs were simulated and systematically varied to produce different aggregate accuracies (53% up to 100%) so causal effects of suggestion quality could be measured.
• Outcomes: Caseworker answer accuracy overall and at the question level; analysis of how correct vs incorrect suggestions affected performance and how gains scaled with chatbot accuracy.

Implications for AI Economics

• Value of accuracy is nonlinear: Returns to investing in model accuracy are large up to a point (substantial gains from low→high accuracy) but exhibit diminishing marginal benefits because humans underuse even highly accurate suggestions. Economic models of AI value should incorporate human adoption/underreliance dynamics, not just model performance.
• Externalities of errors: Incorrect suggestions can impose large negative externalities (reduced worker accuracy, potential harm to vulnerable clients). Cost–benefit analyses must weigh the asymmetric harm of errors, particularly in high-stakes public-service settings.
• Design and deployment matter economically: Interfaces, uncertainty communication, worker training, and trust calibration (e.g., show confidences, provide explanations, allow easy verification) could increase effective uptake and shift the underreliance plateau, raising realized returns to model improvements.
• Incentives & regulation: Procurement and oversight of assistive AI in social services should set accuracy thresholds, require evaluation with real users, and monitor worst-case error modes. Contracts or subsidies to improve UI/education may be as important as spending to improve base model accuracy.
• Labor productivity and distributional effects: High-quality assistive AI can raise caseworker productivity and expand access to accurate guidance, especially for uncommon client situations. But intermediate-accuracy systems risk making outcomes worse for clients of less-experienced workers or on straightforward cases if errors anchor decisions.
• Measurement for policy: Evaluations for public-sector AI should use user-in-the-loop randomized trials (like this study) and question-level analysis, rather than model-only benchmarks, to estimate true social value and risks.
• Research priorities: Study interventions to overcome underreliance (training, calibrated confidence scores, explanation quality), measure downstream client outcomes and welfare, and quantify the economic threshold where additional model accuracy yields negligible incremental social benefit.