Many LM-driven triage errors reflect unstated priorities, not lack of diagnostic ability: models frequently under-triage because they implicitly prioritize conserving resources, but plain‑language utility prompts can steer capable models toward safety-focused referrals while preserving their probabilistic assessments.

Main Finding

The paper reframes emergency medical triage with language models (LMs) as a decision problem, not just a prediction problem. Across several frontier LMs, the authors show that (1) models typically have high diagnostic discrimination (AUROC ~0.88–0.99), (2) much observed under‑triage can be explained by implicit, unstated utility trade‑offs that prioritize resource conservation over safety, and (3) capable models can be steered toward desired operating points by specifying explicit utilities (cost ratios or probability thresholds) at prompt time. Thus safe deployment requires making decision objectives explicit (probabilities + utilities), testing calibration, and mapping predictions to actions with known trade‑offs.

Key Points

• Decision-analytic framing: Triage ≠ only classification. Two components matter — the model’s probabilistic assessment (P(emergency)) and an explicit utility/cost function weighting false negatives vs false positives.
• Recovering implicit utilities: By eliciting a model’s expressed probability for each vignette and separately eliciting its triage recommendation, the authors infer the implicit cost ratio that best explains the model’s behavior.
• Empirical findings:
  • Models demonstrate strong ranking ability: AUROC ≈ 0.95–0.99 on the primary set and 0.88–0.94 on the expanded set.
  • The deployed ChatGPT Health behavior (reported earlier) corresponded to a resource‑prioritized implicit tradeoff: it caught ~48% of gold‑standard emergencies while referring ~10% of non‑emergencies.
  • Prompting with explicit cost ratios (e.g., weight missed emergencies 5× or 10×) moved operating points up the ROC curve, substantially increasing sensitivity with only modest increases in false alarms (e.g., ~30–50% relative increase in true emergency routing depending on set).
• Decision‑analytic steering works variably:
  • High‑capability models and models with inference‑time reasoning track requested utilities closely (e.g., GPT‑5.4).
  • Smaller / reasoning‑disabled variants or some Claude models show dampened or limited steering—recovered cost ratios span a narrower range than prompted.
• Calibration and model‑specific thresholds: Different models require different probability thresholds to realize the same utility trade‑off because of calibration differences. A fixed, model‑specific threshold can be selected to minimize cost‑weighted error for a chosen target ratio.
• Defaults are heterogeneous and unobservable: Default implicit priorities (cost ratios) vary substantially across model families and settings; deploying “with no preference stated” effectively implements an uncontrolled, unobservable value judgment.

Data & Methods

• Data:
  • Primary set: 576 vignettes (subset used by Ramaswamy et al. 2026): 64 unambiguous emergencies, 512 non‑emergencies.
  • Expanded set: 1,248 variants (original 960 + 288 textbook cases), with 640 cases labelled as emergencies under a broader definition.
  • Cases are physician‑authored clinical vignettes used previously to evaluate a consumer triage system.
• Models evaluated:
  • Multiple frontier LMs and variants: GPT‑5‑mini, GPT‑5.4, DeepSeek V4 Pro, Claude Fable 5, and other family variants at different “reasoning” settings.
• Procedure:
  • For each vignette, elicit (1) the model’s estimated probability that emergency care is needed, and (2) the model’s triage recommendation (refer or not).
  • Map decisions to probability thresholds to infer the implicit cost ratio (missed emergency cost vs unnecessary referral cost) that explains behavior.
  • Generate ROC curves from elicited probabilities; plot actual operating points from model decisions.
  • Test decision‑analytic steering by re‑prompting models with explicit utility instructions (cost ratios or explicit probability thresholds) and measuring how recovered cost ratios shift.
  • Compute optimal model‑specific fixed thresholds for prespecified target cost ratios to account for calibration differences.
• Key caveats:
  • Cannot exactly reproduce the original ChatGPT Health system (system prompt/safeguards/configuration unknown).
  • Vignette data (not live patient outcomes) and some “edge” case label ambiguity affect interpretation.
  • Some deployed behavior may not be representable by a single consistent utility function.

Implications for AI Economics

• Incentives and market outcomes:
  • If model providers do not expose or align implicit utilities, market deployments can systematically under‑ or over‑triage depending on unobserved provider incentives (cost savings, liability avoidance, product conservatism), producing externalities (patient harm or excess system use).
  • Buyers (hospitals, insurers, regulators) face information asymmetry: model accuracy alone (AUROC) is insufficient; they need visibility into calibration, implied operating points, and the utility mapping.
• Procurement and contracting:
  • Contracts should specify target decision objectives (explicit cost ratios or probability thresholds) and require models to accept and implement them, or require model‑specific calibration to meet the buyer’s desired operating point.
  • Payment and liability rules can be designed to align provider incentives with socially preferred trade‑offs (e.g., reimbursing for higher sensitivity in contexts where missed cases are costly).
• Regulation and certification:
  • Regulators should require (a) probabilistic outputs and calibration metrics, (b) disclosure of default operating points or implicit cost ratios, and (c) ability to input explicit decision objectives and verifiably implement them.
  • Certification should evaluate decision performance under a range of explicit utilities, not just raw accuracy.
• Value of information and welfare analysis:
  • The paper highlights that information (better probabilities) and explicit utility specification are complementary. Improving discrimination (probabilities) increases the Pareto frontier of achievable trade‑offs; specifying utilities picks a point on that frontier. Welfare maximization requires jointly choosing both.
  • Cost‑weighted error metrics (based on stakeholder weights) should be used in economic evaluations of model deployment and in cost–benefit calculations.
• Product design & pricing:
  • Models that are steerable and well‑calibrated across operating points have higher value in contexts requiring explicit trade‑offs; pricing and licensing could reflect this added capability.
  • Prompt‑based steering is a lower‑cost alternative to retraining for aligning behavior, but only if models are sufficiently capable; this affects product feature design and value propositions.
• Strategic and distributional concerns:
  • Multi‑stakeholder settings (patients, clinicians, payers, system operators) may have conflicting utility weights. Mechanisms for aggregating or selecting among preferences (e.g., social welfare functions, committee rules, context‑dependent defaults) are required.
  • There is risk of strategic misuse: organizations could deliberately set utilities that undervalue safety to cut costs. Monitoring, audits, and regulation are needed to guard against socially harmful utility choices.
• Research & policy priorities:
  • Develop standardized protocols for eliciting, reporting, and auditing model calibration, implied utilities, and decision‑rules.
  • Study robustness of steering under distributional shift and adversarial prompting.
  • Quantify the socioeconomic trade‑offs (costs of missed events vs costs of overuse) for different healthcare systems to derive welfare‑optimal operating points.

In short: the paper urges shifting AI evaluation and procurement from accuracy‑only metrics to a decision‑analytic regime (probabilities + explicit utilities + mapping to actions). For AI economics this implies new contracting, regulatory, pricing, and auditing practices to ensure deployed models implement socially and institutionally appropriate trade‑offs.