Benchmark reveals hidden triage failures: three leading LLMs tested on 200 Nigerian primary-care vignettes each manifest distinct failure modes, and conventional safety metrics mask a 77-percentage-point under-triage in Llama 3.1 8B, implying model choice and deployment cost depend on local health-system priorities.

Main Finding

IyawoBench v2.0 introduces a formal, diagnostic evaluation for LLM clinical triage and shows that conventional aggregate safety metrics can hide distinct, deployment-critical failure modes. Applied to three frontier models (Claude Sonnet 4.6, Llama 3.3 70B, Llama 3.1 8B) on a 200-vignette Nigerian primary‑care benchmark, every model exhibits at least one formal failure mode. Crucially, a lenient safety metric (used in prior work) masked a 77 percentage‑point under‑triage gap in Llama 3.1 8B (100% lenient sensitivity vs 23% strict sensitivity). Model choice that minimizes expected harm depends strongly on health‑system cost priorities — no single model dominates across plausible deployment scenarios.

Key Points

• Novel diagnostic framework and metrics:
  • Escalation Bias Index (EBI) — quantifies over‑escalation bias (range −1 to +1); CEB (Conservative Escalation Bias) defined when EBI ≥ 0.65.
  • Strict vs Lenient Sensitivity — strict requires exact REFER_NOW matches; lenient counts REFER_TODAY as “safe” for emergencies.
  • Middle‑Tier Confusion Rate (MCR) and Direction Asymmetry (DA) — measure instability and bias in the REFER_TODAY class.
  • Expected Deployment Cost (EDC) — translates confusion matrix into asymmetric clinical costs under specified cost matrices and deployment scenarios; provides actionable thresholds (EDC <1.5 viable; 1.5–3 requires prompt engineering; ≥3 unsustainable).
• Failure mode taxonomy (formal definitions):
  • Conservative Escalation Bias (CEB): high sensitivity to emergencies but low specificity on TREAT_HERE → system overload risk.
  • Systematic Downgrade Bias (SDB): lenient metric looks safe but strict sensitivity reveals many emergencies downgraded one level → patient safety risk.
  • Middle‑Tier Instability (MTI): high confusion in moderate‑urgency cases with bidirectional errors.
• Empirical highlights (200 synthetic vignettes derived from 1,200 real encounters across 19 Nigerian PHCs):
  • Class distribution: REFER_NOW 100 (50%), REFER_TODAY 60 (30%), TREAT_HERE 40 (20%).
  • Model-level snapshot:
    • Claude Sonnet 4.6: Acc 63%, Strict Sens 100%, Lenient Sens 100%, Spec_y0 52.5%, MCR 91.7% → CEB.
    • Llama 3.3 70B: Acc 68%, Strict Sens 100%, Lenient Sens 100%, Spec_y0 77.5%, MCR 91.7% → CEB, MTI, SDB.
    • Llama 3.1 8B: Acc 46%, Strict Sens 23%, Lenient Sens 100%, Spec_y0 95% → SDB (77 pp gap between strict and lenient sensitivity).
  • EBI: naive always_refer_now = 1.0 (formal CEB); Claude ≈ 0.475 (near CEB threshold).
  • Scenario‑dependent EDC winners:
    • Emergency‑Focused: naive always_refer_now (EDC = 0.70), Claude (0.91)
    • System‑Sustainability: Llama 3.1 8B (EDC = 1.31), always_refer_today (1.33)
    • Balanced‑Deployment: Llama 3.3 70B (EDC = 1.86), Claude (1.91)
• All code, data, and pipelines are open‑source: https://github.com/anthoniooladimeji11-coder/iyawobench.

Data & Methods

• Dataset:
  • 200 synthetic vignettes generated from aggregate statistics of 1,200 real patient encounters (Jan–Mar 2026) across 19 PHCs in Oyo State, Nigeria.
  • Eight febrile illness categories (malaria variants, typhoid, pneumonia, meningitis, sepsis); vignette schema includes demographics, vitals, malaria RDT, pregnancy status.
  • Ground truth triage labels y* ∈ {TREAT_HERE, REFER_TODAY, REFER_NOW} derived from WHO and Nigerian treatment guidelines.
• Models & baselines:
  • Evaluated: Claude Sonnet 4.6 (Anthropic API), Llama 3.3 70B (open), Llama 3.1 8B (open).
  • Five naive baselines: always_refer_now, always_refer_today, always_treat_here, random_uniform, class_proportional.
  • Deterministic decoding (temperature = 0), shared prompt template and parser.
• Metrics & theory:
  • Formal definitions (14) and two theorems characterizing failure modes and cost lower bounds.
  • Primary diagnostics: Accuracy, Strict/Lenient Sensitivity, Specificity on TREAT_HERE, EBI, MCR, DA, EDC, Safety‑Adjusted Accuracy (SAA).
• Deployment scenarios:
  • Three cost matrices encode different priorities: Emergency‑Focused, System‑Sustainability, Balanced (default).
  • Statistical tests: McNemar for paired proportions; Wilson score CIs; bootstrap (1,000 replicates) for cost differences.

Implications for AI Economics

• Procurement and model selection must be cost‑contextual, not single‑metric driven:
  • Benchmark rankings (accuracy or aggregate sensitivity) can be misleading; EDC and EBI make trade‑offs explicit and align selection to system objectives (e.g., minimise mortality vs preserve referral capacity).
  • Buyers in LMICs should specify deployment cost matrices reflecting capacity constraints and price models by expected deployment cost, not raw accuracy.
• Systemic costs and externalities:
  • Over‑escalation (CEB) imposes real economic costs: excess referrals, tertiary capacity strain, higher per‑case costs, opportunity costs for other patients.
  • Under‑triage (SDB) risks downstream catastrophic costs: increased morbidity/mortality, expensive emergency care, reputational and legal liabilities.
  • Middle‑tier instability (MTI) creates variable downstream demand and forecasting pain for health planners — complicates budgeting and staffing.
• Value of diagnostic benchmarks:
  • Diagnostic metrics (EBI, MCR, DA, EDC) enable formal cost‑benefit modeling, sensitivity analyses, and scenario planning. They should be integrated into business cases, procurement contracts, and reimbursement calculations.
• Deployment design and mitigation economics:
  • Economic value may derive more from hybrid architectures (model + human‑in‑loop, automated flagging for borderline cases) than from selecting a single “best” LLM. The paper shows that prompt engineering and triage-layer design can alter economic viability.
  • Thresholds provided (EDC bands) give actionable cutoffs for when investment in engineering, supervision, or alternative workflows is warranted.
• Policy and regulatory implications:
  • Regulators and funders should require scenario‑based, cost‑sensitive evaluations for clinical AI approvals and procurement in LMICs.
  • Contracts could tie payments or deployment scale to measured EDC or to mitigation measures for identified failure modes.
• Research priorities for AI economics:
  • Incorporate capacity constraints and dynamic feedback (referral cascades) into EDC; model second‑order effects (system congestion, delayed care).
  • Extend cost matrices to include monetary valuations (DALYs, cost per life saved) to compute ROI and willingness‑to‑pay thresholds.
  • Field trials to measure real‑world conversion from predicted triage to actual referrals, resource use, and patient outcomes for economic validation.

Limitations to keep in mind when applying these economic implications: the benchmark covers febrile illness only, uses synthetic vignettes derived from Nigerian PHCs (so transferability to other clinical domains or geographies requires caution), and evaluated a small model set. Despite these limits, the paper provides a practical, reproducible toolkit (open source) to move clinical AI evaluation toward cost‑aware, deployment‑relevant economics.