← Papers

Expert-built underwriting benchmark exposes large gaps in enterprise readiness: leading models hallucinate proprietary knowledge, favor speed over accuracy, and see about a 20% hit in multi-attempt pass^k scores, underscoring brittleness in common agentic frameworks.

Benchmarking Agents in Insurance Underwriting Environments

Amanda Dsouza, Ramya Ramakrishnan, Charles Dickens, Bhavishya Pohani, Christopher M Glaze · January 31, 2026

arxiv descriptive medium evidence 7/10 relevance Full text usable extracted full text Source PDF

Structured author observations

Linked only from stored provider relations; the raw author line above is never matched by name.

Arxiv

Latest observation: July 29, 2026

Amanda Dsouza unresolved corpus identity
Ramya Ramakrishnan unresolved corpus identity
Charles Dickens unresolved corpus identity
Bhavishya Pohani unresolved corpus identity
Christopher M Glaze unresolved corpus identity

Semantic Scholar

Latest observation: August 25, 2026

A. Dsouza provider ID
R. Ramakrishnan provider ID
Charles Dickens provider ID
Bhavishya Pohani provider ID
Christopher Glaze provider ID

UNDERWRITE, an expert-designed multi-turn insurance underwriting benchmark, reveals that leading models still hallucinate domain knowledge, trade accuracy for efficiency, and suffer roughly a 20% drop in pass^k multi-attempt performance, exposing gaps between lab benchmarks and enterprise readiness.

Citation observations

Cumulative provider counts captured on specific dates; providers are never combined.

0 cumulative citations

Semantic Scholar · Observed August 25, 2026

View corpus context

As AI agents integrate into enterprise applications, their evaluation demands benchmarks that reflect the complexity of real-world operations. Instead, existing benchmarks overemphasize open-domains such as code, use narrow accuracy metrics, and lack authentic complexity. We present UNDERWRITE, an expert-first, multi-turn insurance underwriting benchmark designed in close collaboration with domain experts to capture real-world enterprise challenges. UNDERWRITE introduces critical realism factors often absent in current benchmarks: proprietary business knowledge, noisy tool interfaces, and imperfect simulated users requiring careful information gathering. Evaluating 13 frontier models, we uncover significant gaps between research lab performance and enterprise readiness: the most accurate models are not the most efficient, models hallucinate domain knowledge despite tool access, and pass^k results show a 20% drop in performance. The results from UNDERWRITE demonstrate that expert involvement in benchmark design is essential for realistic agent evaluation, common agentic frameworks exhibit brittleness that skews performance reporting, and hallucination detection in specialized domains demands compositional approaches. Our work provides insights for developing benchmarks that better align with enterprise deployment requirements.

Summary

Main Finding

UNDERWRITE is an expert-designed, multi-turn insurance-underwriting benchmark that reveals large gaps between frontier model performance in research settings and enterprise readiness. Evaluations of 13 models show wide variability in correctness (30–90%), frequent tool-use and hallucination failures even when tools/data are available, and a roughly 20% drop in success under pass^k (multi-trial) reliability tests. The paper argues expert co-design, compositional hallucination detection, and framework-robust evaluation are essential for realistic assessment of agentic systems in enterprise domains.

Key Points

Purpose and novelty
- UNDERWRITE models an enterprise underwriting backend with proprietary business logic, noisy interfaces, and imperfect users — realism intentionally missing from many open-domain benchmarks.
- Designed iteratively with Chartered Property Casualty Underwriters (CPCUs); task acceptance improved from 55% to 88% after expert feedback.
Environment design highlights
- Multi-turn, tool-augmented tasks requiring 3–7 reasoning/tool-call steps and 10–20 conversational turns.
- Noisy data: nine-table SQLite DB (NAICS variants, “appetite” matrices), metadata, and free-text proprietary guidelines.
- Tools exposed via a Model Context Protocol (MCP) server; overlapping/redundant tool descriptions create realistic tool-selection noise.
- Simulated user = GPT-4.1 underwriter constrained to give ≤2 facts per answer to force active information elicitation.
Evaluation metrics
- Correctness of final outcome (LLM-as-Judge: GPT-4.1-mini; >95% agreement on a 100-trace holdout).
- Tool-use errors (rule-based detection of exceptions).
- “User cannot answer” counts (proxy for irrelevant or poorly targeted agent questions).
- Hallucination detection: conservative domain-focused LM (GPT-4.1) to identify product/domain hallucinations.
Empirical results (13 models, 300 tasks, up to 50 turns)
- Correctness ranged ~30% (Qwen3-235B, GPT OSS 120B) to ~90% (Claude Sonnet 4.5). Representative top results: Claude Sonnet 4.5 = 90.3%, GPT-5 = 83.3%, Grok 4 = 83.3%.
- Tool errors: models produced at least one tool-call error in ~32% of conversations; some top performers still showed nontrivial tool-error rates.
- Tradeoffs: most-accurate models were not the most efficient or least error-prone (e.g., Claude Sonnet 4.5 higher correctness but higher tool-error and uncertain-response rates than Claude Haiku 4.5).
- Hallucinations: smaller model variants often hallucinated non-existent insurance products at high rates (reported 58–66% for some small OpenAI variants on product recommendation tasks); other metrics show up to ~19% of completed traces with hallucinations for GPT-5-mini. Hallucination rates negatively correlate with accuracy.
- Reliability: pass^k analysis (k up to 4) shows roughly a 20% drop in correctness under realistic multi-trial constraints — highlighting brittleness and inconsistency.
- Failure modes: empty/parseable completions, max-step timeouts, and scaffold/framework artifacts were significant; authors invested engineering effort to separate framework brittleness from genuine model failure.
Design takeaways
- Expert-first co-design materially improved realism and task validity.
- Hallucination detection in specialized domains requires domain-specific, compositional approaches for higher precision.
- Agentic scaffolds (e.g., LangGraph/ReAct) introduce brittle failure modes that can distort benchmarking if not mitigated.

Data & Methods

Environment
- UNDERWRITE simulates an internal underwriting copilot interacting with an underwriter across multi-turn dialogues.
- Backend data: SQLite database with 9 tables (including NAICS versions, small-business qualifiers, appetite matrices), metadata descriptions, and proprietary-style underwriting guidelines.
- Tools: read-only SQL query tool, guideline viewer, metadata tools; redundant tools intentionally create decision friction.
- Tool/server infra: Model Context Protocol (MCP) server; agents implemented as function-calling ReAct agents in LangGraph.
Task generation and diversity
- 6 seed task types (in-appetite screening, qualification, policy limits/deductible decisions, etc.) chosen with CPCU experts.
- 3,000 synthetic applicant profiles created with GPT-4.1 constrained by public statistics and NAICS sampling.
- Final evaluation: 300 tasks sampled to cover range of complexities; tasks explicitly require follow-ups and multi-tool reasoning.
Agents, models, and runs
- 13 frontier models evaluated (proprietary and open-source). Notable models: Claude Sonnet 4.5, Claude Haiku 4.5, GPT-5 (and variants: GPT-5-Mini, GPT-5-Nano), Grok 4, Gemini 2.5 Pro/Flash, DeepSeek V3.1, Qwen3-Coder-480B, Qwen3-235B, GPT OSS 120B, Kimi K2 Instruct.
- Each agent allowed up to 50 turns per task; default model parameters used (temperature, token limits, etc.).
- Evaluations recorded final correctness, tool exceptions, number of “I do not know” user responses elicited, incompletions and root causes.
Evaluation and annotation
- LLM-as-Judge: GPT-4.1-mini to compare agent outputs to reference answers (binary reward + rationale), validated to >95% agreement on a held-out sample annotated by domain experts.
- Hallucination detection: conservative specialized prompt with GPT-4.1 to avoid over-labeling correct but unconventional reasoning as hallucination.
- Expert validation: CPCUs rated task realism and traces; simulated underwriter behavior revised until >90% acceptance.

Implications for AI Economics

Deployment readiness vs lab benchmarks
- Accuracy alone is insufficient for procurement decisions. UNDERWRITE shows high-accuracy models can be less efficient or more error-prone in tool use, raising operational costs (operator time, error correction, monitoring).
- Organizations should require benchmarks that reflect noisy data, ambiguous interfaces, and multi-turn interactions before claiming enterprise readiness.
Cost-efficiency and total cost of ownership (TCO)
- Tradeoffs between model price, latency, and behavioral properties matter: cheaper/faster models may produce fewer tool errors and fewer irrelevant user probes, lowering human-in-the-loop costs even if accuracy is modestly lower.
- Pass^k brittleness implies added costs: retries, supervisory review, and fallbacks increase labor and compute consumption. A ~20% drop under pass^k translates to measurable throughput loss and higher per-decision cost.
Risk, liability, and regulation
- Hallucinations of proprietary products or policy recommendations can produce legally and financially material errors in underwriting and pricing. Domain-specific hallucination detection and audit trails are necessary for compliance and risk management.
- Benchmarks like UNDERWRITE can inform regulatory expectations by exposing failure modes that affect consumer harm and mispricing.
Labor dynamics and productivity
- Realistic agent behavior matters for augmentation outcomes: agents that ask poor or irrelevant questions reduce underwriter productivity; agents that make correct but non-explainable recommendations complicate trust and acceptance.
- Employers and unions should factor agent reliability, explainability, and required oversight into projections of labor displacement/augmentation.
Product/market strategy for vendors
- Vendors should present multi-dimensional benchmarks (accuracy, tool robustness, hallucination rates, pass^k stability, human elicitation efficacy) when pitching enterprise deployments.
- There is opportunity value for models/systems optimized for reliable tool orchestration (not just raw LLM accuracy), and for modular hallucination-detection components targeted to specific domains.
Research & procurement recommendations
- Pay for reliability metrics (pass^k, tool-error rates) and domain-grounded hallucination measures, not just top-line accuracy.
- Require expert-in-the-loop benchmark validation for domain-specific procurement.
- Invest in compositional hallucination detection, audit logging of tool calls, and fallbacks to guarded decision logic for high-stakes use cases.

Summary conclusion: UNDERWRITE demonstrates that realistic, expert-designed benchmarks reveal economically important failure modes — inconsistency, hallucination, and tool brittleness — that materially affect deployment costs, risk, and the value of AI agents in enterprise workflows. Procurement, regulation, and R&D should shift focus from single-shot accuracy to robustness, tool competence, and domain-specific hallucination control.

Assessment

Paper Typedescriptive Evidence Strengthmedium — Presents empirical performance evidence from a purpose-built, expert-designed benchmark applied to 13 frontier models, which credibly reveals gaps in agent readiness for enterprise underwriting tasks; however, results are diagnostic for model behavior on this benchmark rather than causal evidence about real-world economic impacts, and external validity is limited by domain scope, simulation components, and the selected model set. Methods Rigormedium — Benchmark design is strengthened by close collaboration with domain experts, multi-turn scenarios, and explicit realism factors (proprietary knowledge, noisy tool interfaces, imperfect users), which improves ecological validity; nevertheless, reliance on simulated users/tools, a finite set of models, and benchmark design choices that may not capture all enterprise heterogeneity constrain methodological rigor. SampleUNDERWRITE consists of expert-authored, multi-turn insurance underwriting scenarios that encode proprietary-business knowledge, noisy tool interfaces, and imperfect simulated users; the authors evaluate 13 frontier LLMs/agentic frameworks on metrics including accuracy, efficiency, hallucination rates, and pass^k performance. Themeshuman_ai_collab adoption GeneralizabilityDomain-specific: focused on insurance underwriting, so findings may not generalize to other enterprise functions (e.g., sales, logistics, healthcare)., Simulation gap: uses imperfect simulated users and noisy tool interfaces which may not fully capture real employee behavior or production system complexity., Model coverage: evaluation limited to 13 selected frontier models and frameworks, not exhaustive of all architectures or smaller models used in practice., Benchmark design choices (e.g., which proprietary knowledge items, noise profiles, and task formulations were included) may shape outcomes and differ across firms., Temporal validity: model performance may change rapidly with new model releases and fine-tuning, limiting longevity of specific numeric results.

Claims (10)

Claim	Direction	Outcome	Confidence & Evidence	Details
Existing benchmarks overemphasize open-domains such as code, use narrow accuracy metrics, and lack authentic complexity. Research Productivity	negative	realism and scope of benchmarks	Reading fidelity high Study strength medium	not reported 0.18
We present UNDERWRITE, an expert-first, multi-turn insurance underwriting benchmark designed in close collaboration with domain experts to capture real-world enterprise challenges. Adoption Rate	positive	benchmark realism and fidelity to enterprise tasks	Reading fidelity high Study strength medium	not reported 0.18
UNDERWRITE introduces critical realism factors often absent in current benchmarks: proprietary business knowledge, noisy tool interfaces, and imperfect simulated users requiring careful information gathering. Adoption Rate	positive	presence of realism factors in benchmark scenarios	Reading fidelity high Study strength medium	not reported 0.18
Evaluating 13 frontier models, we uncover significant gaps between research lab performance and enterprise readiness. Adoption Rate	negative	enterprise readiness / performance gap	Reading fidelity high Study strength medium	n=13 0.18
The most accurate models are not the most efficient. Organizational Efficiency	mixed	accuracy versus efficiency trade-off	Reading fidelity high Study strength medium	n=13 0.18
Models hallucinate domain knowledge despite tool access. Ai Safety And Ethics	negative	hallucination occurrences / domain-knowledge errors	Reading fidelity high Study strength medium	n=13 0.18
Pass^k results show a 20% drop in performance. Output Quality	negative	pass^k performance (task success rate at k attempts)	Reading fidelity high Study strength medium	n=13 20% drop in performance 0.18
Expert involvement in benchmark design is essential for realistic agent evaluation. Governance And Regulation	positive	realism/validity of agent evaluation	Reading fidelity high Study strength speculative	not reported 0.03
Common agentic frameworks exhibit brittleness that skews performance reporting. Organizational Efficiency	negative	robustness/brittleness of agentic frameworks and effect on reported performance	Reading fidelity high Study strength medium	n=13 0.18
Hallucination detection in specialized domains demands compositional approaches. Ai Safety And Ethics	positive	effectiveness of hallucination detection methods	Reading fidelity high Study strength speculative	not reported 0.03