Large language models can reliably extract key actuarial variables from unstructured claims text in a proof-of-concept, earning high expert scores and moderate inter-rater agreement; integrating these extractions into chain-ladder reserving cut reserve-estimation error from 6.5% to 4.0%.

Main Finding

LLMs can be used to convert unstructured claims documents (medical notes, adjuster notes, transcripts) into standardized actuarial variables at scale. A prototype two-stage LLM pipeline applied to synthetic claims data extracted a 36-variable taxonomy (14 core variables validated) with high practical accuracy (expert-scored ≥4/5) and produced demonstrable actuarial value — integrating LLM-derived classifications into severity‑segmented chain‑ladder reserving reduced ultimate estimate error from 6.5% to 4.0% (a 2.5 percentage‑point improvement).

Key Points

• Goal: bridge rich narrative claims information and quantitative actuarial workflows by automatically extracting structured predictors from unstructured text using LLMs.
• Architecture: two-stage processing
  • Stage 1: document‑level extraction (per document)
  • Stage 2: claim‑level synthesis (aggregate per claim, maintain claim IDs and timestamps)
• Prototype pipeline: four Python scripts, JSON output schemas for integration with actuarial systems.
• Data: synthetic claim ecosystems generated from Synthea FHIR bundles plus LLM‑generated nonmedical documents (adjuster notes, call transcripts, settlement docs) to create consistent, testable claims with known ground truth.
• Variable taxonomy: 36 actuarial variables defined (categorical, numeric, Boolean). Validation focused on 14 core variables including claim/injury severity, medical complexity, treatment type, expected development, ultimate cost, litigation risk, settlement likelihood, management complexity.
• Validation:
  • Two independent clinician reviewers used a taxonomy + Likert rubric (1–5) to score extractions.
  • Extraction accuracy across variable categories averaged at least 4/5.
  • Confidence scores were well calibrated; interrater reliability was moderate.
• Hallucination mitigation and quality controls:
  • Controlled vocabularies / categorical constraints (with “unknown” option)
  • Mandatory rationale fields citing document evidence
  • Multilevel confidence scoring (document/variable/overall)
  • Actuarial consistency checks (mathematical relationships)
• Actuarial integration: LLM‑extracted classifications used for severity‑segmented chain‑ladder analysis and produced measurable reserve estimation improvement.
• Artefacts released: demonstration code, JSON schemas, and synthetic test datasets to enable reproducibility and extension.
• Limitations noted: reliance on synthetic data for development, remaining risks of coherent-but-incorrect LLM outputs, need for production-grade calibration, OCR and provider documentation variability when moving to real data, and regulatory/governance considerations.

Data & Methods

• Synthetic data generation:
  • Synthea configured to produce FHIR R4 patient bundles representing realistic clinical trajectories for injury claims (e.g., worker’s compensation scenarios).
  • Script 1 (FHIR Bundle Processor): transforms structured FHIR elements into narrative clinical notes using an LLM to emulate real medical text.
  • Script 2 (Document Augmentation Engine): generates nonmedical documents (adjuster notes, phone transcripts, claimant statements, settlement notes) using an ~800+ character master profile created by an LLM to ensure cross-document factual consistency (same claim numbers, dates, clinical facts).
• Pipeline & schema:
  • Modular four-script Python implementation producing standardized JSON outputs keyed by claim ID and document timestamps to permit time-aware augmentation of actuarial datasets.
• Extraction approach:
  • Two-stage LLM workflow: per-document extraction then claim-level aggregation/synthesis.
  • 36-variable taxonomy defined; prototype focuses on 14 high-value variables.
  • Outputs include categorical values, numeric fields, and mandatory textual rationale for auditability.
• Validation:
  • Ground truth is known due to synthetic generation.
  • Two independent clinician experts reviewed extracted variables and rated correctness using a Likert 1–5 rubric linked to taxonomy.
  • Performance metrics: average accuracy ≥4/5; confidence calibration and moderate interrater reliability reported.
• Actuarial testing:
  • Severity-segmented reserve triangles built using LLM classifications feeding into chain‑ladder reserving.
  • Comparison to baseline showed reduction of ultimate estimate error from 6.5% to 4.0%.

Implications for AI Economics

• Improved reserving and capital accuracy:
  • More granular, text‑driven segmentation can reduce reserve estimation error and therefore lower capital misallocation and pricing uncertainty.
  • Example: a 2.5 percentage‑point reduction in ultimate estimate error suggests measurable balance‑sheet and solvency implications for insurers using LLM‑derived features.
• Better risk pricing and segmentation:
  • Extracted indicators (medical complexity, litigation risk, causation type, provider quality, etc.) enable finer risk classification, potentially improving rate adequacy and experience rating.
• Operational efficiency and scalability:
  • Automation reduces reliance on manual reviewers, enabling broader coverage of documents, faster triage (identifying high‑complexity or litigation‑prone claims earlier), and potential cost savings in claims handling.
• Product and portfolio management effects:
  • Earlier detection of claim trajectories supports targeted interventions, settlement strategies, and reinsurance/retention decisions informed by richer text-derived signals.
• Labor and task-shift considerations:
  • Potential reduction in routine manual review jobs but increased demand for model validators, domain experts for auditing, and governance roles.
• Model risk, governance, and regulatory constraints:
  • Deployments must address LLM hallucination risk, explainability, audit trails, calibration, fairness, and compliance (e.g., NAIC guidance). Mandatory rationale fields and confidence scores help but are not sufficient — ongoing empirical validation and monitoring are required.
• Research and market effects:
  • Open synthetic datasets and reproducible code lower barriers to adoption and benchmarking, accelerating innovation but also raising competitive pressure to adopt text-enabled actuarial analytics.
• Practical next steps before production adoption:
  • Phased validation on real claims with human-in-the-loop auditing.
  • Calibrate confidence scores against empirical error rates; implement anomaly detection and automated escalation.
  • Integrate outputs into actuarial software and governance frameworks to meet regulatory and professional standards.

Summary takeaway: This study demonstrates a practical, reproducible approach showing that LLMs can reliably extract high‑value actuarial predictors from unstructured claims texts and deliver tangible improvements in reserving accuracy — but production adoption requires robust calibration, governance, and staged validation on real-world claims.