An AI conversational assistant for independent insurance agents retrieves policies with 93% accuracy and cuts average search time by about 2.4 seconds, suggesting faster workflows; however, the evidence is limited to system-level tests without randomized field validation of broader productivity or economic impacts.

Main Finding

Axlerod, an LLM-based agent-assistive chatbot built for independent insurance agents, can reliably retrieve and summarize policy information from large, live insurance data sources. In internal evaluation it achieved 93.18% overall accuracy on structured policy-retrieval tasks and reduced average information-search time per task by 2.42 seconds (from 7.55 s to 5.13 s). Per-answer inference cost during testing was estimated at about $0.0075.

Key Points

• Purpose: Agent-facing LLM assistant (not consumer-facing) designed to augment independent insurance agents by automating policy lookup, document retrieval, and answering coverage/billing questions.
• Architecture:
  • LLM: Google Gemini 2.5 Pro (previously tested with Llama 3.1 80B).
  • Middleware & microframework: Smoltalk (custom) + LiteLLM gateway.
  • Interface: FastAPI web API; prototype embedded via Tampermonkey/Open WebUI.
  • Retrieval/tools exposed to LLM: policy detail, policy search (Typesense), documentation search.
  • Approach: Retrieval-augmented generation (RAG) + structured tool calls for deterministic data access.
• Data scale:
  • Policy database ≈ 730,000 active personal and commercial policies (policy metadata, billing, coverages, claims).
  • Documentation corpus ≈ 400 MB (company procedures and state/type specific docs).
• Performance breakdown:
  • Overall accuracy: 93.18% (1,118 / 1,200 queries across four task types).
  • Policy-number identification: 80.7% (errors often due to ambiguous/common names).
  • AutoPay enrollment detection: 99.0%
  • Covered vehicles retrieval (auto policy): 93.7%
  • Bill plan identification: 99.3%
• Time savings:
  • Average search time without Axlerod: 7.55 s
  • Average with Axlerod: 5.13 s
  • Mean improvement: 2.42 s per query; larger gains for complex multi-step searches (e.g., name → policy).
• Cost: estimated ~$0.0075 per answer based on LiteLLM reporting during tests.
• Evaluation methods: controlled internal testing (scripted + ad-hoc queries), automated unit tests against ground truth, and small user time trials.

Data & Methods

• Data sources:
  • Large live policy database (730k records): policy IDs, names, addresses, effective/termination dates, billing plans, payment amounts, coverage lists, claims metadata.
  • Documentation database (∼400 MB): unstructured text for policy procedures, state-specific rules, manuals.
• System design:
  • RAG + tool-call pattern: LLM receives tool metadata and either returns a direct answer or issues a structured call to fetch authoritative data via the middleware.
  • Search engine: Typesense for textual/policy search over records.
  • Smoltalk: custom microframework that exposes Python tool functions as LLM-callable APIs, keeping the tool orchestration lightweight and modular.
• Models & infra:
  • Primary inference on Google Vertex AI (Gemini 2.5 Pro) accessed through LiteLLM; earlier experiments with Llama 3.1 when running self-hosted.
• Experimental protocol:
  • Accuracy: 1,200 queries covering four task types with multiple phrasing variants to test intent recognition and robustness.
  • Time-trial: Two users each completed four tasks with two policies each; measured time to locate requested info with and without Axlerod.
  • Cost: runtime API/inference costs logged via LiteLLM and averaged per answer.
• Limitations in methods:
  • Tests performed in an internal/controlled environment (limited real-world deployment data).
  • Ambiguities (e.g., common names) remain a primary error source; system prompts for disambiguation when >5 matches.
  • Small-scale human time trials (two users) limit generalizability of measured time savings.

Implications for AI Economics

• Productivity gains and ROI:
  • Even modest per-query time savings (≈2.42 s) scale across large agent workforces and high call/interaction volumes, implying meaningful labor-efficiency gains and potential increases in agent throughput and carrier profitability.
  • Low per-response inference cost (~$0.0075) suggests favorable marginal economics for deployment, though total ROI depends on integration, maintenance, and compliance costs.
• Labor market effects:
  • Agent-assistive systems are positioned as augmentation (not replacement). They can shift agent tasks toward higher-value, relationship-driven activities (advice, sales)—potentially increasing value per agent rather than reducing headcount in the near term.
  • Over time, increased productivity could change demand for agent labor, training needs, and wage composition (more emphasis on interpersonal and regulatory skills).
• Value capture and supplier dynamics:
  • Use of proprietary cloud-hosted LLMs (e.g., Gemini on Google Cloud) channels recurring costs and bargaining power to large model/cloud providers—creating potential vendor lock-in and concentration of margins.
  • Open-weight/self-hosted models (e.g., earlier Llama integration) can reduce per-query vendor costs but may increase engineering/ops expenses.
• Risk, regulation, and compliance economics:
  • Agent-facing deployment reduces some hallucination risk (human-in-the-loop validation), but residual error rates (≈6.8% incorrect responses in this study) imply non-trivial operational risk that carries economic costs (error-correction, audits, potential regulatory liability).
  • Compliance and monitoring requirements (audit trails, explainability, bias mitigation) will add recurring costs and constrain deployment strategies—raising the effective marginal cost of safe operation relative to raw inference cost.
• Market adoption and competition:
  • Early movers (insurers or insurtech vendors) that successfully integrate agent-assistive LLMs may obtain productivity-based competitive advantages; incumbents with large policy databases and integration capabilities have an edge.
  • Small agencies can access similar tooling via SaaS offerings, potentially compressing margins for traditional agents unless they leverage differentiated service (human advisory).
• Externalities and data economics:
  • Improvements require high-quality, well-integrated internal data. Firms with clean, centralized data can realize greater benefits—creating data-driven advantages and raising the economic value of data-cleaning investments.
  • Privacy, data-sharing constraints, and cross-carrier data fragmentation limit larger network effects; investments to unify/standardize data could be high-return for the sector.
• Measurement and policy implications:
  • Economic value should be quantified beyond per-query time saved—measure downstream effects (conversion rates, retention, error/call-escalation reductions).
  • Regulators may require validation standards and reporting for agent-assistive LLMs; compliance costs will factor into economic feasibility and competitive dynamics.

Summary conclusion: Axlerod demonstrates that a tool-oriented RAG approach using current LLMs can materially assist insurance agents by improving retrieval accuracy and modestly reducing task times at low per-query cost. The economics favor pilot and scaled deployment where data quality, governance, and human oversight are in place, but real-world ROI will hinge on integration costs, compliance overhead, and how productivity gains affect labor and market structure.