Using LLMs as smart interfaces, not solvers, slashes simulated SME inventory costs by about a third; the gain comes because LLMs miscompute stochastic optimization, not because they fail to understand managers.

Main Finding

Using LLMs as natural-language interfaces that orchestrate verified optimization engines—rather than as end-to-end solvers—substantially improves inventory outcomes. The paper’s hybrid agentic system (LLM front-end + rigorous solver back-end) reduces total inventory costs by 32.1% relative to an interactive baseline that uses GPT-4o as an end-to-end decision-maker. The authors formalize this loss as a “hallucination tax”: efficiency lost when policy computation relies on unconstrained language-model reasoning instead of grounded stochastic optimization.

Key Points

• Hallucination tax: End-to-end LLM recommendations are often fluent but systematically suboptimal because LLMs struggle with grounded stochastic reasoning, feasibility guarantees, and consistent probabilistic computation.
• Hybrid agentic architecture: Strict separation of concerns into three agents:
  • Information Extraction Agent: conducts multi-turn dialogue with the user to elicit and disambiguate parameters, maintain an explicit Parameter Specification Table (persistent memory), and resolve contradictions.
  • Optimization Agent: invokes rigorous solvers (classical (s,S) policies, DRL where appropriate) to compute policies from the structured parameter set.
  • Policy Interpretation Agent: translates computed policies back into actionable, understandable instructions for the manager and checks feasibility against stated constraints.
• Human Imitator: a fine-tuned LLM “digital twin” trained on hundreds of real human–machine dialogues to simulate boundedly rational managers (ambiguous, inconsistent, incomplete inputs). This enables scalable, reproducible stress-testing of interactive decision-support workflows.
• Drivers and robustness:
  • The hybrid system’s advantage grows with problem complexity—longer lead times, high-penalty/high-flexibility regimes produce larger gains (a “complexity premium”).
  • The approach is distribution-agnostic (robust across demand shapes).
• Information vs. computation bottleneck: Supplying GPT-4o with perfect ground-truth parameters does not materially improve its performance, implying the dominant bottleneck is computational/cognitive (LLM reasoning limits), not information elicitation or prompt engineering.

Data & Methods

• Evaluation setup:
  • Baselines: (1) interactive end-to-end GPT-4o assistant (used as solver); (2) hybrid agentic pipeline (LLM-based elicitation + solver back-end + interpretation).
  • Counterfactual: “perfect information” experiment where GPT-4o receives ground-truth parameters to disentangle information vs. computational limits.
• Human Imitator:
  • Fine-tuned LLM trained on >100s of recorded human–assistant dialogues to replicate real managerial ambiguities and inconsistencies.
  • Used as a repeatable user simulator to run multi-turn interactions and stress-test the system at scale.
• Optimization toolkit:
  • Optimization Agent selects between established analytic solvers (e.g., (s,S)-style methods) and Deep Reinforcement Learning algorithms depending on problem structure and user preferences (trade-offs between expected cost and variance).
• Information Extraction mechanics:
  • Explicit memory: a persistent Parameter Specification Table (schema examples provided) is maintained and incrementally filled by the Information Extraction Agent to avoid context drift and hallucinated defaults.
  • The Information Extraction Agent was implemented using a smaller model (GPT-5-mini) for cost reasons, while GPT-4o is used as the main LLM baseline elsewhere.
• Metrics:
  • Primary outcome: total inventory cost (holding + ordering + shortage).
  • Secondary analyses: performance by lead time, penalty regimes, demand distribution; statistical tests comparing hybrid vs. GPT-4o baselines and perfect-information counterfactuals.

Implications for AI Economics

• Complementarity, not replacement: LLMs are most valuable as interfaces and orchestrators that democratize access to well-established optimization tools. Economic gains come from unlocking solver value for non-experts, not from replacing OR/DRL engines.
• Large welfare potential for SMEs: A 32.1% reduction in inventory costs is economically meaningful for small and medium retailers that lack analytics teams. Savings concentrate in complex, high-stakes settings (long lead times, high stockout penalties).
• Product design and marketization:
  • Practical decision-support products should adopt modular, auditable architectures: LLM front-ends for elicitation/translation + verified optimization back-ends for computation.
  • Firms should avoid packaging LLMs as black-box solvers for operational decisions without a grounded solver and explicit auditing.
• Evaluation methodology: The Human Imitator provides a scalable template to test interactive decision systems under realistic human imperfections—useful across operational domains beyond inventory (healthcare, finance, logistics).
• Policy and governance:
  • Risk of overreliance: The hallucination tax warns against regulatory or procurement choices that treat LLM outputs as authoritative without verification.
  • Transparency and auditability: Modular systems make it easier to log, audit, and validate the computational component (solvers) separately from the language interface.
• Research directions:
  • Extend to multi-echelon and multi-product systems, richer constraints (cashflow, promotions), and real-world deployment trials.
  • Quantify heterogeneous returns to SME adoption (which firm types benefit most) and do cost–benefit analyses including human-in-the-loop costs.
  • Investigate whether future LLMs with improved probabilistic reasoning reduce the hallucination tax or whether hybridization remains the robust design pattern.

Short takeaway: For operational decisions with stochastic structure, architect LLMs as intelligent interfaces that capture messy human inputs and hand off computation to verified solvers—the largest gains come from combining natural-language accessibility with grounded optimization.