Pairing large language models with classic operations algorithms makes ordering decisions more robust and profitable; in a controlled experiment, human–AI teams outperformed both standalone AI agents and humans. The paper shows OR-augmented LLMs beat either approach alone across stressed demand scenarios and proves a distribution-free lower bound on how many individuals benefit from AI assistance.

Main Finding

Combining traditional operations-research (OR) inventory heuristics, large language model (LLM) agents, and human judgment produces complementary gains: an OR-augmented LLM pipeline (OR→LLM) outperforms OR or LLM alone across a broad benchmark of inventory problems, and human–AI teams (OR→LLM→Human) outperform both humans and automated systems acting alone. The paper also gives a distribution-free theoretical lower bound showing a substantial fraction of individuals benefit from AI collaboration (empirically 30–60%).

Key Points

• Benchmark and result magnitudes (Gemini 3 Flash reported):
  • InventoryBench: 1,320 instances (720 synthetic, 600 real) capturing changepoints, trends, seasonality, varying cost structures, and three lead-time regimes.
  • Mean normalized reward: OR = 0.445 ± 0.018; LLM = 0.494 ± 0.018; OR→LLM = 0.538 ± 0.016; LLM→OR = 0.501 ± 0.018.
  • OR→LLM yields a ~21% improvement over OR alone.
• Human-in-the-loop experiment (69 participants, pre-registered):
  • Modes: A (OR→Human), B (OR→LLM→Human), C (OR→LLM + Human guidance).
  • Mean normalized reward: Mode A = 0.466 ± 0.025; Mode B = 0.534 ± 0.020; Mode C = 0.464 ± 0.018.
  • Mode B significantly outperforms Modes A and C and yields higher profits than automated OR→LLM without a human (OR→LLM no human = 0.482 ± 0.007).
• Mechanisms of complementarity:
  • LLM strengths: detecting demand regime shifts, incorporating product/world knowledge (e.g., seasonality), identifying supply disruptions (lost orders), flexible context reasoning.
  • OR strengths: mathematical precision for base-stock calculations, disciplined handling of long deterministic lead times, avoidance of overfitting to noise.
  • Human strengths: oversight, catching LLM failures (e.g., false detection of lost orders), and adding judgement beyond automated outputs.
• Individual-level complementarity:
  • The authors formalize a notion of individual complementarity (collaborative outcome exceeds best solo outcome) and prove a distribution-free lower bound on the fraction of individuals who strictly benefit.
  • Empirical lower-bound estimates vary by instance but lie roughly between 30% and 60%.

Data & Methods

• Inventory model:
  • Single-item multi-period inventory with lead times and lost sales, finite horizon T; per-period profit = revenue p·sales − holding cost h·ending inventory; critical fractile ρ = p/(p+h).
  • Initial history: five past demand observations; agents know anticipated lead time L but not realized ℓt which may differ or be infinite (lost orders).
• InventoryBench:
  • 1,320 instances: 720 synthetic (designed to stress test nonstationarity: changepoints, trends, seasonality) and 600 from real demand data.
  • Cost structures varied to test different under/overstock tradeoffs; three lead-time regimes (zero, fixed, stochastic with lost orders).
• Algorithms and LLM agents:
  • OR baseline: capped base-stock policy (data-driven estimation of per-period demand mean & SD, base-stock target using anticipated lead time and critical fractile, capped ordering).
  • LLM-alone: LLM directly issues orders.
  • Two hybrid pipelines:
    • OR→LLM: OR heuristic produces recommendation and structured info; LLM receives that plus contextual prompts and can accept/override.
    • LLM→OR: LLM estimates uncertain parameters (demand, lead-time signals) that feed the OR heuristic.
  • LLM architecture for OR→LLM: fixed system prompt describing mechanics and OR math, a per-period stateless decision loop plus carry-over insights for memory; tested across three frontier models (Gemini 3 Flash, Grok 4.1 Fast, GPT-5 Mini).
• Evaluation:
  • Normalized reward metric reported with 95% confidence intervals across inventory instances and LLMs.
  • Human experiment: web-based inventory game, randomized assignment of mode–instance, each participant played three real-data instances (one per mode). Comparisons against automated baselines are pre-registered and statistically evaluated.
• Theoretical contribution:
  • Theorem giving a distribution-free lower bound on the fraction of individuals who experience strictly positive individual-level complementarity, estimable from cross-sectional outcome distributions even when the same person is not observed in both solo and collaborative conditions.

Implications for AI Economics

• Evidence for complementarity (not substitution): The results show measurable productivity gains when integrating LLMs with domain algorithms and human oversight. This supports economic models where AI augments rather than replaces skilled labor for tasks with rich contextual information and sequential decisions.
• Reallocation of labor and tasks:
  • Operational roles may shift from routine ordering/calculation toward oversight, exception handling, and strategic guidance. Firms should anticipate investments in training workers to interpret LLM reasoning and intervene effectively.
• Design and deployment prescriptions:
  • Hybrid pipelines (OR→LLM→Human) are particularly valuable: maintain structured algorithmic heuristics for precision, use LLMs for context-sensitive inference, and preserve human authority for final decisions.
  • Systems should offer explainable LLM outputs (reasoning and provenance) so humans can validate or override when necessary.
• Welfare and productivity measurement:
  • Benchmarks like InventoryBench provide a practical evaluation standard for measuring real-world gains from AI in operations; reported normalized rewards give a sense of achievable improvements under nonstationarity and supply disruptions.
  • The distribution-free bound for individual complementarity offers a robust tool to evaluate how many workers benefit from AI assistance (important for policy and workforce planning).
• Risks and limitations to incorporate into economic assessments:
  • LLM miscalibration to cost asymmetries (under/overstock tradeoffs) and occasional false detection of regime changes highlight risks of naïvely deploying LLMs without OR scaffolding and human oversight.
  • The study focuses on single-item settings and specific heuristics; scaling to multi-item, networked supply chains, or automated RL-based controllers may change tradeoffs.
  • Real-world deployment raises accountability, auditing, and regulatory concerns (e.g., ordering failures leading to lost sales), so cost–benefit analyses should include risk mitigation costs.
• Directions for research and policy:
  • Evaluate hybrid systems across broader operational settings (multi-item, multi-echelon networks).
  • Study labor-market effects: which worker cohorts gain or lose, retraining needs, and wage implications for oversight roles.
  • Encourage open benchmarks and replication (InventoryBench and the AI-human game are open-sourced) to produce comparable, policy-relevant evidence on AI in operations.