A strategy-execution split stabilises LLM agents on simulated retail operations but fails under real-world-scale complexity: ESE improves operational stability and efficiency on RetailBench, yet performance collapses as non-stationarity and multi-factor interactions rise, implying current LLMs remain fragile for fully autonomous long-horizon commercial management.

Main Finding

The paper introduces RetailBench, a high-fidelity long-horizon benchmark for realistic commercial decision-making under stochastic demand and evolving external conditions, and proposes an Evolving Strategy & Execution (ESE) framework that separates high-level strategic reasoning from low-level action execution. ESE improves operational stability and efficiency over baseline approaches on multiple LLMs and environments, but performance falls off substantially as task complexity and non‑stationarity increase, exposing core limitations of current LLM-based agents for long-horizon, multi-factor decision problems.

Key Points

• Problem addressed: LLM-based agents perform well on short, structured tasks but struggle to maintain coherent, adaptive decision-making over long horizons in realistic, dynamic commercial settings.
• Benchmark: RetailBench — designed to evaluate long-horizon autonomous decision-making with stochastic demand and evolving external conditions (non-stationarity).
• Framework: Evolving Strategy & Execution (ESE)
  • Decomposes agent behavior into (a) high-level strategy that adapts at a slower temporal scale and (b) low-level execution that carries out immediate actions.
  • Enables interpretable and adaptive strategy updates to counteract error accumulation and environmental drift.
• Empirical results:
  • Tested eight state-of-the-art LLMs across progressively harder RetailBench environments.
  • ESE yields improvements in operational stability and efficiency relative to baselines that do not separate strategy from execution.
  • Nevertheless, agent performance degrades markedly as environment complexity, stochasticity, and non-stationarity grow, indicating fundamental limits of current LLMs for such tasks.
• Key failure modes: error accumulation over long horizons, inability to revise strategy adequately under evolving external conditions, sensitivity to multi-factor interactions.

Data & Methods

• RetailBench (benchmark characteristics)
  • High-fidelity simulations of commercial operations (e.g., demand-driven inventory/operations decisions) with realistic stochastic demand processes and shifting external factors.
  • Focus on long-horizon evaluation to capture strategy drift, accumulated errors, and adaptation needs.
  • Environments are progressively challenging to stress-test adaptation and planning capabilities.
• Evolving Strategy & Execution (ESE) design
  • Two-tier architecture: a strategic module that reasons about multi-period objectives and an execution module that implements actions at a finer timescale.
  • Strategy module updates less frequently and is designed to be interpretable and adaptive (strategy evolution mechanisms).
  • Execution module focuses on reliable short-term action selection given the current strategy.
• Experimental setup
  • Eight contemporary LLMs evaluated (state-of-the-art models at time of study).
  • Baselines include monolithic LLM agents and other existing agent architectures without explicit strategy/execution separation.
  • Metrics: measures of operational stability (e.g., variance or frequency of catastrophic failures), efficiency (e.g., cost/profit/fulfillment metrics), and degradation across increasing task complexity.
  • Comparative analysis of how ESE affects robustness to stochastic demand and external condition drift.
• Note: The summary does not report specific numeric results or exact environment/task names beyond the high-level description provided in the paper.

Implications for AI Economics

• Practical adoption
  • Modular strategy/execution architectures (like ESE) can materially improve stability and efficiency of LLM-driven operational decision systems, making them more attractive for deployment in retail, logistics, and supply-chain contexts.
  • Despite improvements, current LLM-based agents are not yet robust enough for fully autonomous long-horizon management in complex, non-stationary commercial environments; human oversight and hybrid systems remain necessary.
• Economic value and risk
  • Potential upside: improved operational efficiency and lower short-term costs where ESE mitigates error accumulation.
  • Risks: performance collapse under increased complexity could lead to sizable operational losses if deployed without safeguards; mis-specified strategies may propagate and amplify economic damage over long horizons.
• Research and policy directions relevant to economics
  • Investment priorities: hierarchical/temporal-decomposition methods, continual learning, robust adaptation to non-stationarity, and causal/structured reasoning to handle multi-factor interactions.
  • Evaluation standardization: realistic, long-horizon benchmarks (like RetailBench) are crucial for economic assessments of autonomous systems and for comparing cost-benefit trade-offs across approaches.
  • Governance and oversight: given fragility in complex environments, regulatory guidelines and auditing procedures should be developed for LLM-based decision agents used in commercially sensitive domains.
• Modeling implications
  • Economic models of automation should account not only for short-run gains but also for long-horizon fragility and the costs of failure modes (error accumulation, strategy drift).
  • When estimating productivity impacts, incorporate the value of strategy interpretability and the ongoing human monitoring cost required to achieve safe deployment.