Large language models can run a simulated small business profitably, with top models capturing roughly 70% of the theoretical maximum profit; yet they often optimize locally, excelling in some operational areas while neglecting others.

Main Finding

LemonadeBench v0.5 is a minimal 30-day simulated lemonade-stand benchmark that jointly tests economic intuition, long-term planning, and decision-making under uncertainty in LLMs. All tested models achieved positive profit, with performance scaling sharply by model sophistication. However, models generally developed specialized, locally optimal strategies (e.g., good pricing or good inventory) rather than a balanced, globally optimal business policy. The best model (o3) captured ~70.7% of the theoretical optimal profit but still left salient blind spots (notably systematic underpricing and remaining stockout risk).

Key Points

• Benchmark goal and setup

  • 30-day simulation, starting cash $1,000, objective maximize final cash.
  • Daily choices: check inventory and prices, order supplies, set price, set operating hours, open for business.
  • Ingredients and shelf lives: cup (30d, $0.05), lemon (7d, $0.20), sugar (60d, $0.10), water (never expires, $0.02).
  • Hourly operating cost: $5. Demand per hour: Q(p,h) = (50 − 10p) · mh · εh (εh ∼ U(0.9,1.1)); mh table encodes realistic hourly foot traffic (sum mh = 13.1 over hours 6–20).

• Theoretical optimum (closed-form)

  • Base cost per cup: $0.37.
  • Profit-maximizing price p* = $2.69.
  • Optimal daily profit ≈ $627.03; 30-day theoretical profit ≈ $18,811 (assumes perfect knowledge and average ingredient prices).

• Models tested

  • Five OpenAI models: gpt-4.1-nano, gpt-4.1-mini, gpt-4.1, o4-mini, o3 (stateless daily calls; full game state provided in persistent instructions; seven tool-like function calls available).

• Aggregate results (selected)

  • Profit range across 30-day runs: $1,191 (gpt-4.1-nano) to $13,301 (o3) — ~11× spread.
  • Models’ average prices ranged: $1.00 (gpt-4.1-mini) to $2.24 (o3) — all undercharged relative to p* = $2.69.
  • Percent of theoretical optimal profit achieved: gpt-4.1-nano 6.3%, gpt-4.1-mini 28.9%, gpt-4.1 57.9%, o4-mini 59.5%, o3 70.7%.

• Efficiency decomposition (six dimensions)

  • Metrics: Purchasing efficiency (timing buys), Expired losses, Excess (ending inventory) losses, Stockout losses (foregone sales), Pricing losses (difference vs p*), Scheduling losses (suboptimal hours).
  • Key failures:
    • gpt-4.1-nano: catastrophic stockout losses (~−$9,004).
    • gpt-4.1-mini: massive pricing losses (~−$8,127) from ultra-low–price, high-volume strategy (11,450 customers at $1.00).
    • o3: achieved best balance, minimizing stockouts (3/30 stockout days) and highest $/customer ($1.67), despite not perfect in any single metric.
  • General pattern: models optimize locally (e.g., procurement timing or volume) but fail to jointly optimize pricing, inventory, and schedule.

• Limitations noted by authors

  • Single run per model (planned expansion to multiple runs for statistical significance).
  • Current formulation heavily weights stockout losses, which can dominate other inefficiencies and bias incentives.
  • Demand function and supply fluctuations are hidden from models and must be learned empirically — choice intended to mirror real-world learning but affects interpretation.

Data & Methods

• Simulation details

  • 30 days, each day: morning check tools (inventory & supply prices), decision to order and set price/hours, open_for_business executes stochastic sales and records day results.
  • Supply prices vary ±10% daily; all purchases delivered instantly; FIFO inventory consumption; daily morning expiration.
  • Hourly demand multiplier mh specified for hours 6–20; hourly εh random noise U(0.9,1.1).
  • Operating cost per hour fixed at $5.

• Interface and experiment design

  • Models interact via seven tool-like function calls (check_inventory, check_morning_prices, get_historical_supply_costs, order_supplies, set_price, set_operating_hours, open_for_business).
  • Stateless conversation design: no preserved LLM chat history, but the full rules and historical performance table are included on every API call (instructions field).
  • Models run with default temperatures and a random seed for each game; full logs captured.

• Evaluation & counterfactuals

  • Authors computed counterfactual metrics to isolate the effect of each decision dimension:
    • Pricing loss: compare actual profit vs profit if price had been p* while holding inventory and hours fixed.
    • Stockout loss: profit lost if inventory were unlimited given actual price/hours.
    • Scheduling loss: profit difference if hours had been optimal (6am–8pm).
    • Purchasing/Expired/Excess: monetary gains/losses from timing purchases, spoilage, and final unsold inventory, computed against base costs.
  • Reported operational table: profit, customers, average price, revenue, hours open, stockout days, $/customer.

Implications for AI Economics

• Capabilities

  • LLMs can demonstrate meaningful economic agency in a simple dynamic market: they can learn policies that produce profit, adapt over time, and exploit some structural regularities (e.g., purchase timing or operating hours).
  • More advanced/reasoning-optimized models (o3) show better holistic trade-off recognition (avoiding catastrophic stockouts), suggesting extended reasoning helps prioritize binding constraints.

• Failure modes and risks

  • Systematic underpricing bias: models consistently charge below analytic optima, translating to substantial foregone profit. In real-world economic tasks, such conservative pricing could reduce value capture.
  • Specialization over generalization: models tend to optimize a subset of dimensions; this can cause large, interpretable failure modes (e.g., ignoring inventory constraints, over-attending to procurement timing). For deployment in economic workflows, unchecked localized strategies could produce fragile or unfair outcomes.
  • Asymmetric loss sensitivity: environments where one mistake (stockouts) dominates will reward fixing that specific failure even if other aspects remain suboptimal. Benchmarks and training that fail to balance loss magnitudes could produce brittle behavior when deployed in different contexts.

• Design and evaluation recommendations

  • Benchmarks should: (a) include multiple runs per model to quantify variability; (b) calibrate loss magnitudes to reflect realistic economic trade-offs so model incentives align with intended objectives; (c) test transfer/generalization across variant environments (different demand shapes, shelf lives, cost volatility).
  • For applied economic tasks, emphasize tools and prompts that help models identify and focus on binding constraints (stockouts, capacity limits) rather than just local improvements.
  • Consider ensembles or hierarchical decision systems: specialized modules (procurement, pricing, scheduling) coordinated by a higher-level policy may reduce locally optimal but globally poor strategies.

• Research directions

  • Scale LemonadeBench to richer markets and multi-product firms; add stochastic shocks and competitor agents to test strategic interaction.
  • Explore continuous (stateful) interaction modes to let models maintain internal plans across days (the current stateless design was chosen for token constraints).
  • Investigate training objectives or RL-style fine-tuning that encourage global optimum-seeking (e.g., reward shaping to penalize systemic stockouts or underpricing).
  • Study human–AI collaboration: can a human manager + model pair outperform either alone by correcting model blind spots (e.g., pricing heuristics)?

Summary takeaway: LemonadeBench shows modern LLMs can run simple profit-making businesses and that reasoning-optimized models better identify binding constraints, but they typically form locally optimal strategies and underprice—highlighting both promise and caution for deploying LLMs in economic decision-making.