EcoGym, a new open benchmark for long-horizon economic decision-making, finds that leading LLM agents fail to consistently deliver robust business outcomes: across three simulated marketplaces no model dominates and agents typically sacrifice strategic coherence for execution efficiency (or vice versa), producing substantial performance shortfalls.

Main Finding

EcoGym is an open-source, multi-scenario benchmark for continuous, long-horizon plan-and-execute evaluation of LLM-based agents in economic settings. Across three economically grounded environments (Vending, Freelance, Operation) and eleven leading LLMs, the authors find no single model dominates all tasks. Agents typically fail either on high-level strategic coherence (long-run planning / exploration) or on efficient short-term execution. The benchmark exposes systematic performance variance and stability issues (especially in Vending) and demonstrates that design choices like context window length and “thinking” modes materially affect long-horizon outcomes.

Key Points

• Benchmark design
  • EcoGym focuses on unbounded/very long horizons (effectively 1000+ steps if scaled) with a compact discrete action space to isolate long-term planning ability from action-space complexity.
  • Contains three unified environments that represent commonly encountered economic activities:
    • Vending — retail procurement and dynamic pricing (goal: maximize Net Worth).
    • Freelance — gig-labor balancing, skill accumulation, and wellness (goal: maximize Income while avoiding burnout).
    • Operation — digital platform operator optimizing DAU under decay dynamics (goal: maximize average DAU).
  • Latent mechanics: key environment physics (seasonality, elasticity, burnout thresholds, quality–engagement relationships) are hidden from agents to force exploration, hypothesis testing, and causal discovery.
• Evaluation targets and metrics
  • Outcome-oriented, business-relevant metrics: Net Worth (Vending), Income (Freelance), DAU (Operation).
  • Emphasis on strategic coherence, robustness under partial observability and stochasticity, and long-term cumulative optimization rather than episodic task completion.
• Empirical findings
  • No model dominates across all three environments. Example highlights:
    • Gemini-3 family performed best in Vending and stayed competitive in Freelance.
    • Claude-Sonnet-4.5 ranked highest on Operation in their runs.
    • GPT-5.2 and open-weight models had mixed performance depending on scenario.
  • Vending shows high run-to-run variance; Freelance and Operation are more stable.
  • Diagnostics indicate improvements with longer context windows and with enabling “thinking” / internal reasoning modes in some cases (e.g., DAU gains with larger windows and “thinking on”).
  • Two failure modes: poor long-term strategy (e.g., no exploration, myopic pricing) and poor execution (inefficient or stochastic action outcomes).
• Bench is fully open-source (code link provided), intended as an extensible community testbed.

Data & Methods

• Formalization
  • Interaction formalized as a partially observable decision process ⟨S,A,O,T,G⟩ with discrete action sets (4–5 primitives) but very long interaction horizons.
  • Agents receive structured observations (action responses, daily reports, market/job/ platform views) and must select actions under daily budget constraints.
• Environment specifics (summary)
  • Vending: 600+ SKUs, 37 categories; actions include Market Exploration, Inventory Procurement, Price Adjustment; latent market parameters (seasonality, elasticity).
  • Freelance: assembled from 8 datasets → 5k+ curated tasks across domains (coding, finance, STEM, legal/admin); actions include Exploration (task sourcing), Labor Execution, Settlement, Wellness; hidden burnout threshold.
  • Operation: continuous parameter space (content volume/quality/engagement), single daily action budget, stochastic intervention outcomes (Acquisition Boost, Engagement Tune, Creator Incentive, Moderation Tighten); zero-attractor decay dynamics require continual intervention.
• Models evaluated
  • Eleven LLMs spanning proprietary and open-weights: GPT-5.2, Gemini-3-Pro, Gemini-3-Flash, Claude-Sonnet-4.5, Kimi-k2.6, MiniMax-M2.7, Qwen3.5-397b-A17b, DeepSeek-v3.2, GLM-5, etc.
  • Standardized generation parameters (Temperature=1.0, Top-p=0.95). Default sliding context window: most recent 128 steps; diagnostics explored 32 → 1024.
• Experimental setup
  • Runs: Vending averaged over 5 runs (high variance); Freelance and Operation averaged over 3 runs.
  • Diagnostics: 8 analyses including context window length, memory modules, “thinking on/off”, stochastic stability, failure modes, temporal evolution of behavior, human baselines.
• Data & synthetic mechanics
  • Vending market physics synthesized (seasonality, elasticity) and product catalogs constructed via external sources; Freelance tasks reworked and filtered through LLM + human checks to avoid memorization; Operation system coefficients hidden and stochastic.
• Release
  • Code and environment released: https://github.com/OPPO-PersonalAI/EcoGym

Implications for AI Economics

• Evaluation matters: grounding agent evaluation in economic outcomes (net worth, income, DAU) provides more actionable and deployment-relevant measures of agent competence than generic reward signals or short-horizon benchmarks.
• Specialization vs generality: current LLMs show complementary strengths across economic scenarios, suggesting practical deployments should consider model specialization or ensembles rather than seeking a single universal planner.
• Long-horizon planning is distinct from short-horizon execution: improvements need to target both high-level strategy (exploration, discovery of latent mechanics, risk-sensitive planning) and reliable low-level execution (robust actions under stochastic outcomes).
• Memory, context, and reasoning affordances matter: longer context windows and internal reasoning (“thinking”) can yield substantial gains in long-horizon objectives; investment in persistent memory / episodic modules is likely beneficial for economic agents that must accumulate and exploit long-term knowledge.
• Robustness and variance: high stochastic variance in some economic regimes (e.g., Vending) underscores the need for repeatable evaluation, robust planning under noise, and uncertainty-aware policies in production systems.
• Controllability–utility trade-offs: Operation environment highlights trade-offs (e.g., boosting engagement may degrade quality). Benchmarks like EcoGym can help study social and economic externalities of agent policies (e.g., incentives vs moderation).
• Research directions prompted by EcoGym
  • Mechanisms for explicit causal discovery and experimentation by agents to reveal latent environment physics.
  • Hierarchical planners that separate strategic policy discovery from reliable execution scheduling.
  • Memory systems that retain and summarize long-run dynamics for bounded context windows.
  • Better calibration of stochastic action outcomes and safety/constraint handling to prevent catastrophic long-term drift (bankruptcy, burnout, platform collapse).
  • Economically meaningful human-in-the-loop evaluation and deployment guardrails.
• Policy and deployment caution: results highlight that strong short-term capabilities do not guarantee durable economic performance; practitioners should validate agents in long-horizon, stochastic simulations before real-world economic deployment.

Short summary: EcoGym provides an open, multi-scenario long-horizon economic benchmark that reveals current LLMs’ complementary strengths and systematic weaknesses in sustained planning and execution. It is useful both as a rigorous testbed for research (memory, causal discovery, hierarchical planning) and as a practical tool to stress-test agentic systems before economic deployment.