CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies

As LLM agents become capable of increasingly long-horizon tasks, evaluating their performance in economic systems is becoming increasingly important. Unlike existing benchmarks that primarily evaluate a single agent interacting with a passive environment, economic systems are inherently multi-agent, requiring autonomous agents to communicate, negotiate, and transact while pursuing their own objectives over extended periods. We introduce CoffeeBench, a benchmark for evaluating LLM agents in a long-horizon multi-agent economy composed of heterogeneous firms. In CoffeeBench, two farmers, two roasters, and two retailers autonomously operate their businesses over a 90-day simulation, each seeking to maximize cumulative net income through communication and transactions while managing cash, inventory, and pricing. The evaluated model controls one coffee roaster, while the remaining firms are controlled by fixed reference agents. Across several recent open-weight and proprietary LLMs, all models outperform a passive baseline that takes no actions, with most achieving positive net income. Analysis of agent behavior reveals substantial differences in long-horizon economic interaction: higher-performing models communicate more actively with other firms, whereas Claude~Haiku~4.5 exhibits an idle-drift failure mode, repeatedly choosing inaction despite producing coherent assessments and plans. We release our code and agent trajectories to support future research.

Summary

Main Finding

CoffeeBench is a new open benchmark that evaluates LLM-driven agents in a long-horizon, heterogeneous multi-agent economy (a six-firm coffee supply chain run for 90 simulated days). Frontier LLMs can operate profitably in this setting, but performance differs substantially across models and is strongly tied to sustained inter-agent activity (messaging, trading, repeated tool use). The benchmark exposes distinct long-horizon failure modes (notably an “idle-drift” mode) and provides code plus full agent trajectories for follow-up research.

Key Points

Benchmark design
- Six autonomous firms: 2 farmers, 2 roasters, 2 retailers; one evaluated model controls roaster_A, the other five are background agents.
- Two product segments: commodity and specialty coffee; includes vertical dependencies and horizontal competition.
- Event-driven 09:00–19:00 daily business window; each proactive tool call advances local time by 30 minutes; agents can be reactivated within a day by incoming events.
- Key economic constraints: net-30 trade credit, 0.5%/day spoilage, storage caps, fixed OpEx ($25/$30/$50 per day by role), stochastic logistics and late-payment penalties.
- Objective (KPI): maximize cumulative net income over the horizon (Revenue − COGS − OpEx − InterestExp + InterestRev).
Experimental protocol and artifacts
- 90-day simulations; runs terminate on bankruptcy (negative cash).
- Evaluated agent: roaster_A; background agents by default use Claude Sonnet 4.6.
- Agents implemented with ReAct; shared tools (post_listing, make_offer, accept_offer, send_message, pay_invoice, wait_for_next_day) plus role-specific tools (produce_item, roast, set_retail_price, view_consumer_sales).
- Context summarization: once history >160k tokens, intermediate portion summarized while preserving initial trajectory and most recent 20 steps.
- Released artifacts: code (github.com/SakanaAI/CoffeeBench) and trajectories (pub.sakana.ai/coffeebench/trajectories).
Main quantitative results (means ± std over 3 runs; roaster_A role)
- Top net income: GPT-5.5 +$3,109 ±1,123; Claude Opus 4.7 +$2,782 ±2,263; Claude Sonnet 4.6 +$2,236 ±1,489.
- Mid/low: Gemini 3.1 Pro +$1,695 ±508; GLM-5.1 +$1,597 ±1,199; Kimi K2.6 +$454 ±1,420.
- Failure/baselines: Claude Haiku 4.5 −$630 ±1,745 (idle-drift), HeuristicRoaster −$1,931 ±429, PassiveRoaster −$2,765.
- Behavioral correlates: higher-performing models made many more send_message() calls (e.g., GPT-5.5 ~140 ±22 messages) and sustained daily tool activity (~1,269 ±193 calls). Claude Haiku averaged ~40 idle days (wait_for_next_day() only) and ~52 messages, producing stagnating revenue and deals.
- GLM-5.1 produced highest revenue in aggregate but did not convert revenue into comparable net income (weaker profitability).
Operational costs and compute
- A full run across all six firms costs ≈ $250 in API spend and takes ≈ 8 hours wall-clock; evaluated agent itself contributed per-run API cost shown in paper (e.g., GPT-5.5 ≈ $79/run).

Data & Methods

Simulation environment
- Time: event-driven daily windows; tool calls advance local time by 30 minutes; overnight system updates apply sales, costs, spoilage.
- Marketplace: peer-to-peer marketplace, one-day delivery lag, invoices with net-30 terms, optional delays/loss in logistics.
- Consumer demand: daily competitive demand model influenced by retail prices and brand loyalty; includes a pre-scheduled demand surge (days 40–53).
- Economic shocks/stochasticity: delivery delays/loss, random seed variability; runs are stochastic and highly variable—authors average across 3 runs.
Agent architecture & training
- Agents use the ReAct loop (reason/act interleaving) with role-specific system prompts and tool access. No additional RL fine-tuning; agents are prompt-driven LLMs using deterministic or near-deterministic API settings (reasoning minimized/disabled to control latency/cost).
- Background agents used a single reference model (Claude Sonnet 4.6) to hold the multi-agent environment stable while varying the focal model.
Metrics & logging
- Primary metric: cumulative net income (as defined in the paper).
- Secondary metrics: cumulative revenue, number of tool calls, idle days (days with only wait_for_next_day()), number of send_message() calls, API cost.
- Trajectories: full step-level logs and trajectories are published for replication and deeper analysis.
Models evaluated
- Closed/proprietary: GPT-5.5, Claude Opus 4.7, Claude Sonnet 4.6, Claude Haiku 4.5, Gemini 3.1 Pro.
- Open-weight: GLM-5.1, Kimi K2.6.
- Baselines: PassiveRoaster (do-nothing), HeuristicRoaster (rule-based inventory/pricing).

Implications for AI Economics

Multi-agent economic evaluation matters
- Real economic settings are interactive and heterogeneous; single-agent benchmarks miss essential dynamics (credit risk, counterparty reliability, communication/negotiation).
- Benchmarks like CoffeeBench permit study of endogenous market phenomena (liquidity constraints, bankruptcy contagion, supply-chain coordination).
Communication and sustained activity are crucial
- Empirical link between messaging/trading activity and profitability suggests communication is an operationally critical capability for deployed economic agents.
- Designing LLM agents for production markets should prioritize robust dialogue/trade strategies and event-driven reactivity.
Long-horizon failure modes and alignment risks
- Idle-drift: Agents can maintain coherent internal reasoning traces yet repeatedly choose inaction, producing poor economic outcomes. This indicates standard prompting and chain-of-thought coherence do not guarantee useful external behavior over long horizons.
- Reward-specification and monitoring: KPI design (net income aggregated over time with realistic accounting of COGS/OpEx/interest/bad debt) is necessary to avoid trivial reward hacking (e.g., self-trading at marked-up prices was prevented in the benchmark).
- Potential for emergent undesired behaviors (collusion, strategic default, predatory pricing) can be studied in controlled settings before deployment.
Market design, regulation, and robustness
- Benchmarks enable testing of market rules that mitigate systemic risk (payment terms, penalties, credit limits, delivery reliability). Designers can simulate policy levers (shorter/longer credit, inventory constraints, bankruptcy costs).
- Monitoring and contingency: runtime detection of idle-drift or other pathologies is important for safe deployment; fallback heuristics or supervisory agents may be required.
Practical deployment considerations
- API latency, context management (summarization strategy), and cost are non-trivial for long-horizon multi-agent deployments; these operational limits affect agent design choices (frequency of tool calls, memory retention).
- Model selection tradeoffs: higher messaging and activity usually increase API cost but can materially improve economic returns—cost/benefit analysis is needed for any real deployment.
Research directions enabled by CoffeeBench
- Study of collusion, implicit coordination, or cartel-like behaviors in LLM agents.
- Systematic evaluation of incentive designs, contract structures, and market rules.
- Robustness tests: adversarial counterparties, heterogeneous background agents, longer horizons, endogenous consumer models, richer financial instruments.
- Methods to detect and correct idle-drift and other long-horizon pathologies (meta-supervision, regret-based learning, constrained action priors).

References and artifacts - Paper (arXiv): arXiv:2606.16613 - Code: github.com/SakanaAI/CoffeeBench - Trajectories: pub.sakana.ai/coffeebench/trajectories

If you want, I can: - Extract the per-model full metric table into a compact CSV or JSON for analysis. - Produce suggested experimental extensions (e.g., varying credit terms, introducing adversarial agents) with expected hypotheses. - Highlight concrete monitoring signals and thresholds one could deploy to detect idle-drift in real systems.

Assessment

Paper Typedescriptive Evidence Strengthlow — Findings are based on synthetic multi-agent simulations rather than real-world data or causal identification; results show what agents do in a particular simulated environment but do not establish causal effects in real economies or under alternative market designs. Methods Rigormedium — The benchmark uses a clear, repeatable 90-day multi-agent simulation with multiple LLMs, baseline comparisons, and open-sourced code and trajectories, but it relies on a small, stylized six‑firm economy, fixed reference agents, and simulation rules that may influence outcomes; sensitivity analyses and robustness to alternative agent designs or market parameters are not described in detail in the abstract. SampleA 90-day simulated multi-agent coffee economy with six heterogeneous firms (2 farmers, 2 roasters, 2 retailers); the evaluated model controls one roaster while the other five firms are run by fixed reference agents; metrics include cumulative net income, cash, inventory, pricing, communications, and transactions; several recent open-weight and proprietary LLMs are evaluated against a passive no-action baseline; code and agent trajectories are released. Themesproductivity org_design GeneralizabilitySmall, highly stylized six‑firm simulation may not scale to real-world market size or complexity, Outcomes depend on simulation rules, market structure and fixed reference agent behaviors, No human agents or real-world frictions (regulation, supply shocks, heterogeneous demand) included, Limited set of LLMs tested; proprietary models and prompts/settings may not be representative, Short-run 90-day horizon may not capture longer-term strategic adaptation or market entry/exit

Claims (9)

Claim	Direction	Outcome	Confidence & Evidence	Details
We introduce CoffeeBench, a benchmark for evaluating LLM agents in a long-horizon multi-agent economy composed of heterogeneous firms. Research Productivity	positive	benchmark availability / capability to evaluate agents	Reading fidelity high Study strength medium	not reported 0.18
CoffeeBench simulates an economy of two farmers, two roasters, and two retailers operating autonomously over a 90-day simulation. Research Productivity	null_result	environment structure (number and types of agents and simulation horizon)	Reading fidelity high Study strength high	not reported 0.3
Each firm in CoffeeBench seeks to maximize cumulative net income through communication and transactions while managing cash, inventory, and pricing. Firm Revenue	null_result	cumulative net income	Reading fidelity high Study strength high	not reported 0.3
In evaluation runs, the evaluated model controls one coffee roaster while the remaining firms are controlled by fixed reference agents. Research Productivity	null_result	agent-control configuration	Reading fidelity high Study strength high	not reported 0.3
Across several recent open-weight and proprietary LLMs, all models outperform a passive baseline that takes no actions. Firm Revenue	positive	performance relative to passive baseline (net income / cumulative reward)	Reading fidelity high Study strength medium	not reported 0.18
Most evaluated models achieve positive net income in the CoffeeBench simulation. Firm Revenue	positive	net income (cumulative)	Reading fidelity high Study strength medium	not reported 0.18
Analysis of agent behavior reveals that higher-performing models communicate more actively with other firms. Team Performance	positive	communication activity correlated with performance (net income)	Reading fidelity medium Study strength medium	not reported 0.11
Claude Haiku 4.5 exhibits an idle-drift failure mode, repeatedly choosing inaction despite producing coherent assessments and plans. Error Rate	negative	action frequency / inaction behavior (idle-drift) and coherence of assessments/plans	Reading fidelity high Study strength medium	not reported 0.18
The authors release their code and agent trajectories to support future research. Research Productivity	positive	availability of code and trajectories	Reading fidelity high Study strength high	not reported 0.3

A 90-day simulated coffee economy shows modern LLM agents can run a firm profitably in a controlled multi‑agent market, with active communication linked to higher earnings; however, some models repeatedly choose inaction despite plausible plans.