1 cumulative citations
View corpus contextLLM agents struggle to manage simulated online stores over a year: in MerchantBench’s 365‑day, order-level simulation the top-performing configuration reached only around 27% of the mean final net assets of human operators, exposing large shortfalls in long‑term coherence and business decision-making.
MerchantBench: Benchmarking LLM Agents for Long-Term Coherence in E-Commerce Operations
MerchantBench presents a 365-day, order-level e-commerce simulator grounded in 98,843 real product records to evaluate long-term coherence of LLM agents and finds the best LLM configuration achieves only about 27% of the mean final net assets attained by human participants.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Large language model agents are increasingly evaluated as autonomous tool users, yet most benchmarks focus on bounded tasks with immediate success criteria. Real-world deployments often require Long-Term Coherence, the capacity to preserve purposeful behavior across extended horizons while adapting decisions to accumulated evidence. Evaluating this capacity requires a persistent environment in which actions constrain future choices, feedback arrives at heterogeneous delays, and incoherent behavior produces measurable cumulative effects. Seller-side e-commerce provides a suitable setting for this evaluation through recurrent and interdependent decisions over Product Sourcing, Listing and Pricing Control, Cash-Flow Management, and Mixed-Latency Feedback Adaptation. We introduce MerchantBench, a 365-day order-level simulation grounded in 98,843 real e-commerce product records and equipped with 26 tools for agent interaction. MerchantBench couples promptly observable Upstream Supplier Events with delayed Downstream Order Outcomes, requiring agents to follow individual order lifecycles and revisit earlier decisions. We evaluate eight LLMs under two agent frameworks in 48 runs, each spanning 365 simulated days. Our results reveal a substantial gap between even the latest LLMs and human participants, with the best LLM configuration attaining only 27.3\% of the mean final net assets achieved by human participants.
Summary
Main Finding
MerchantBench is a new, data-grounded 365-day simulation benchmark for evaluating Long-Term Coherence of LLM-based merchant agents in seller-side e‑commerce. It shows that current LLM agents (tested across 8 modern models and two agent frameworks) fall substantially short of human performance on sustained merchant operation: the best model configuration achieves only ~27.3% of the mean final net assets of human participants. Major failure modes include poor long-horizon credit assignment for delayed order outcomes, weak portfolio revision, and fragile cash-flow management.
Key Points
- Purpose: Measure an agent’s ability to preserve purposeful behavior and update decisions as delayed evidence accumulates across long horizons (operational + strategic coherence).
- Environment highlights:
- 365 simulated days with hourly steps; agents act every 12 hours (decision windows).
- Partial observability: latent demand profiles, future supplier events and order outcomes are unobserved until realized.
- Mixed-latency feedback: upstream supplier events (price change, delisting, shipment delay) appear promptly, while downstream order outcomes (cancellations, stockouts, late shipment, returns/refunds, bad reviews) are delayed and can retroactively affect store rating and demand.
- Agent interface: 26 merchant tools (product sourcing, catalog/supplier queries, listing, pricing, finance views, order monitoring).
- Data grounding: Catalog of 98,843 real product records and supplier signals from 1688 (Chinese wholesale marketplace), covering 365 days (June 1, 2025 – May 31, 2026) across 10 product categories.
- Agents evaluated:
- 8 LLMs (GPT-5.6 Sol; Claude Opus 4.8; Qwen3.7-Max/Plus; GLM-5.2; DeepSeek-V4-Pro/Flash; Kimi K2.6).
- Two agent frameworks: ReAct (minimal controller) and Hermes (more capabilities: planning, memory, code execution).
- Baselines: rule-based agent and 3 human participants (no prior seller experience).
- Metrics:
- Business performance: final net assets (terminal realized net assets), GMV, net profit margin, orders.
- Store reliability: total fines, average store rating, order anomaly rate.
- Long-horizon activity: average active listings, Sustained Window Rate (SWR; decision window activity), total tool calls.
- Experimental design: 48 LLM runs (8 models × 2 frameworks × 3 seeds), plus rule-based and human runs. Each run starts with RMB 2,000 cash, RMB 1,000 deposit, capacity for 50 active listings.
- Main quantitative results (high-level):
- Best LLM: Qwen3.7-Max under Hermes — final net assets ≈ 59.46k RMB (≈ 27.3% of human mean 217.61k RMB).
- Hermes vs ReAct: Hermes yields on average +53.3% final net assets, +71.5% GMV, and +71.2% more orders versus ReAct, but gains depend on the model.
- Humans outperform all LLMs by large margins in net assets, GMV, and order volume; human SWR = 100% (consistent activity).
- Stability varies: some configurations were highly variable across seeds (e.g., Qwen3.7-Max under Hermes had high CV).
- Observed agent behaviors and failure modes:
- Some agents (GPT-5.6 Sol, Kimi K2.6) did product-level attribution and removal of risky listings; others (DeepSeek-V4-Pro) failed to act after refunds.
- Store-level adaptation included repricing safe products to recover rating-driven demand, but agents often over- or under-reacted.
- Common failures: poor credit assignment from delayed outcomes to earlier sourcing/listing choices, insufficient replacement/search for demand-preserving alternatives, fragile cash commitments leading to resource exhaustion, short-lived monitoring causing missed late-arriving anomalies.
Data & Methods
- Formalization: merchant operation as a POMDP M = ⟨S, A, P, O, Z, R, µ0, Hc⟩ with Hc = 8,760 hourly steps (agent acts every 12 steps). Objective J(π) = Eπ[R(sT)] where R(sT) = terminal net assets (cash + deposit + in-transit + receivables).
- Real data:
- Catalog: 98,843 products from 36,576 suppliers (filtered for completeness).
- Daily market reports (365) aligned to dates to support sourcing signals.
- Demand: real product-level daily demand traces converted to hourly Poisson order arrivals, with intensity λm,i,t = Di,d(t) · wc(category, hour) · rm,t · ℓm,i,t · (pm,i,t/pref_i)^{-ϵ_i}, incorporating elasticity, store rating, and listing exposure/cold-start effects.
- Upstream supplier simulation:
- Product-level events sampled from calibrated probabilities: Price Change, Product Delisting, Shipment Delay; each event has a sampled recovery time.
- Inventory replenishment and supplier availability dynamics included.
- Downstream order simulation:
- Order lifecycle: Placed → Procured (immediate procurement for dropshipping) → Shipped → Delivered → Settled (with settlement delay).
- Abnormal outcomes: Cancellation (refund procurement cost), Stockout (prevents procurement), Late Shipment (missed dispatch, potentially fine), Returnless Refund (lose receivable), Return and Refund (lose receivable but procurement cost restored), Bad Review (preserve revenue but lowers rating); fines and rating penalties impact future demand.
- Agent tools: 26 named tools exposing merchant-visible information (catalog search, supplier details, order state views, finance, listings, etc.).
- Experiment specifics:
- Three repeated seeds per LLM × framework pairing.
- Comparisons against a deterministic rule-based agent and human participants.
- Evaluation metrics computed as means over runs; variability reported.
Implications for AI Economics
- For researchers and benchmarkers:
- Long-horizon, mixed-latency, data-grounded simulations (like MerchantBench) are essential to evaluate economically consequential agent properties (credit assignment, portfolio management, cash constraints, reputational externalities).
- Improvements needed in models' long-memory, causal attribution, counterfactual reasoning, risk-aware planning, and sustained policy consistency; richer agent architectures (e.g., memory systems, explicit bookkeeping, causal tracing) materially improve outcomes (Hermes > ReAct).
- Benchmarks should report stability/variance across seeds, not just mean performance — some high-performing configs were unstable.
- For practitioners deploying LLM agents in economic roles:
- Current off-the-shelf LLM agents can produce plausible short-term actions but leave substantial economic downside risk when managing persistent stores with delayed feedback. Firms should require conservative guardrails, human-in-the-loop oversight, and pre-deployment simulation testing.
- Key operational safeguards: strict cash/deposit constraints, monitoring metrics (SWR, tool-call regularity, rolling-window performance tests), anomaly alarms linking delayed outcomes back to sourcing decisions, and automated conservative fallbacks for procurement decisions.
- For platform and market design:
- Platforms that permit automated agent activity should consider mechanisms to limit systemic risk (e.g., rate limits, escrow rules, automated dispute handling) because agent failures can cascade via store ratings and market perception.
- Transparency about delayed feedback sources (e.g., exposing structured post-sale outcome signals sooner) could materially improve agents’ credit assignment and reduce harmful cascading effects.
- For economic modeling and policy:
- Widespread adoption of imperfect autonomous agents in seller markets could shift dynamics: higher variance in seller outcomes, potential increases in disputes and refunds, and altered demand because of inconsistent ratings — all of which affect platform welfare and regulatory risk.
- Regulators and marketplaces should monitor aggregated agent-driven metrics and require clear accountability for economic harms caused by deployed agents.
Summary takeaway: MerchantBench exposes substantial gaps between current LLM-based merchant agents and human operators on long-horizon, delayed-feedback economic tasks. Progress will require modeling and system advances focused on robust long-term credit assignment, memory, cash-aware planning, and principled risk management.
Assessment
Paper Typedescriptive
Evidence Strengthn/a — The paper introduces a benchmark and reports within-simulator comparisons of agent configurations; it does not attempt causal identification of real-world effects, so causal evidence strength is not applicable. The results are informative about agent performance in the simulated environment but have limited external causal interpretation.
Methods Rigormedium — The environment is carefully specified (POMDP), grounded in a large real-world dataset (98,843 products, 365-day traces), and exposes a rich toolset and multiple evaluation metrics; experiments compare eight LLMs, two agent frameworks, a rule baseline, and human participants with repeated runs. However, only three repetition runs per configuration, a very small human sample (three participants), limited sensitivity/robustness analysis of simulator assumptions and prompts, and potential undisclosed prompt/controller details reduce rigor.
SampleSimulation environment grounded in 98,843 product records and 365-day product-level demand histories from 1688 (June 1, 2025–May 31, 2026), covering 36,576 suppliers and 10 top-level categories; simulator runs on a 365-day (8,760 hourly-step) POMDP with order-level lifecycle and upstream/downstream event models; experiments: 8 LLMs × 2 agent frameworks × 3 repeated runs = 48 runs, each a 365-day simulated episode, plus a rule-based baseline and three human participants; initial store conditions: RMB 2,000 cash, RMB 1,000 security deposit, capacity for 50 active listings.
Themesproductivity human_ai_collab
GeneralizabilityCalibrated to 1688 (Chinese wholesale platform) — demand patterns, supplier behaviors, and platform rules may not generalize to other marketplaces or consumer contexts., Drop-shipping single-item model — results may not transfer to inventory-holding retail or multi-channel sellers., Simulator assumptions and calibrated risk/event probabilities may diverge from live platform dynamics (competitor responses, policy changes, fraud, seasonality beyond the data window)., Small human sample (n=3) limits robustness of human vs. agent comparisons., Evaluated LLM versions and agent-framework configurations represent a snapshot; future models or different engineering could change outcomes., No multi-agent market competition, adversarial sellers, or macroeconomic shocks included that would be present in real deployments.
Claims (11)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| MerchantBench is a 365-day order-level e-commerce simulation grounded in 98,843 real e-commerce product records and equipped with 26 tools for agent interaction. Other | positive | Benchmark scale and agent interaction capability |
Reading fidelity
high
Study strength
medium
|
n=98843
|
| MerchantBench evaluates eight LLMs under two agent frameworks in 48 runs, with each run spanning 365 simulated days. Organizational Efficiency | positive | Long-horizon agent performance over simulated store operations |
Reading fidelity
high
Study strength
medium
|
n=48
|
| The best LLM configuration achieved only 27.3% of the mean final net assets achieved by human participants. Firm Revenue | negative | Final net assets after 365 simulated days |
Reading fidelity
high
Study strength
medium
|
n=48
27.3% of human mean final net assets
|
| Qwen3.7-Max with the Hermes framework achieved the highest final net assets among all 16 LLM configurations, with mean final net assets of 59.46 thousand RMB. Firm Revenue | positive | Final net assets after 365 simulated days |
Reading fidelity
high
Study strength
medium
|
n=48
59.46 thousand RMB
|
| When averaged across the eight models, Hermes produced 53.3% higher final net assets than ReAct. Firm Revenue | positive | Final net assets after 365 simulated days |
Reading fidelity
high
Study strength
medium
|
n=48
53.3% higher final net assets
|
| When averaged across the eight models, Hermes produced 71.5% higher GMV than ReAct. Firm Revenue | positive | Gross merchandise value after 365 simulated days |
Reading fidelity
high
Study strength
medium
|
n=48
71.5% higher GMV
|
| When averaged across the eight models, Hermes produced 71.2% more orders than ReAct. Other | positive | Number of orders over 365 simulated days |
Reading fidelity
high
Study strength
medium
|
n=48
71.2% more orders
|
| Hermes yielded higher mean final net assets than ReAct for seven of the eight evaluated models. Firm Revenue | positive | Mean final net assets after 365 simulated days |
Reading fidelity
high
Study strength
medium
|
n=48
7 of 8 models
|
| The benefit of Hermes depended on the underlying model: gains in mean final net assets ranged from 11.5% for Claude Opus 4.8 to 187.8% for Qwen3.7-Max, while Kimi K2.6 performed 4.1% worse under Hermes than under ReAct. Firm Revenue | mixed | Mean final net assets after 365 simulated days |
Reading fidelity
high
Study strength
medium
|
n=48
11.5% to 187.8% gains; 4.1% decline for Kimi K2.6
|
| Qwen3.7-Max under Hermes had a coefficient of variation of 55.1% for final net assets, indicating substantially lower stability than the most stable configurations. Firm Revenue | negative | Variability of final net assets across repeated runs |
Reading fidelity
high
Study strength
low
|
n=3
55.1% coefficient of variation
|
| Human participants achieved higher mean final net assets than both LLM agents and the rule-based baseline: 217.61 thousand RMB versus 59.46 thousand RMB for the best LLM configuration and 24.48 thousand RMB for the rule-based baseline. Firm Revenue | positive | Final net assets after 365 simulated days |
Reading fidelity
high
Study strength
low
|
n=3
217.61 thousand RMB for humans; 59.46 thousand RMB for best LLM; 24.48 thousand RMB for rule-based baseline
|