0 cumulative citations
View corpus contextA new ERP benchmark finds LLM enterprise agents’ perceived superiority depends on market ecology: DeepSeek outperforms in fixed-rule contests, but Gemini wins in shared-model markets, with only 21 of 100 tasks agreeing on the best model and execution-audits exposing frequent capacity and parsing interventions.
ERPBench: Evaluating LLM Agents for Enterprise Decision-Making Across Competitive Market Ecologies
ERPBench shows that LLM enterprise-agent rankings are ecology-dependent: model leaderboards change when the same ERP problems are embedded in fixed-rule-opponent (Solo) versus shared-model (Arena) competitive markets, and execution-audit logs reveal frequent feasibility and parsing interventions that qualify terminal valuations.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Large language model (LLM) agents are increasingly proposed for enterprise workflows, yet existing evaluations rarely test whether business-decision conclusions transfer across competitive market ecologies. We introduce ERPBench, an execution-instrumented benchmark for enterprise decision agents in a six-round Enterprise Resource Planning (ERP) simulation with coupled pricing, production, procurement, inventory, finance, and shared-market competition. ERPBench evaluates the same 100 fixed problems in two matched competitive market ecologies: Solo, where each evaluated LLM agent competes against fixed rule-based opponents, and Arena, where six evaluated LLM agents compete in a shared market. Across six model families, this yields 1,200 model-level trajectories spanning 7,200 decision rounds. Under the observed service configuration, the leading model differs between ecologies: DeepSeek leads in Solo (252.29M mean valuation; mean rank 1.67), whereas Gemini leads in Arena (263.95M; 1.76). The two ecologies identify the same task-level winner on only 21 of 100 problems, and Gemini's bottom-rank rate falls from 22 % to 0 % in Arena. ERPBench supports paired evaluation of whether enterprise-agent rankings transfer across competitive market ecologies, supplemented by aggregate execution-intervention analysis. Code and benchmark resources are available in our https://github.com/GAIR-NLP/erp-bench.
Summary
Main Finding
ERPBench — an execution-instrumented Enterprise Resource Planning (ERP) benchmark — shows that model rankings for enterprise decision agents change substantially when the competitive market ecology is altered. Evaluating the same 100 ERP problems in matched Solo (fixed rule-based opponents) and Arena (six evaluated LLM agents competing) ecologies produces different winners, widespread task-level disagreement, and shifts in tail-risk and execution fragility. In the observed setup, DeepSeek leads in Solo (mean valuation 252.29M) while Gemini leads in Arena (263.95M); Gemini gains +94.54M between Solo and Arena, and the two ecologies pick the same task-level winner on only 21/100 problems.
Key Points
-
Benchmark design
- Paired-ecology protocol: identical 100 problem instances evaluated under two matched ecologies (Solo vs Arena) to isolate ecology as a measurement variable.
- 6 model families evaluated: Claude Opus 4.6, DeepSeek-V4-Flash, Gemini 3.1 Pro Preview, GPT-5.5, Doubao Seed 2.0 Pro, Qwen 3.7 Max.
- Scale: 1,200 model-level trajectories, 7,200 decision rounds (six rounds per trajectory), 700 simulator runs total.
-
Task and action space
- Simulates a cereal-manufacturing ERP over six rounds (30 days each).
- Decisions: SKU pricing, production quantities, procurement, marketing allocation, finance actions for 12 SKUs across 3 channels.
- Environment: batch production with capacity and lot-size constraints, supplier lead times, warehouse inventory, cash/credit dynamics, carbon costs, and a multinomial competitive interaction demand model.
-
Primary metrics and diagnostics
- Primary outcome: terminal company valuation via a dividend discount model at round 6.
- Complementary metrics: within-problem rank, bottom-rank rate, normalized regret, task-level winner agreement, short-horizon adaptation.
- Execution-audit logs: parsed decisions, feasibility checks, clamping, repairs/fallbacks, tracebacks; used to qualify terminal scores.
-
Main quantitative findings
- Aggregate leaderboard changes across ecologies:
- DeepSeek: Solo 252.29M (rank 1.67) → Arena 234.71M (rank 2.36); Δ = −17.58M (p = .0116).
- Gemini: Solo 169.41M (rank 3.51) → Arena 263.95M (rank 1.76); Δ = +94.54M (p ≈ 1.28×10−15).
- Qwen: 205.87M → 187.13M; Δ = −18.74M (p = .0061).
- GPT-5.5, Doubao, Claude show smaller/non-significant average shifts.
- Task-level disagreement: same winner in Solo and Arena for only 21 of 100 problems; Spearman rank correlation ≈ 0.182 (CI [0.090, 0.273]).
- Tail-risk shifts: Gemini bottom-rank rate drops from 22% (Solo) to 0% (Arena); Gemini P90 normalized regret from 0.73 → 0.27; DeepSeek P90 regret increases 0.16 → 0.39.
-
Execution fragility
- Execution interventions are common and informative: production-capacity skips dominate (3,761 events, 82.5% of logged interventions), followed by unavailable investments (7.8%), fallback decisions (6.0%), production-quantity clamping (2.6%), and tracebacks (1.1%).
- Execution-audit records reveal whether a score was achieved via clean execution or via repeated repairs/fallbacks — an important qualifier for deployment reliability.
Data & Methods
-
Paired experimental protocol
- Same 100 ERP problem seeds, same agent scaffold, same six-round decision horizon; only opponent composition changes between Solo and Arena.
- Solo ecology: focal LLM vs five deterministic rule-based opponents with different policies (conservative, aggressive, balanced, growth-oriented, risk-averse).
- Arena ecology: six evaluated LLM agents compete simultaneously in a shared market.
-
Simulation specifics
- ERPsim-style cereal manufacturing environment with operational constraints (capacity, lot sizes, lead times), cash-flow and finance dynamics, carbon costs, and a multinomial competitive interaction demand model.
- Agents output structured JSON actions that the simulator parses and executes, recording both proposed and executed values plus any interventions.
-
Statistical analysis
- Paired contrasts computed per model–problem pair: Δp,m = V_Arena − V_Solo.
- Model-level mean effects reported with 95% percentile-bootstrap confidence intervals over problems.
- Paired tests: two-sided Wilcoxon signed-rank tests with Benjamini–Hochberg correction across model-wise tests.
- Reliability diagnostics: operational-intervention rates summarized into an L2 operational reliability score (1 − r_m/max r_m′), plus descriptive L1–L5 Decision Quality Pyramid.
-
Audit logging
- Logs include parser outcomes, feasibility checks, clamping/repairs, fallback choices, and runtime tracebacks; intervention counts and categories used as process diagnostics.
Implications for AI Economics
- Competitive ecology matters for agent evaluation
- Opponent composition is not background noise: it systematically alters demand allocation, prices, inventory pressure, and cash-flow dynamics, producing different agent rankings and equilibria. Benchmarks and procurement processes should treat opposing agents/market structure as an experimental variable.
- Deployment and procurement guidance
- Selecting a model for enterprise decision-making requires testing across realistic market ecologies (e.g., mixtures of rule-based incumbents and adaptive LLM agents) rather than a single fixed benchmark. Tail-risk and execution-fragility diagnostics (not only mean valuation) should factor into deployment decisions.
- Economic modeling and strategic interactions
- The observed reversals and rank shifts point to strong strategic coupling and potential path dependence: identical policies can yield different payoffs depending on others’ adaptive behavior. This motivates theoretical and empirical work on equilibria of LLM-agent populations and on stability/robustness of agent strategies in markets.
- Importance of execution costs and constraints
- Execution interventions (capacity skips, clamping, fallback) function as transaction/implementation costs. Economic evaluation of agents must include these operational frictions because they change effective payoffs and reliability.
- Research directions
- Design robust agents trained or adapted for multi-agent, ecology-aware settings (e.g., multi-agent RL, opponent-aware planning).
- Extend benchmarks to study equilibrium formation, dynamics under repeated adaptation, incentive design, and regulatory interventions (e.g., disclosure of agent behavior, standards for execution fidelity).
- Incorporate richer economic externalities (network effects, market entry/exit, supplier dynamics) and longer horizons to assess systemic impacts of deploying LLM decision agents at scale.
In short: ERPBench formalizes that market ecology and execution feasibility are core measurement conditions for enterprise decision agents. For AI economics, this means agent evaluation, selection, and policy must account for strategic interactions and operational frictions, not just isolated leaderboard performance.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper provides extensive, well-instrumented simulation evidence (100 matched problems, paired Solo vs Arena design, execution-audit logs, bootstrap CIs), which supports strong internal conclusions about benchmark behavior, but its results are limited to a single ERP simulation, a short horizon, and specific model/service configurations, so external (real-world) economic inference is weak.
Methods Rigormedium — The design is careful: paired within-problem comparisons, matched seeds and strata, bootstrap CIs, nonparametric tests, and execution-audit logging improve internal validity and reproducibility; however, reliance on a single simulated ERP domain, deterministic rule-based Solo opponents, limited rounds (six), and lack of sensitivity analyses to simulator parameter choices reduce rigor relative to a broader empirical evaluation.
SampleSimulation experiment: an ERP-style cereal-manufacturing simulator with 100 distinct problem instances (baseline, dual-pressure, hard scenarios). Six evaluated LLM model families (Claude Opus 4.6, DeepSeek-V4-Flash, Gemini 3.1 Pro Preview, GPT-5.5, Doubao Seed 2.0 Pro, Qwen 3.7 Max) run under two matched ecologies — Solo (each evaluated model vs five deterministic rule-based opponents) and Arena (six evaluated LLM agents competing together). Total: 1,200 model-level trajectories, 7,200 decision rounds, 600 Solo simulator executions, 100 six-model Arena executions (700 runs total). Environment: six rounds of 30 simulated days, 12 SKUs, multi-channel demand via an MCI model, production capacity, procurement lead times, inventory, cash/finance, and carbon costs. Agents produce structured JSON actions; simulator logs parsing, feasibility checks, clamping, fallbacks, and terminal DDM-based valuations.
Themesproductivity adoption
GeneralizabilityFindings are confined to a single ERP simulation (cereal-manufacturing analog) and may not generalize to other industries or real-world ERP instances., Results depend on specific simulator assumptions (demand model, capacity rules, lead times, cost structures); different simulators or parameters could change rankings., Short decision horizon (six rounds) limits inference about long-run strategic behavior and adaptation., Performance is tied to the particular agent scaffold, parsing/execution interface, and service configurations; different toolchains or integration layers could alter outcomes., Solo opponents are deterministic rule-based agents, which may not represent realistic competitor behavior; Arena results depend on the chosen evaluated models., No human-in-the-loop or organizational constraints (e.g., multi-agent coordination, governance, or regulatory limits) are modeled, limiting deployment inference.
Claims (11)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Under the observed service configuration, DeepSeek is the leading model in the Solo ecology, while Gemini is the leading model in the Arena ecology. Decision Quality | mixed | Mean terminal company valuation and mean within-problem rank |
Reading fidelity
high
Study strength
high
|
n=100
DeepSeek: 252.29M mean valuation and 1.67 mean rank in Solo; Gemini: 263.95M mean valuation and 1.76 mean rank in Arena
|
| Gemini's mean terminal valuation increases by 94.54M when moving from Solo to Arena on the same problems. Firm Productivity | positive | Mean terminal company valuation |
Reading fidelity
high
Study strength
high
|
n=100
+94.54M
|
| DeepSeek's mean terminal valuation decreases by 17.58M in Arena relative to Solo. Firm Productivity | negative | Mean terminal company valuation |
Reading fidelity
high
Study strength
high
|
n=100
−17.58M
|
| The two competitive market ecologies identify the same task-level winner on only 21 of 100 problems. Decision Quality | negative | Agreement in task-level model winner across ecologies |
Reading fidelity
high
Study strength
high
|
n=100
21 of 100 problems
|
| The mean within-problem Spearman rank correlation between Solo and Arena rankings is 0.182. Decision Quality | negative | Consistency of model rankings across competitive ecologies |
Reading fidelity
high
Study strength
medium
|
n=100
Spearman rank correlation = 0.182; bootstrap CI [0.090, 0.273]
|
| Thirty-five problems won by DeepSeek in Solo are won by Gemini in Arena. Decision Quality | mixed | Task-level winner transitions between ecologies |
Reading fidelity
high
Study strength
medium
|
n=100
35 problems
|
| Gemini's bottom-rank rate falls from 22% in Solo to 0% in Arena. Decision Quality | positive | Bottom-rank rate |
Reading fidelity
high
Study strength
medium
|
n=100
22% to 0%
|
| Gemini's P90 normalized regret falls from 0.73 in Solo to 0.27 in Arena. Decision Quality | positive | P90 normalized regret relative to the best model |
Reading fidelity
high
Study strength
medium
|
n=100
0.73 to 0.27
|
| DeepSeek's P90 normalized regret rises from 0.16 in Solo to 0.39 in Arena. Decision Quality | negative | P90 normalized regret relative to the best model |
Reading fidelity
high
Study strength
medium
|
n=100
0.16 to 0.39
|
| Production-capacity skips are the most frequent logged execution intervention, accounting for 82.5% of displayed intervention events. Organizational Efficiency | negative | Execution-intervention rate and operational feasibility |
Reading fidelity
high
Study strength
medium
|
3,761 events; 82.5%
|
| The benchmark contains 1,200 model-level trajectories and 7,200 model-level decision rounds. Other | positive | Benchmark evaluation coverage |
Reading fidelity
high
Study strength
high
|
n=1200
1,200 trajectories; 7,200 decision rounds
|