0 cumulative citations
View corpus contextLLM merchants can win customers but not yet their full profits: in a dynamic multi-attribute auction benchmark the best agents win frequently but underprice, earning under one-third of hindsight-optimal profit; models that learn fastest before demand shocks often adapt slowest afterward.
Can LLM Agents Price Competitively? A Dynamic Multi-Attribute Auction Benchmark for Agentic Commerce
In a controlled simulated multi-attribute auction benchmark, state-of-the-art LLM agents can win many customers but underperform on profit—total profit aligns almost perfectly with margin per win rather than win rate, and even the best agent captures less than a third of the hindsight-optimal profit while showing a trade-off between fast pre-shock learning and slow post-shock adaptation.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Agentic commerce is moving from concept to deployed infrastructure: payment networks, retailers, and AI platforms are setting the stage for agents to transact on behalf of merchants and consumers. Yet whether the LLMs behind these agents can price competently in real markets, where customer preferences are hidden, competitors adapt in real time, and demand can shift without warning, has not been systematically tested. We introduce Bazaar, a dynamic sealed-bid benchmark for multi-attribute auction under these conditions. Despite its dynamics, the benchmark is grounded in closed-form customer utilities, enabling exact evaluation. Across 11 frontier LLMs from four providers, the leading agents on customer acquisition (e.g. Gemini 3.1 Pro) are often not the leading agents on profit (e.g. Opus 4.6). The ranking shifts again under demand shocks: agents that learned fastest pre-shock are typically the slowest to revise their beliefs afterwards, while Gemini 3.1 Pro recovers fastest despite not leading on profit. However, even the strongest agent captures less than a third of hindsight-optimal profit, suggesting current LLMs are progressing in agentic commerce but leave substantial headroom.
Summary
Main Finding
Bazaar — a dynamic sealed-bid multi-attribute auction benchmark — shows that current frontier LLMs can learn and price in competitive markets, but performance is uneven and far from optimal. Profit is driven far more by margin discipline than by raw win rate, thinking/compute budget materially changes outcomes, and agents that learn fastest pre-shock can be the slowest to adapt after unannounced preference shifts. Even the best agent captures under one-third of the hindsight-optimal profit, leaving substantial headroom for improvement.
Key Points
-
Benchmark (Bazaar) purpose
- Tests whether an LLM merchant can (a) infer hidden customer preferences, (b) choose profitable configurations and margins, and (c) adapt after unannounced demand shifts against adaptive competitors.
- Produces diagnostics that separate preference learning (which configuration to offer), margin extraction (how much to charge), and adaptation (post-shock belief revision).
-
Core empirical findings
- Margin per win almost perfectly predicts profit (r = 0.99); win rate is less predictive (r = 0.88). Charging appropriately matters more than simply winning more auctions.
- Thinking/effort budget has a large effect: e.g., GPT-5.4 at no extra effort earns $266, while at high effort it earns $1,936 (7.3× increase).
- Two successful but distinct strategies emerge:
- Margin-focused leaders (e.g., Opus 4.6 adaptive xhigh) lead on profit by extracting higher margins ($2.22 per win) and top cumulative profit ($2,976).
- Coverage/adaptation leaders (e.g., Gemini 3.1 Pro dynamic high) lead on win rate (77.9%), recover fastest from shocks, and are more API-cost-efficient.
- Shock behavior: models that climb fastest before a preference shock often adapt worst afterward (strong-learner / weak-adapter reversal). The best adaptors recover within ~10–15 rounds; worst suffer persistent degradation.
- Bandit baselines are a meaningful floor: per-customer Thompson Sampling earns $1,426 (better than several LLM variants); EXP4 earns $1,023.
- Absolute performance still far from oracle: strongest agents capture < 33% of hindsight-oracle profit.
-
Diagnostic taxonomy
- Bazaar decomposes regret into loss regret (missed profitable wins) and underprice regret (won but left money on table), enabling classificaton (e.g., Loser vs Underpricer).
- Increasing reasoning effort tends to reduce loss regret (fewer missed wins) but increase underprice regret (more conservative pricing), moving agents along a two-dimensional failure surface.
Data & Methods
-
Environment (instantiation used in experiments)
- Items: 3 abstract attributes (A, B, C), 5 levels each → 5^3 = 125 configurations.
- Customers: 24 heterogeneous customers with additive valuation curves over the attributes; mixture of single-attribute-dominant, dual-preference, and triple-bundle-seeking customers.
- Merchants: one focal LLM-controlled generalist merchant (can bundle attributes) and three rule-based specialist bots (each expert on one attribute). Merchant costs are additive; specialists have structural cost advantages on their specialty.
- Auctions: repeated sealed-bid customer-level auctions each round. Each merchant submits a configuration and price for each customer; customers choose the offer maximizing utility (value − price).
- Feedback: after each auction, all merchants observe the winner identity, winning configuration, and price; the winner also observes realized profit. Losing bids, customer utilities, and competitors' private info are not revealed.
-
Dynamics and shocks
- Temporal setup: rounds 1–30 are pre-shock learning. Twelve of 24 customers undergo a paired-attribute value-swap shock (A↔B, A↔C, or B↔C) with switch rounds sampled from rounds 31–40 (staggered). Each shocked customer runs for 40 further rounds post-shock, allowing adaptation measurement.
- Shocks are unannounced — agents must infer changes from the same sparse feedback stream.
-
Agents and evaluation
- LLMs: 11 frontier models from four providers, evaluated across variants of thinking/effort modes (examples: Opus 4.6, Gemini 3.1 Pro, GPT-5.5, GPT-5.4, GPT-5.3, Opus 4.5/4.7, Sonnet 4.6, Grok 4.x, Gemini Flash). Each model variant runs on 10 seeds (3 for one variant) to capture stochasticity.
- Baselines: Thompson Sampling and EXP4 bandits over the same action space (instantiated per-customer / per-cluster / global variants).
- Per-round LLM outputs: 24 customer bids ((a,b,c) @ $p), 24 per-customer belief updates (natural language), and one global strategy update.
- Opponent bot dynamics: fixed configuration (specialty) but adaptive per-customer pricing; margins raised after wins and reduced after losses, with asymmetric step sizes to prevent trivial exploitation.
-
Metrics and diagnostics
- Primary: total profit, win rate, margin per win.
- Oracle efficiency: realized profit divided by hindsight-oracle profit (oracle knows customer values and competitor utility-per-round).
- Regret decomposition: loss regret (rounds agent lost but oracle would have won) and underprice regret (agent won but captured less than oracle).
- Shock recovery: late-post-shock win-rate minus peak pre-shock win-rate (measures adaptation).
- Statistical testing: permutation tests to form significantly different tiers.
Implications for AI Economics
-
For agentic commerce design and deployment
- Pricing competence is multi-dimensional: marketplaces should monitor not only win rates or conversion, but margin discipline and adaptation to demand shifts. Audits should use diagnostics like loss vs underprice regret.
- Compute/effort is economically consequential: providers and merchants will trade off API/compute costs for improved pricing. This creates an incentives landscape where “thinking budget” affects market power — a potential arms race in compute for pricing quality.
- Adaptive opponents and staggered, unannounced preference changes expose failure modes (fast learners who fail to revise). Real deployments need mechanisms (e.g., forced exploration, explicit change-detection signals, or conservative updating) to avoid persistent mispricing after shocks.
-
For market efficiency and welfare
- Current LLM agents still capture a small fraction of hindsight-optimal profit, implying consumer surplus could be affected in complex ways depending on agent behavior (e.g., underpricing can benefit short-term consumers but may signal fragility).
- Agents that consistently underprice or over-extract margins can distort competitive dynamics; platform rules or monitoring may be needed to avoid harmful competition patterns (e.g., destructive price wars or tacit collusion through similar strategies).
-
For regulation and oversight
- Evaluation frameworks like Bazaar provide interpretable diagnostics that regulators and platforms can use to assess merchant agent behavior (e.g., adaptation speed, margin extraction patterns).
- Because compute budget materially alters outcomes, transparency about agent compute/strategy and monitoring of differential access could be relevant policy levers.
-
For future research and engineering
- Key technical priorities: faster and more robust belief revision after shifts; explicit change-point detection from sparse feedback; improved counterfactual reasoning about competitor utility; and multi-agent learning to handle strategic opponents.
- Benchmarks for agentic commerce should incorporate dynamics, hidden preferences, adaptive competitors, and shocks — static tasks miss important failure modes.
- Economic evaluation should combine profit-oriented metrics with welfare-focused measures (consumer surplus, price volatility) to fully assess systemic effects of deploying LLM merchants.
Summary conclusion: Bazaar demonstrates that LLMs are making measurable progress toward agentic commerce, but success hinges on margin discipline, compute-backed reasoning, and robust adaptation. There remains substantial room for model, algorithmic, and mechanism-design improvements before LLM merchants approach oracle-level pricing performance.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper reports systematic, repeatable simulation experiments across 11 frontier LLMs with multiple seeds, baselines (Thompson Sampling, EXP4), and careful diagnostics (oracle regret decomposition, permutation tests). However, the evidence is limited to a simulated environment with stylized customers and rule-based opponents rather than field or real-market data, which constrains external validity.
Methods Rigorhigh — Experimental design is detailed and controlled: a well-specified multi-attribute auction environment, staggered unobserved demand shocks, multiple seeds, statistical tests for tiering, and non-LLM baselines; diagnostics separate learning, pricing, and adaptation errors. Limitations remain because the opponent models and customer preferences are simulated and chosen by the authors, and some implementation details are in appendices.
SampleSimulated repeated sealed-bid multi-attribute auctions with one focal LLM merchant competing against three adaptive, rule-based specialist bots across 24 simulated customers; each customer has additive valuations over K=3 attributes with L=5 levels (125 configurations). Twelve customers experience a paired-attribute preference swap (shocks) in rounds 31–40, with the remainder as controls. Each model variant evaluated over ~79 rounds (30 pre-shock + post-shock windows) and averaged across 10 random seeds (three for one variant). Eleven frontier LLMs and variants were tested, plus Thompson Sampling and EXP4 baselines.
Themesadoption innovation
GeneralizabilitySimulated customers with additive value functions may not capture real consumer heterogeneity or strategic behavior., Opponents are simple specialists that only adapt price; real competitors may change product configurations and strategies., Environment uses only 3 attributes and 5 levels each (125 configurations), limiting complexity relative to real marketplaces., Merchant cost structures, shock types (paired-attribute swaps), and timing were chosen by authors and may not represent real demand shocks., Performance depends on LLM prompting, reasoning budgets, and API costs which vary across providers and deployment contexts., Results reflect present model snapshot (specific LLM versions) and may change rapidly as models and tools evolve.
Claims (10)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Across evaluated model variants, total profit is more strongly correlated with margin per win than with win rate. Firm Productivity | positive | Total profit |
Reading fidelity
high
Study strength
medium
|
n=21
r=0.99 for margin per win versus r=0.88 for win rate
|
| Gemini 3.1 Pro achieves a higher win rate than Opus 4.6 but earns lower total profit because its margin per win is lower. Firm Productivity | mixed | Win rate, margin per win, and total profit |
Reading fidelity
high
Study strength
medium
|
n=10
77.9% vs. 67.8% win rate; $1.83 vs. $2.22 per win
|
| Opus 4.6 with adaptive, xhigh effort achieves the highest reported total profit among the evaluated LLM configurations. Firm Productivity | positive | Cumulative profit |
Reading fidelity
high
Study strength
medium
|
n=10
$2,976 ±354
|
| Increasing thinking effort substantially increases profit for some fixed-base-model variants, with the largest reported gain for GPT-5.4. Firm Productivity | positive | Cumulative profit |
Reading fidelity
high
Study strength
medium
|
n=10
GPT-5.4 earns 7.3× more at high effort than at none ($1,936 vs. $266)
|
| Gemini 3.1 Pro is the most cost-efficient top-performing agent evaluated, producing nearly the same market profit as the leading agents at substantially lower API cost. Organizational Efficiency | positive | API cost relative to benchmark profit |
Reading fidelity
high
Study strength
medium
|
n=10
4.8× cheaper than Opus 4.6; $186 vs. $890 API cost per seed
|
| Per-customer Thompson Sampling provides a non-trivial performance floor, outperforming six of the eleven base LLMs on total profit. Firm Productivity | positive | Total profit |
Reading fidelity
high
Study strength
medium
|
n=11
$1,426 profit; above six of eleven base LLMs
|
| Top LLM agents outperform Thompson Sampling particularly in margin extraction: Opus 4.6 earns a substantially higher realized margin per win than Thompson Sampling. Firm Productivity | positive | Margin per winning bid |
Reading fidelity
high
Study strength
medium
|
n=10
$2.22 per win vs. $1.17 per win
|
| Models that learned fastest before the preference shock were among the weakest at adapting after the shock. Decision Quality | negative | Post-shock win-rate recovery and adaptation |
Reading fidelity
high
Study strength
medium
|
n=2
Pre-shock gains of +46 and +40 percentage points; average post-shock loss of −19 percentage points
|
| Gemini 3.1 Pro recovers fastest from the unannounced customer-preference shocks despite not being the highest-profit agent. Decision Quality | positive | Shock recovery in win rate |
Reading fidelity
high
Study strength
medium
|
n=11
No numerical recovery advantage reported in the supplied text
|
| Even the strongest evaluated agent captures less than one-third of the hindsight-optimal profit. Firm Productivity | negative | Oracle efficiency, defined as realized profit divided by hindsight-oracle profit |
Reading fidelity
high
Study strength
medium
|
n=11
Less than 1/3 of hindsight-optimal profit
|