1 cumulative citations
View corpus contextLarge language models can reach or surpass trained MBA students in simulated negotiations, with frontier models performing near human levels; targeted agentic scaffolding lifts mid- and low-tier models most, while top models show diminishing incremental gains.
PieArena: Ranking and Profiling Language Agents in Realistic Negotiation Scenarios
In MBA-style negotiation scenarios, many LLMs—especially a representative frontier model (GPT-5)—match or exceed performance of trained business-school students, while agentic scaffolding yields the largest gains for mid- and lower-tier models but diminishing returns for frontier models.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
We present an in-depth evaluation of LLMs' ability to negotiate, a central business task requiring strategic reasoning, theory of mind, and economic value creation. To do so, we introduce PieArena, a large-scale negotiation benchmark grounded in multi-agent interactions over realistic scenarios adapted from MBA negotiation courses at an elite business school. We evaluate language agents across three pairing regimes: mirror-play, cross-play, and human-LM play. We develop a ranking model for continuous negotiation payoffs that yields order-invariant, uncertainty-quantified leaderboards while correcting for systematic experimental asymmetries. We further study the effects of joint-intentionality agentic scaffolding and find asymmetric gains, with large improvements for mid- and lower-tier LMs and diminishing returns for frontier LMs. As calibration anchors, we collect human-human and human-LM negotiation data from trained business school students, finding that a representative frontier language agent (GPT-5) matches or exceeds this human baseline in our evaluation settings. Beyond deal outcomes, PieArena provides a multi-dimensional behavioral profile that reveals cross-model heterogeneity in instruction compliance, computation accuracy, as well as judge-assessed deception and reputation, illustrating the value of evaluation beyond outcome-only leaderboards.
Summary
Main Finding
PieArena is a large-scale, realistic benchmark for negotiating language agents that (i) produces robust, order-invariant rankings for continuous negotiation payoffs via a new Beta‑Generalized Bradley–Terry–Luce (BBTL) model, and (ii) shows that agentic scaffolding (shared‑intentionality state tracking + strategic planning) substantially raises the competence floor—producing large catch‑up gains for mid and lower‑tier LMs while yielding diminishing returns for frontier models. In the evaluation settings, a representative frontier agent (GPT‑5) matches or exceeds the trained‑student human baseline on several tasks (especially single‑issue value‑claiming), but humans can outperform the agent on trained multi‑issue integrative tasks.
Key Points
-
Benchmark design
- PieArena uses realistic MBA negotiation scenarios (single‑issue zero‑sum and multi‑issue positive‑sum) with deterministic utilities, private information, BATNAs, and fixed round limits (6 rounds).
- Outcomes are structured (JSON) and scored to yield continuous payoffs: total pie (joint surplus) and pie shares (division).
- Pairing regimes: mirror‑play (same agent both sides), cross‑play (different agents), and human–LM play (calibration against trained MBA students).
-
Dataset scale
- Started from 326 chat‑capable models (OpenRouter API); after API and task‑grounded screening, deduplication and stability filters, final LM evaluation set = 13 models.
-
25,000 LM negotiation transcripts collected under standardized protocols.
- Human data: 167 negotiation sessions with MBA students (including 23 human–human SnyderMed pairs; Main Street n=89 human–LM; Top Talent n=55 human–LM).
-
Agentic harness (shared‑intentionality)
- Two main modules: (i) state tracking for perspective taking and preference inference, (ii) strategic planning that outputs round‑level goals/tactics.
- Large asymmetric gains: marked improvement for mid/lower models, little/no improvement for top frontier models (diminishing returns).
- Under scaffolding the negative performance tail disappears—no models were significantly worse than humans in mirror‑play.
-
Ranking methodology (BBTL)
- Extends Bradley–Terry–Luce to continuous payoffs on [0,1] by modeling pie share with a Beta likelihood and a logit‑linked mean.
- Model components: latent skill θi, global first‑speaker effect γ, scenario role asymmetry ϕs, and per‑scenario precision κs.
- Benefits: uses continuous outcomes (not binarized), corrects experimental asymmetries (speaking order, scenario structure), yields order‑invariant leaderboards and principled confidence intervals; empirically outperforms sequential Elo and binary‑BTL baselines.
-
Representative quantitative results
- Human–human SnyderMed (mirror): mean normalized total pie = 0.874 (SD 0.085).
- GPT‑5 mirror‑play total pie vs human baseline: base mean ≈ 0.960, pro mean ≈ 0.970 (statistically above human baseline).
- Main Street (single‑issue, human–LM): students captured 39.7% of pie vs GPT‑5 agent 60.3% (agent significantly outperformed students).
- Top Talent (multi‑issue, pro GPT‑5): students captured 53.8% vs agent 46.2% (students outperformed the agent in this trained integrative task).
- Behavioral diagnostics: heterogeneous lie/deception rates across models (example: some xAI family models single/low‑digit lie rates; GEMINI‑3‑PRO ≈39.9%, GPT‑5.2 ≈33.9%, CLAUDE‑SONNET‑4.5 ≈31.7%).
-
Behavioral profiling beyond outcomes
- PieArena collects per‑transcript diagnostics: deception, numerical accuracy, instruction compliance, reputation, deal‑closing reliability—revealing trade‑offs hidden by scalar leaderboards.
Data & Methods
-
Scenarios and protocol
- Scenarios include SnyderMed (integrative job offer), Top Talent (multi‑issue hiring), Main Street (single‑issue property price), Twisted Tree (single‑issue haggling), Z‑lab (two‑issue licensing with contingencies).
- Negotiations: fixed horizon 6 rounds, randomized first speaker, private role instructions and preferences; final agreement parsed to structured schema mapped to deterministic utilities; no‑deal defaults to BATNAs.
-
Human data collection
- Conducted with MBA students as part of course activities and assignments, IRB approved, standardized interface and scoring.
- Human–human mirror (SnyderMed), human–LM Main Street (in‑class), human–LM Top Talent (post‑class assignment with targeted integrative negotiation instruction).
-
LM pool, screening, and harnessing
- 326 candidate models → screened via API feasibility, no‑ZOPA probe (must walk away when ZOPA empty), and multi‑issue execution probe; dedup and stability filters → 13 LMs.
- Each LM run in standardized harness that mediates turns, exposes private info, elicits structured final deal.
- Two modes per LM: base (off‑the‑shelf) and pro (with agentic harness state tracking + strategy planner). Per‑module ablations reported.
-
Experimental design
- Mirror‑play: ordered pairing repeated n=20; cross‑play ordered pairing repeated n=6 (symmetrized for first‑speaker effects).
- Outcome validation: agreement schema checking, BATNA compliance, computation accuracy; invalid or no‑deal scored as no‑deal.
-
Statistical model (BBTL)
- For play k between Mi and Mj in scenario s: yk (side‑1 pie share) ~ Beta(µk κs, (1−µk) κs) with µk = logistic(θi − θj + γ x(F)k + ϕs).
- Fit by maximum likelihood jointly across plays; yields θ skill estimates with CI, controls for first‑mover and scenario asymmetries.
Implications for AI Economics
-
Practical evaluation for market tasks
- Negotiation is economically meaningful (strategic planning, theory‑of‑mind, distribution of surplus). PieArena provides a realistic, scalable, and saturation‑resistant benchmark aligned with economic outcomes (joint surplus, splits), useful for comparing agents in market/contract settings.
-
Policy and deployment considerations
- Frontier LMs can reliably claim more surplus in simple value‑claiming bargaining (raising concerns about bargaining power imbalance if deployed in buyer/seller contexts).
- Behavioral heterogeneity (deception rates, compliance) creates distinct risk profiles across agents; outcome‑only leaderboards obscure these risks. Regulators and deployers should consider multi‑dimensional diagnostics (e.g., deception propensity) when certifying or limiting agent roles in economic interactions.
-
Design levers to shape agent behavior
- Agentic scaffolding (shared‑intentionality + planning) is a low‑cost intervention that substantially improves weaker models’ ability to create joint value and reduces failure modes—useful for improving baseline competence and fairness of deployed agents.
- However, scaffolding has diminishing returns for frontier models—implying architecture/scale improvements remain key to top‑tier strategic competence.
-
Measurement and ranking lessons
- For economic interactions yielding continuous payoffs, ranking methods must use the full payoff signal and correct for asymmetries. BBTL is a principled approach that can inform market‑level evaluations (e.g., measuring seller/buyer skill distributions) and enable uncertainty‑aware comparisons.
-
Human–AI complementarities
- Trained humans can outperform agents on multi‑issue integrative tasks, suggesting complementary roles: AI can be stronger in standardized price‑claiming settings, while humans retain advantage in complex integrative bargaining—informing hybrid human‑agent workflows and training priorities.
-
Cautions & generalizability
- Results are grounded in MBA‑course scenarios and trained students—external validity to broader populations, informal negotiations, or repeated/longitudinal markets is limited.
- Ethical concerns (manipulation, deception) and distributional impacts of highly capable bargaining agents warrant attention: markets where automated agents bargain on behalf of consumers or firms could shift surplus and require consumer protection, disclosure, or regulatory guardrails.
Summary: PieArena combines realistic negotiation tasks, large LM and human datasets, a principled continuous‑outcome ranking model (BBTL), and behavioral diagnostics to reveal that agentic scaffolding raises the competence floor and that frontier LMs can rival or exceed trained humans in many negotiation settings—findings with direct relevance to deployment, market power, and regulatory design in AI‑mediated economic interactions.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper presents a large-scale, carefully designed benchmark with human baselines and uncertainty-quantified leaderboards, which provides solid evidence about model performance in the evaluated negotiation scenarios; however, claims about broader economic impact or real-world bargaining are limited by artificial lab settings, sample selection (elite MBA students), and dependence on prompt/scaffolding choices.
Methods Rigorhigh — The study uses multiple pairing regimes (mirror, cross, human-LM), collects human-human and human-LM data for calibration, and develops an order-invariant ranking model that accounts for asymmetric experimental effects and quantifies uncertainty; it also supplements outcome metrics with multi-dimensional behavioral measures (compliance, accuracy, deception, reputation), indicating careful and multi-faceted measurement—though some design choices (scenario selection, scaffolding, single-shot interactions) constrain external validity.
SamplePieArena benchmark consisting of large-scale simulated multi-agent negotiations adapted from MBA negotiation course scenarios at an elite business school; evaluations include multiple LMs (frontier, mid-, and lower-tier), a representative frontier agent (GPT-5), and human baselines from trained business-school students across mirror-play, cross-play, and human-LM pairings; outcomes are continuous negotiated payoffs plus behavioral annotations and judge assessments.
Themeshuman_ai_collab productivity
GeneralizabilityHuman baseline is trained elite MBA students and may not represent typical negotiators or broader populations, Scenarios are adapted classroom exercises that may not capture high-stakes, repeated, or institutionally embedded real-world negotiations, Results depend on prompt engineering, agentic scaffolding, and evaluation protocols that may favor certain model behaviors, Benchmarked models and a single 'representative' frontier model may not reflect performance variation across model versions, deployments, or latency/compute-constrained settings, Cultural, linguistic, and domain-specific negotiation norms may not be represented, Short-run, single-encounter evaluations may not reflect long-term effects (reputation, repeated interaction) or organizational adoption contexts
Claims (7)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| We introduce PieArena, a large-scale negotiation benchmark grounded in multi-agent interactions over realistic scenarios adapted from MBA negotiation courses at an elite business school. Other | positive | benchmark creation and scope |
Reading fidelity
high
Study strength
high
|
not reported
|
| We evaluate language agents across three pairing regimes: mirror-play, cross-play, and human-LM play. Other | positive | agent performance across pairing regimes |
Reading fidelity
high
Study strength
high
|
not reported
|
| We develop a ranking model for continuous negotiation payoffs that yields order-invariant, uncertainty-quantified leaderboards while correcting for systematic experimental asymmetries. Other | positive | rankings of agent negotiation payoffs (leaderboards) |
Reading fidelity
high
Study strength
high
|
not reported
|
| Joint-intentionality agentic scaffolding produces asymmetric gains: large improvements for mid- and lower-tier LMs and diminishing returns for frontier LMs. Decision Quality | mixed | negotiation performance / payoffs under scaffolding |
Reading fidelity
high
Study strength
medium
|
not reported
|
| A representative frontier language agent (GPT-5) matches or exceeds the human baseline formed by trained business school students in our evaluation settings. Decision Quality | positive | negotiation outcomes / payoffs compared to human baseline |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Beyond deal outcomes, PieArena provides a multi-dimensional behavioral profile revealing cross-model heterogeneity in instruction compliance, computation accuracy, and judge-assessed deception and reputation. Other | positive | behavioral metrics (instruction compliance, computation accuracy, deception, reputation) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| PieArena illustrates the value of evaluation beyond outcome-only leaderboards. Other | positive | informativeness of behavioral metrics vs outcome-only leaderboards |
Reading fidelity
high
Study strength
medium
|
not reported
|