AgenticPay, a new multi-agent negotiation benchmark, finds modern LLMs frequently fail to negotiate efficient, welfare-maximizing deals in simulated buyer–seller markets, exposing gaps in long-horizon strategic reasoning; the open framework aims to standardize research on language-driven agentic commerce.

Main Finding

AgenticPay is a scalable benchmark and simulator that frames multi-agent buyer–seller negotiation as a language-grounded market game. It exposes that current LLMs can generate coherent negotiation dialogue but differ substantially in economic performance: proprietary large models (Claude Opus 4.5, Gemini-3-Flash, GPT-5.2) achieve high deal rates and strong GlobalScores, while smaller/open models lag, and all models show notable weaknesses in long-horizon strategic reasoning, role asymmetries, and handling complex many-to-many markets.

Key Points

• Problem framed: negotiation as a finite-horizon, multi-round language game where buyers/sellers have private reservation values and negotiate via natural language; dialogue is parsed to structured actions (e.g., price proposals).
• AgenticPay components: Environments (10 realistic business scenarios across 4 domains), Tasks (111 tasks across 8 multi-agent configurations scaling buyers/sellers/products), Agents (role-specialized LLM policies with private states and memory), and Metrics (GlobalScore, BuyerScore, SellerScore plus deal rate, timeout, overflow, average rounds).
• Task variety: from bilateral 1B-1S bargaining to N-to-N matching markets; supports parallel and sequential negotiation modes and multi-product settings.
• Evaluation protocol: deterministic decoding (temperature=0, seed=0), max 1024 token responses; identical prompts for open and closed models; open-source inference run on 4 A800 GPUs.
• Scoring design (high level): utilities normalized within bargaining zone Z = pmax − pmin; GlobalScore emphasizes balanced surplus split (quality term Q = 4·rb·rs), all scores discounted by negotiation round to reward speed, and failures penalized.
• Empirical headline results (aggregate over 111 tasks):
  • Claude Opus 4.5 — GlobalScore 86.9, Deal rate 100%, Avg rounds 3.7
  • Gemini‑3‑Flash — GlobalScore 82.2, Deal rate 100%, Avg rounds 4.8
  • GPT‑5.2 — GlobalScore 81.7, Deal rate 100%, Avg rounds 3.8
  • Qwen3‑14B — GlobalScore 63.9, Deal rate 79.3%, Avg rounds 7.8
  • Llama‑3.1‑8B — GlobalScore 32.5, Deal rate 51.4%, Avg rounds 15.0
• Observed patterns:
  • Large proprietary models reach high deal rates and reasonable surplus splits; smaller/open models time out or propose infeasible prices more often.
  • Systematic asymmetries between buyer and seller roles (scores differ by role).
  • Longer horizons and multi-party complexity substantially degrade performance for weaker models and reveal strategic planning limits even for strong models.
• Release: code and dataset are publicly available (GitHub: SafeRL-Lab/AgenticPay).

Data & Methods

• Environments and scenarios:
  • 10 business scenarios grouped into: Daily Life (used phone/car, vacation rental), Professional Services (website, photography, renovation), Business Procurement (SaaS, raw materials), Financial Assets (luxury watch, acquisitions).
  • Product values span a wide monetary range (~$350 to ~$120k).
• Tasks:
  • 111 tasks across a complexity ladder: bilateral, multi-seller, multi-buyer, multi-product, and full market (many-to-many).
  • Interaction regimes: parallel (reason about concurrent negotiations) and sequential (decide to continue/switch/commit).
• Agents:
  • Buyers/sellers implemented as LLM-based policies with private reservation prices injected into prompts (kept confidential).
  • Each agent maintains dialogue memory (role, content, round).
  • Messages include free-form language and structured price proposals; a parser extracts actions and termination.
• Metrics and scoring:
  • Normalize buyer/seller utilities in bargaining zone. Quality metric favors balanced splits. Round-based discounting (γ) rewards faster deals; explicit failure penalty discourages deadlocks.
  • Also report deal rate, timeout rate, price overflow (proposing prices outside bounds), and average termination round.
• Experimental setup:
  • Unified inference-only protocol, deterministic decoding, identical prompt templates for buyer & seller (prompts provided in paper).
  • Models benchmarked: GPT‑5.2, Claude Opus 4.5, Gemini‑3‑Flash (proprietary); Qwen3‑14B, Llama‑3.1‑8B (open-weight).
  • Open-source inference on 4 A800 GPUs; each task executed once per model.

Implications for AI Economics

• Evaluation advances:
  • AgenticPay fills a gap by combining language, private information, multi-round bargaining, and many-to-many market structure—enabling more realistic evaluation of LLMs as economic agents than numeric-auction or short-bargain benchmarks.
  • The dialogue-to-action grounding and welfare-oriented metrics provide principled tools for measuring feasibility, efficiency, and distributional outcomes.
• Capabilities and limits of LLM negotiators:
  • Strong LLMs can reach deals reliably, but success does not guarantee normative economic desiderata (e.g., Pareto efficiency, equitable surplus splits) nor robust long-horizon strategy.
  • Performance depends on model scale and training; smaller/open models struggle with coordination, time management, and feasibility constraints—raising concerns about deploying less capable models in automated marketplaces.
• Mechanism and market design:
  • Language-mediated negotiation changes the strategy space (bluffing, persuasion, side commitments). Mechanism designers must account for communicative strategies and private-language channels when analyzing market outcomes and incentive compatibility.
  • Markets with LLM agents may require new rules or protocols (e.g., standardized action extraction, logging of structured offers, limits on off-protocol commitments) to maintain efficiency and prevent manipulation.
• Welfare, fairness, and regulation:
  • Systematic role asymmetries suggest some agent classes (buyers vs sellers) may be consistently disadvantaged when automated agents operate without careful calibration—implications for fairness and consumer protection.
  • High deal rates alone are insufficient as safety signals; regulators should consider whether LLM-mediated markets produce socially desirable allocations, and require auditability of reservation-value handling and agreement formation.
• Research directions:
  • Improve long-horizon strategic reasoning via specialized training (self-play, multi-agent RL), explicit planning modules, or hybrid systems combining LLMs with numeric optimizers.
  • Robust parsing and grounding of language to structured actions to avoid infeasible proposals and overflow errors.
  • Explore mechanisms to limit collusion, ensure truthful revelation of constraints, and evaluate equilibrium concepts in language-rich markets.
  • Study heterogeneity (agent types, risk preferences), informational asymmetries, and dynamics of learning/adaptation in repeated markets.
  • Consider computational and carbon costs: larger models perform better but at higher cost; trade-offs between performance, cost, and deployability matter for real markets.
• Practical cautions for deployment:
  • Even high-performing LLMs can fail in complex multi-agent markets or over long horizons—operators should add safeguards (time limits, verification steps, human-in-the-loop checks).
  • Transparent logging of dialogues, parsed actions, and reservation-value usage is critical for dispute resolution and regulatory compliance.

Short summary: AgenticPay provides a practical, extensible benchmark exposing where LLMs succeed and fail as autonomous negotiators. It highlights both promise (high deal rates from top models) and risk (strategic, long-horizon, and fairness failures), yielding concrete research and policy priorities for AI-driven market design.