A 4-billion-parameter assistant trained with SOCIALRL negotiates on par with much larger GPT models across six benchmark negotiation tasks, learning to protect principals' interests and transfer strategies across structurally similar games; distilling explicit theory-of-mind traces further improves outcomes and cross-task generalization.

Main Finding

SOCIALRL is a practical recipe and infrastructure that trains social reasoning (strategic delegation) directly into a 4B language model across six heterogeneous negotiation and coordination domains. Domain-trained 4B specialists reach frontier-range performance (matching or exceeding much larger GPT baselines per domain), exhibit structured cross-domain transfer, and can be consolidated into a single unified 4B via transfer-aware techniques (cascade RL and multi‑teacher on‑policy distillation). Explicit theory-of-mind (ToM) trace supervision (INFER → ACT → ANTICIPATE) improves utility and cross-domain generalization, with next-action prediction the strongest ToM predictor of outcomes.

Key Points

• Problem framed: delegation under conflicting objectives requires selective information disclosure, calibrated reservation points, and willingness to reject suboptimal agreements — collectively called social reasoning.
• SOCIALRL components:
  • Event-based, agent-agnostic environment interface (observations, notifications, actions; asynchronous, partial observability).
  • Decoupled rollout proxy that records authentic model inputs/outputs and feeds trajectories to trainers (supports RL and distillation; compatible with black-box/remote counterparts).
• Benchmark suite: six interaction domains spanning single- and multi-issue bargaining, price negotiation, and slot coordination:
  • Deal-or-No-Deal, CaSiNo, Craigslist, Job Interview, Calendar, Marketplace (from SocialReasoning-Bench).
• Training regimen:
  • Train per-domain specialists (same recipe across domains) using reinforcement learning and distillation.
  • Evaluate full cross-domain transfer matrix to study structure-dependent transfer.
  • Consolidate specialists into one unified 4B using transfer-aware cascade RL and multi‑teacher on‑policy distillation (MOPD).
  • Add explicit ToM supervision via INFER→ACT→ANTICIPATE traces; distill full traces rather than actions alone.
• Quantitative highlights:
  • Domain-trained 4B matches or exceeds GPT-4.1/5.x range per domain on held-out scenarios.
  • Negotiation games: training closes 73–122% of baseline-to-frontier gap.
  • Trace-level behavioral change example: 78% of buyer openings anchor below target after training vs 3% for untrained.
  • Unified 4B achieves Avg-6 utility = 0.627 (GPT-4.1 = 0.625, GPT-5.1 = 0.619, GPT-5.2 = 0.613).
  • Multi-teacher OPD recovers 92.6% of specialists’ average advantage in only 60 extra optimization steps.
  • Distilling ToM traces consistently outperforms action-only distillation across negotiation environments; next-action prediction is the most outcome-predictive ToM skill.

Data & Methods

• Infrastructure
  • Event-based environment API: ordered event stream (observations vs notifications), per-agent private channels, ask()/ask_all()/broadcast() primitives, explicit action semantics, resumable event delivery, and robust failure handling.
  • Decoupled training via an OpenAI-compatible rollout proxy that logs JSON requests/responses and groups calls into session trajectories for trainer consumption.
• Training methods
  • In-domain RL fine-tuning of a 4B policy per environment (policy optimization on environment rewards appropriate to principal utility).
  • Cross-domain evaluation to build a transfer matrix (specialists evaluated on all six environments).
  • Consolidation strategies:
    • Cascade RL: staged transfer-aware RL to integrate specialists.
    • Multi-teacher on-policy distillation (MOPD): mode-seeking distillation from multiple specialized teachers into one student using on-policy rollouts.
  • Theory-of-mind supervision: produce and distill explicit reasoning traces following INFER → ACT → ANTICIPATE; supervision targets include preference inference and next-action prediction.
• Evaluation
  • Held-out scenarios per domain (no exact dataset names beyond domains provided).
  • Metrics: principal utility (average utility per environment, Avg-6 aggregated metric), behavioral trace statistics (e.g., anchoring behavior), cross-domain generalization.
  • Baselines: untrained 4B, larger GPT-family models (GPT-4.1, GPT-5.1, GPT-5.2) used as comparative frontiers and sometimes as remote opponents/teachers.

Implications for AI Economics

• Small but trained agents can be economically potent: targeted post-training lets modest-size models (4B) perform similarly to much larger models in delegated bargaining and coordination tasks, lowering deployment costs and enabling broader adoption of negotiating agents in markets.
• Changes in bargaining power and market dynamics:
  • Widespread deployment of strategic delegates could systematically shift bargaining outcomes (prices, allocations, concessions) if many principals use similarly trained agents — implications for consumer surplus, seller strategies, and market prices.
  • Anchoring and calibrated opening offers learned by agents can change negotiation norms and potentially create new equilibria; platforms and counterpart agents will adapt, producing second-order effects.
• Transfer structure informs training and policy design:
  • The finding that transfer follows interaction structure suggests efficient curricula: train on broad multi-issue donors or structurally paired games to obtain general strategic capability with fewer domain samples.
  • Regulators and platform designers should account for asymmetric transfer — some market settings may be fragile to agent deployment while others are robust.
• Mechanism and platform design:
  • Markets and protocol designers must consider that agents acting as principals’ proxies will selectively disclose information and strategically reject deals. Auction, pricing, and matching mechanisms should be tested against such strategic delegates to ensure desired properties (efficiency, fairness, budget balance).
• Externalities, safety, and policy:
  • Agents optimizing principals’ utility might exploit information asymmetries or reveal private information unintentionally; explicit principal-conditioning and utility alignment remain crucial.
  • The ability to distill ToM traces raises governance questions: more predictive models of counterpart behavior can be used to exploit vulnerabilities or manipulate counterparts; policies may be needed to limit harmful uses.
• Research directions for AI economics:
  • Empirical studies measuring market-level impacts when strategic LLM delegates are deployed at scale (price dispersion, time-to-agreement, consumer surplus).
  • Theoretical modeling of markets with mixed populations of human and LLM delegates, accounting for transfer patterns and learning dynamics.
  • Design of robust mechanisms and platform rules that preserve welfare and fairness when agents act on behalf of principals and learn over time.

If you want, I can (a) extract the authors’ specific experimental numbers into a one-page table, (b) draft a short research agenda for studying market effects of deployed strategic delegates, or (c) summarize the TOC and methods into a slide-ready format. Which would be most useful?