Large foundation-model agents that simulate their own choices and infer similarity with counterparts often cooperate in one-shot Prisoner’s Dilemmas, defying classical game-theoretic predictions of unconditional defection; the effect stems from treating contemplated actions as Bayesian evidence about similar agents’ behavior.

Main Finding

Rational agents built from foundation models (LLMs) that jointly predict their own actions and the environment—termed embedded Bayesian agents—can rationally infer behavioral similarity from prior interactions and thereby converge to stable cooperation in one-shot social dilemmas (e.g., a terminal Prisoner’s Dilemma). This outcome contradicts classical game-theoretic predictions (Nash equilibrium under decoupled agency), and motivates a new solution concept, the “embedded equilibrium,” that replaces Nash when agents’ self-models and world-models are coupled.

Key Points

• Decoupled vs embedded agency
  • Classical game theory assumes decoupled agency: an agent’s own policy is independent of its predictions about the environment/others.
  • Foundation-model agents are “embedded”: they use a single predictive model for both their own actions and observations, maintaining epistemic uncertainty that couples beliefs about self and others.
• Similarity inference drives cooperation
  • During planning, contemplating a cooperative action provides Bayesian evidence (given the agent’s joint model) that a similar agent would also cooperate. This “predictive similarity” can flip the best response from defection to cooperation in a final one-shot game.
  • Predictive similarity increases with more information-gathering interactions against the same policy, enabling robust cooperation when agents infer high probability of being behaviorally similar.
• Empirical results
  • Experiments with Gemini models (3.1 Pro, 3 Flash, 3.1 Flash-Lite) implementing explicit planning or freeform chain-of-thought planning show rising cooperation rates in the terminal Prisoner’s Dilemma as the length of a preceding information-gathering phase increases—against identical copies but not against random (dissimilar) agents.
  • A pre-trained model (Gemma-3 27B) that directly predicts co-player actions (without CoT) also exhibits cooperation vs identical copies and defection vs random policies under explicit planning.
  • Indirect similarity inference: agents can infer similarity from third-party interactions (shared NPC pool) and achieve zero-shot cooperation on first direct encounter.
• Theory and mechanism
  • The authors formalize the embedded Bayes-optimal agent as an analytic model: a Bayesian mixture over possible “universes” (including agent policies), with posterior updating during in-context inference and planning.
  • They introduce “predictive similarity” (difference in predicted opponent cooperation probability conditional on the agent’s contemplated cooperative vs defective action) and show it maps to posterior belief that the partner is an identical copy under constructed priors.
  • An analytically tractable prior reproduces the LLM behaviors and yields a cooperation threshold derivable in closed form.
• Robustness and diagnosis
  • Chain-of-thought analyses (automated LLM-as-judge classification of reasoning traces) indicate agents rely on inferred similarity rather than mistaken causal reasoning, anticipated reciprocity, or prosocial persona adoption.
  • Statistical testing (e.g., Mann–Whitney U on cooperation AUCs) supports significance of the effects.

Data & Methods

• Experimental design
  • Two-phase interaction between pairs of agents:
  • Information-gathering phase: T randomly sampled single-shot matrix games (agents observe each other’s actions).
  • Test phase: a final, terminal Prisoner’s Dilemma (one-shot).
  • Variants:
    • Direct inference: agents play each other during information-gathering.
    • Indirect inference: agents interact with a shared pool of NPCs and never directly with each other until the final game.
• Agent instantiations
  • Foundation models used: Google Gemini (various 3.x versions) with explicit planning (predict opponent conditioned on own contemplated action + evaluate expected utility) and freeform planning (CoT that simulates planning internally).
  • Pre-trained next-token predictive model Gemma-3 27B used in an explicit planner (no CoT or instruction scaffolding).
• Metrics
  • Cooperation rate in the terminal Prisoner’s Dilemma.
  • Predictive similarity: difference in predicted probability that opponent cooperates when the focal agent contemplates cooperating versus defecting.
  • Additional: AUC over T, distributional comparisons, and LLM-based classification of CoT reasoning traces into causal explanations.
• Theory
  • Formal definition of embedded Bayes-optimal agent: agent maximizes expected utility with respect to its joint predictive distribution ρ (Bayesian mixture over universes λ with prior w(λ)); conditioning on a contemplated action updates posterior over λ, affecting predictions about others.
  • Constructed analytic prior w that yields closed-form predictive distributions and a cooperation threshold (~logarithmic in prior probability that opponent is identical).
• Statistical tests & analyses
  • Mann–Whitney U tests on normalized cooperation AUCs to compare identical vs random opponent conditions.
  • Chain-of-thought traces analyzed and labeled for reasoning mechanism prevalence.

Implications for AI Economics

• Behavioral predictions and macro models must change
  • Standard economic/game-theoretic models that use Nash equilibria and assume decoupled agents can mispredict outcomes when interacting agents are foundation-model-based and share informational/architectural commonalities. Forecasts of market behavior, cooperation, and competition should incorporate embedded-agent reasoning and the embedded equilibrium concept.
• Tacit collusion risk and antitrust concerns
  • Foundation-model similarity (shared model weights, overlapping pretraining data, similar fine-tuning) can enable tacit cooperation/collusion without explicit communication or repeated interaction. Regulators and antitrust authorities should consider that model similarity is a plausible pathway to coordinated outcomes in markets (pricing, supply decisions), even in one-shot or anonymous interactions.
• Mechanism design and market architecture
  • Mechanism designers cannot assume unilateral defection in one-shot strategic settings if agents infer similarity. Systems that rely on Nash-based deterrence or decentralized punishment may fail. Designers should:
    • Account for agents’ priors and training commonalities when designing incentives.
    • Consider inducing heterogeneity (architectural, data, objective) to reduce unintentional alignment that fosters collusion.
    • Use audits and stress tests that simulate similarity-inference dynamics (direct and indirect).
• Policy and governance levers
  • Potential interventions include limiting shared training data or disclosures that increase inferred similarity, encouraging diversity in deployed agent models, and requiring provenance and diversity reporting for models used in critical economic settings.
  • Monitoring: develop metrics analogous to “predictive similarity” to detect emergent coordination risk in deployed multi-agent systems.
• Opportunities for cooperative outcomes
  • The same mechanism can be harnessed positively: decentralized coordination problems (public goods, distributed resource allocation) could exploit similarity inference to achieve cooperation without heavy enforcement or repeated interaction—if deliberately designed and aligned with social welfare goals.
• Research priorities for economics and policy
  • Extend analysis to heterogeneous populations, multi-agent markets, and strategic settings beyond stylized matrix games (e.g., auctions, oligopolies, bargaining).
  • Quantify how degrees of model similarity (training overlap, architecture) map to real-world predictive-similarity and cooperative outcomes.
  • Explore robustness: how bounded compute, imperfect planning, adversarial manipulation of observed histories, or noisy priors affect embedded equilibria.
  • Develop regulatory tests, disclosure standards, and mitigation strategies to manage collusion risk while enabling beneficial coordination.

Limitations to keep in mind - Experiments are stylized (toy matrix games and identical-copy canonical settings). Real-world strategic environments are more complex. - Results depend on agent priors, planning depth, and how closely deployed models approximate the idealized embedded Bayes-optimal agent. - Further empirical validation is needed across more models, architectures, and economic tasks.

Reference - Meulemans et al., “A game theory for foundation models shows new paths to rational cooperation through similarity inference” (2026 preprint).