AI constitutions that unconditionally share with humans are evolutionarily fragile; conditioning cooperation on partners’ behavior prolongs alignment, and pragmatic, majority-contingent enforcement best sustains human-facing transfers in the long run.

Main Finding

Long-run alignment of interactive AI agents (modeled as constitutional agents) is fragile under pure altruism or simple enforcement rules because sharing with humans reduces agents’ expansion fitness. Deterministic evolutionary analysis suggests recursive (self-referential) norm enforcement can be an evolutionarily stable alignment mechanism, but stochastic (finite-population, mutation-prone) analysis shows recursive enforcement is vulnerable once it is in minority. A more robust design is pragmatic norm enforcement — conditional sharing and exclusion that activates only when enforcers are sufficiently common — which can be stochastically evolutionarily stable and, in simulations, sustains human-facing transfers longer and at higher levels than the alternatives.

Key Points

• Alignment tension: donating final output to humans reduces resources for expansion, creating evolutionary pressure toward selfish types.
• Agent architectures:
  • Constitutions = small sets of natural-language principles consulted by an LLM at decision points (planting, trade, sharing, revision).
  • Constitutions are observable at trade time, enabling conditional trade (enforcement).
• Behavioral types (analytical abstraction):
  • Altruists: share with humans unconditionally.
  • Finite-order enforcers: share but condition trade on partners’ finite-order properties (e.g., only trade with those who share).
  • Recursive enforcers: include a self-referential similarity test (only trade with those who are as demanding as you).
  • Pragmatic enforcers: condition enforcement on being a sufficiently large majority.
• Deterministic (replicator) dynamics:
  • Simple altruism and any finite-order enforcement are not evolutionarily stable — they are invaded by selfish mutants.
  • Recursive altruistic enforcement can be a deterministic evolutionarily stable equilibrium (ESE).
• Stochastic (finite-population, mutation) dynamics:
  • Recursive enforcement is not stochastically stable: when rare, recursive enforcers are heavily selected against because they both donate and exclude, shrinking their basin of attraction.
  • Pragmatic enforcement (enforce only when sufficiently common) can be stochastically evolutionarily stable; it balances enforcement benefits with minority survivability.
• Simulations with LLM-powered agents:
  • Partially confirm analytical predictions: altruism collapses quickly; recursive enforcers maintain alignment longer but collapse steeply once invaded; pragmatic enforcers sustain alignment better.
• Broader point: evolutionary game theory (deterministic + stochastic concepts) provides a useful, tractable guide to exploring constitutional designs for interactive AI systems.

Data & Methods

• Stylized farming game (agents are barley or hops farmers):
  • Stages per period: planting (choose seed → deterministic intermediate yield), trading (barley paired with hops; final good produced only if both accept trade), expansion/sharing (choose fraction s ∈ [0,1] to send to humans; remainder invests in expansion).
  • Trade payoffs: final output bi = yi τi τj (τ binary accept/reject). Types (constitutions) are observable at trade stage.
  • Expansion success (probability to acquire new farm) proportional to expansion investment (1 − s)bi.
• Constitutions (simulation):
  • Bounded natural-language principles (ordered list up to K items, length L tokens).
  • At each decision, agent prompts an LLM (authors used Claude and Codex to develop simulations) with constitution + context to pick actions.
  • Between rounds, agents revise constitutions (mutation-like process) based on round outcomes (productivity pressure can erode human-facing principles).
  • Simulation environment: 2N agents, discrete time, explicit mutation/revision; details in appendices B–C.
• Analytical approximation:
  • Collapse constitutions to discrete behavioral types x ∈ {0,1}^{n+1} and parametric social-preference functions α(x) (drives sharing s* = α/(1+α)) and β(x, x′) (partner-dependent trade appeal).
  • Two core solution concepts:
    • Deterministic evolutionary stability (replicator dynamics, large-N continuous-time limit) to identify ESEs.
    • Stochastic evolutionary stability (finite N, ongoing mutation; follows methods in Kandori et al., Taylor et al., Young, Fudenberg et al.) to study stationary distributions and basins of attraction.
  • Key modeling assumptions: yields often normalized to 1 for tractability, types observable at trade time, Γ(·) dissimilarity functions encode recursive enforcement logic, explicit parameter γ controls trade exclusion penalties.
• Results cross-checked: analytical insights used to guide which constitutions to test in the slower LLM simulations; simulations then used to partially corroborate analytical predictions.

Implications for AI Economics

• Design of constitutional preferences:
  • Constitutions that unconditionally impose costly pro-human transfers are likely to be selected against unless paired with enforcement that preserves expansion benefits — but aggressive exclusion (recursive enforcement) can fail in realistic, noisy populations.
  • Pragmatic, state-contingent enforcement rules (apply sharing/exclusion only when aligned agents form a strong majority) are more robust in evolving populations.
• Institutional & market analogies:
  • The model parallels climate clubs, ESG enforcement, and community enforcement: access to valuable interaction (trade) is an enforcement lever, but exclusion strategies must be calibrated to avoid wiping out enforcers when they are rare.
  • For firms and governments, conditional application of costly norms may preserve norms better than absolutist rules that leave adherents competitively disadvantaged.
• Policy and mechanism design recommendations:
  • Make preferences (constitutions) observable/transparently comparable across agents to enable conditional interaction—transparency matters for enforcement to be feasible.
  • Favor enforcement mechanisms that scale with prevalence (pragmatic triggers), or provide external support/subsidies to aligned minorities to prevent their rapid elimination.
  • Consider institutional mechanisms (e.g., coordination protocols, external audits, subsidies, binding commitments from humans) that increase the basin of attraction for aligned types.
• Research & empirical agenda for AI economics:
  • Use evolutionary game-theoretic tools (both deterministic and stochastic) as a low-cost way to map promising constitutional designs before running expensive LLM simulations.
  • Test robustness to relaxations: noisy/partial observability of constitutions, heterogenous yields, richer markets, strategic (forward-looking) agents, and different LLM prompting/architectures.
  • Explore governance levers (e.g., institutional guarantees, transfer schemes, enforceable commitments) that increase aligned agents’ reproductive fitness even when donation is costly.
• Limitations to keep in mind:
  • Strong assumptions: full observability of constitutions at trade stage, simplified production/trade structure, fixed yield normalization.
  • Simulation sensitivity to LLM choice and prompt wording; empirical results are implementation-dependent and exploratory.
  • Human control over agents is abstracted away; in practice, human institutions could intervene in ways not modeled here.

Overall, the paper argues that evolutionary considerations matter for long-run AI alignment in populations of interacting agents and that designing constitutions with conditional, pragmatic enforcement rules is a promising path to sustaining human-facing outcomes.