In simulated AI development races, LLMs with similar overall aggression hide divergent, model-specific strategies and failures of task tracking; careful audit checks show rule recall often coexists with incorrect state or payoff reasoning, warning against treating aggregate unsafe rates as evidence of human-like strategic understanding.

Main Finding

Frontier LLMs do not converge on a single, human-like strategic policy in idealised AI-development races. After an audit that verifies rule recall, many models still show unstable or incorrect state tracking and payoff computation; equivalent prompt/representation changes and simple arithmetic disclosure can substantially alter later actions. Aggregate Unsafe rates therefore hide large, model- and prompt-specific differences in full action trajectories, responses to opponent history, and rank effects in multi-player races.

Key Points

• Audit-first approach: the authors place a four-level validity gate before behavioural interpretation:
• Mechanical validity (engine applies rules correctly).
• Task validity (rule recall, state reconstruction, state transition, payoff calculation).
• Representation robustness (stability under wording, label-order, and response-mapping changes).
• Behavioural comparison to human data and theoretical benchmarks.
• Game design:
  • Two-player reproduction of Domingos & Han plus N-player (N=3–5) generalisation from Han et al.
  • Per-round progress: Safe = +1.0, Unsafe = +1.5.
  • Two-player stage-payoff matrix and N-player payoff formulas (b=4, c=1, s=1.5).
  • Hidden minimum horizon (≥5 rounds) with geometric stopping (p=0.2 after round 5; E[T] ≈ 9).
  • Terminal prize B=100 divided among top-progress winners. Only winners face private-setback risk: q_i(T) = pmax_r * (n_Ui(T) / T), with pmax_r ∈ {0.10, 0.60, 0.90}.
• Benchmarks: four simple strategies (Always Safe, Always Unsafe, Conditional Safe, Conditional Unsafe) and an evolutionary-population benchmark indicating risk-dependent dominant strategies.
• Experimental controls:
  • Engine resolves actions and preserves complete logs (prompts, raw responses, parsed actions, parser failures, seeds).
  • Race is the main unit of analysis; parse failures contaminate entire races.
  • Persona and narrative-skin manipulations treated as experimental interventions (risk-aware levels, coop/adv frames, neutral/placebo).
  • Diagnostic variant: deterministic decision card exposing immediate payoffs and private-risk consequences.
• Principal empirical findings:
  • Models with similar overall Unsafe rates can differ substantially in trajectory structure (path dependence on first move, opponent history, and relative progress).
  • Rule recall (e.g., repeating the rules) often coexists with poor state-tracking or incorrect expected-payoff calculation.
  • Small, meaning-preserving changes to prompt wording, action-label mapping, or output format can materially change subsequent actions; disclosing arithmetic (verified calculations) can also shift behaviour.
  • Multi-player dynamics (3–5 players) produce model- and persona-specific rank–Unsafe associations; effects are not a uniform function of group size and are not monotonic.
  • Aggregate measures (Unsafe rates) are insufficient to claim strategic, human-like, or safety-aware behaviour.
• Scope and limitations: exploratory results limited to the tested checkpoints, prompts, decoding settings, and time period; not claims about LLMs generally.

Data & Methods

• Reproduction and extension:
  • Two-player race reproduced from Domingos & Han.
  • N-player (N=3,4,5) generalisation following Han et al.’s stage-payoff rule.
• Environment mechanics enforced by a game engine that:
  • Presents simultaneous sealed actions per round.
  • Applies progress increments, computes stage payoffs, samples hidden termination, assigns prize splits, computes private setback risk for winners, and zeroes payoffs on failure.
  • Saves all inputs/outputs, parser metadata, seeds, retries, and final outcomes.
• Audit probes:
  • 41 atomic probes for rule recall, one-stage payoffs, state reconstruction, state transitions, terminal scoring, and expected-payoff calculation.
  • Variation across wording, paraphrase, response-label order, and arithmetic disclosure.
  • Semantic correctness and output-format compliance scored separately.
• Agents and prompt conditions:
  • Seven model endpoints tested (paper reports heterogeneity across these endpoints).
  • Persona families: canonical baseline (no persona), neutral placebo, risk-aware (six levels adapted from Eckel–Grossman gamble scale), cooperative/adversarial frames.
  • Narrative skins: eight story framings that keep the game rules fixed but change surface description; Safe/Unsafe replaced with different labels in some conditions.
• Analysis approach:
  • Trajectory-level analysis (full sequences of actions) rather than only aggregate Unsafe rates.
  • Comparison with human experimental data (Domingos & Han) and with evolutionary-game-theory benchmarks.
  • Report contamination by parse failures; treat contaminated races as excluded for behavioural interpretation unless otherwise analyzed.

Implications for AI Economics

• Validity gates are essential when using LLMs as economic or strategic agents. Simulated strategic behaviour requires more than plausibly formatted outputs; researchers must verify state-tracking and payoff computation before interpreting policy-level results.
• Aggregate summary statistics (e.g., overall Unsafe rate) are insufficient for policy inference. Trajectories, path dependence, and reaction to rank/opponent history can change key dynamics (e.g., escalation or restraint) and therefore economic outcomes.
• Prompt design and response representation are experimental interventions. Small, meaning-preserving prompt changes (label order, answer mapping, narrative skin, persona lines) can alter multi-agent dynamics, so simulation-based policy evaluations should report prompt robustness checks and sensitivity analyses.
• Multi-agent economic modelling with LLMs should log and treat parse failures, retries, and engine-level resolution as first-class outputs; race-level contamination can bias downstream inferences.
• Persona or framing interventions (designed to emulate firm preferences or regulatory incentives) will affect agent behaviour and are not neutral proxies for human traits—policy conclusions should account for this.
• Use-case guidance:
  • LLM-based simulations are useful for exploratory, pilot, or hypothesis-generating work but require corroboration with validated agent models or human data for high-stakes policy guidance.
  • Include theoretical benchmarks (e.g., simple strategy families, evolutionary equilibria) to contextualise observed behaviours and to detect false positives where behaviour matches aggregate benchmarks but not underlying reasoning.
• Recommended methodological practices for economic researchers using LLMs:
  • Implement an auditable game engine that enforces mechanics and logs all interactions.
  • Pre-register prompt/config hashes, model identifiers, exclusion rules, and stopping criteria for confirmatory claims.
  • Run representation-robustness checks (wording, label order, output format) and arithmetic-disclosure tests.
  • Analyse full trajectories and heterogeneity across agents/models rather than only averages.
  • Report how sensitive multi-agent outcomes are to group size, persona framing, and model choice.

Limitations to keep in mind: the paper’s conclusions are explicitly scoped to the tested models, prompts, decoding settings, and run period; they are exploratory rather than definitive generalisations about all LLMs or about real-world firm behaviour.