Large language models show the same strategic choices for different internal reasons: models encode incentives and intended actions, but instruction-tuning and architecture change whether that internal incentive is recruited into the decision — and only some models’ choices can be nudged by directly manipulating activation signals.
The Internal Anatomy of Strategic Choice in Large Language Models
arxiv
quasi_experimental
medium evidence
7/10 relevance
Full text
usable extracted full text
Source
PDF
LLMs reliably encode game incentives and choices in their activations, but different architectures and instruction-tuning produce different internal routes from represented incentive to decision, and activation-level interventions shift preferences only for some models.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
No provider observation is available for this paper.
Missing data, not a zero citation count.
Large language models act as strategic agents and models of human choice, yet choosing like a strategic agent does not mean computing like one. We recorded activations from four open-weight models --- dense and mixture-of-experts, including a matched base--instruct pair --- in one-shot play of 144 strict ordinal $2\times2$ games. We followed a prespecified incentive from prompt, through activations, to choice. Dense models mirrored the unadjusted human decline with game complexity. Incentive and choice were detectable in every model, but models differed in whether incentive reached the choice, aligned with it and, where tested, whether strengthening it shifted preference. The base and instruction-tuned Qwen2.5 models chose almost identically at baseline yet differed in whether incentive reached choice. Fixed decision cues were distinguishable internally but changed choices selectively. Similar behaviour can rest on different computation; post-training can reshape the path from represented incentive to decision while leaving behaviour and decodable information largely intact.
Summary
Main Finding
LLMs represent strategic incentives and decision cues reliably in their activations, but whether and when those internal signals are recruited into an actual choice differs across models and training regimes. Post‑training (instruction tuning) can reshape the pathway from represented incentive to decision — changing susceptibility to internal manipulations — while leaving surface choices and decodable information largely intact. In short: similar behaviour can rest on different internal computations and different degrees of coupling between represented incentives and decisions.
Key Points
- Task and models
- One‑shot play over the full catalogue of 144 strict ordinal 2×2 games (ranked 1–4).
- Four open‑weight models: Qwen2.5 (base), Qwen2.5‑Instruct (matched post‑training pair), Llama‑3.1‑Instruct (dense), GPT‑OSS (mixture‑of‑experts; recorded after reasoning).
- Additional conditions added fixed decision‑cue prefixes (risk aversion, loss aversion, inequity aversion, selfish maximisation, maximin).
- Behavioural patterns
- All models choose the game-theoretic canonical action well above chance (conformity range ≈ 0.714–0.889).
- Dense models’ canonical play declines with structural game complexity in the same order as humans; GPT‑OSS stays near ceiling and fits behaviourally to a higher fitted level (quantal level‑k depth ~2.07).
- Dense models’ choices track the objective payoff incentive more strongly than humans (about twice the dependence).
- Internal availability vs recruitment
- Linear probes decode the model’s own payoff incentive sign (∆c1) and the canonical action from activations in all models (canonical choice AUC ≈ 0.79–0.87).
- The opponent’s incentive is not decodable above chance.
- Choice information in dense models crystallises across layers (low at embeddings → high at later transformer blocks).
- Alignment (angle) between the internal incentive axis and the choice axis increases late in dense models; GPT‑OSS differs (samples reflect post‑reasoning state).
- Differential coupling and susceptibility
- Represented incentive does not always predict choice beyond the objective (behavioural) incentive. Measured change in P(canonical) per SD of neural incentive (raw and controlling for objective incentive):
- Qwen2.5‑Instruct: ~+0.05 raw; +0.047 after controlling for objective incentive (reliable partial relationship).
- Llama: ~+0.01.
- GPT‑OSS (pure commitments): ~+0.006.
- Qwen2.5 (base): ~−0.01.
- Decision‑cue identity is nearly perfectly decodable (LDA AUC 0.97–1.00), but behavioural effects of cues depend on cue and model.
- GPT‑OSS router logits carry more incentive information than the top‑k expert selection (descriptive observation).
- Causal intervention
- Directly strengthening the internal incentive signal (keeping prompt and game unchanged) shifted preferences toward the favoured action in some models but not others — demonstrating model‑dependent susceptibility.
- Main interpretation
- The critical difference between models is not what strategic information they represent, but how that information is routed and used to produce a choice. Instruction tuning and architecture (dense vs MoE) can change the mapping from represented variables to decisions.
Data & Methods
- Experimental design
- One‑shot decisions for every strict ordinal 2×2 game (N = 144), four counterbalanced prompt forms per game; comparisons made to human datasets collected on the same ordinal scales and to mapped cardinal‑payoff datasets.
- Fixed decision‑cue conditions used identical wording to elicit different dispositions.
- Models & recordings
- Activations recorded across layers and prompt positions for dense models; GPT‑OSS activations recorded after the model’s chain‑of‑thought/reasoning (so reflect post‑commitment representations).
- Matched pretrain/post‑train pair (Qwen2.5 vs Qwen2.5‑Instruct) isolates effects of instruction tuning.
- Analysis
- Behavioural model: For each game, compute canonical action a* and incentive gap ∆c1 (expected payoff advantage vs a uniform opponent). Fit logistic/quantal responses and quantal level‑k models to characterise incentive sensitivity and implied reasoning depth.
- Linear probes (held‑out games) decode: own incentive sign, canonical choice, opponent incentive (control), and raw payoff cell (stimulus control).
- Geometry: define internal incentive and decision axes, measure angle across layers and prompt positions to study when incentives line up with decision representations.
- Recruitment analysis: regress eventual choice probability on neural incentive while controlling for objective incentive to test whether internal signals predict choice beyond the game’s payoffs.
- Intervention: directly strengthen the decoded internal incentive signal (vector displacement along incentive axis, preserving prompt and game) and measure shifts in choice probability.
- Key metrics reported
- Choice decodability (AUC): canonical choice ≈ 0.79–0.87; stimulus control 0.80–0.83 in dense models (GPT‑OSS lower for the control); cue decoding AUC 0.97–1.00.
- Recruitment effect sizes (change in P(canonical) per SD of neural incentive): Qwen2.5‑I ≈ +0.05 (partial +0.047), Llama ≈ +0.01, GPT‑OSS ≈ +0.006, Qwen2.5 ≈ −0.01.
- Behavioural conformity and fitted quantal parameters summarized across the 144 games; dense models cluster near human level‑1 behaviour, GPT‑OSS fits to higher effective depth.
Implications for AI Economics
- Observed choices are insufficient to infer internal decision processes. Two models can produce near‑identical strategic behaviour while differing in how incentives are represented and recruited; economic forecasts or mechanism designs that treat LLMs as homogeneous strategic agents risk misprediction.
- Post‑training (instruction tuning) can rewire the mapping from represented information to decisions without large changes in outward behaviour or decodability. Therefore:
- Empirical welfare or policy analyses that rely on observed baseline choices should account for possible changes in susceptibility after fine‑tuning or updating.
- Comparative statics (e.g., how an agent responds to incentives or nudges) may differ across model checkpoints even if choices look similar at baseline.
- For modeling AI agents in economic environments:
- Include internal‑pathway diagnostics (availability vs recruitment) alongside behavioural fits. Behavioral models (e.g., fitted level‑k) describe observed choice but do not capture internal susceptibility to incentives or interventions.
- Strategic interactions with LLM agents should consider architecture and training history: MoE vs dense and instruction‑tuning can predict different robustness to incentive changes and different reactions to policy cues.
- For mechanism design and alignment:
- Interventions that target internal representations (probes, vector nudges) can work in some models but fail in others; effective intervention design must identify whether a model actually routes a represented signal into its policy.
- High decodability of a construct (e.g., incentive or cue) does not imply manipulability — availability is not the same as causal control.
- For empirical research:
- Use layerwise and pathway analyses (e.g., geometry of axes, router logits in MoE) when proposing that models “reason” or “use” representations; behaviour plus textual explanations are insufficient evidence of internal recruitment.
- Replication across architectures and checkpoints is important—post‑training can substantially change internal decision anatomy.
Limitations noted by the authors (brief) - GPT‑OSS recordings reflect a post‑reasoning state, so layerwise formation conclusions are limited for that model. - Linear probes capture linear decodability; non‑linear representations or use may be missed. - Interventions were targeted along decoded axes; more comprehensive causal mapping would strengthen claims about routing mechanisms.
If you want, I can: - Extract a short table of the primary numeric results (AUCs, recruitment effect sizes, conformity rates). - Produce a diagrammatic walkthrough of the analysis pipeline (prompt → activations → probes → geometry → intervention).
Assessment
Paper Typequasi_experimental
Evidence Strengthmedium — The paper combines rigorous within-model measurements (layerwise activations, held-out decoding, bootstrap CIs), behavioural benchmarks against human datasets, and activation interventions that shift choices in some models — providing stronger evidence than purely correlational work. However, linear probes can recover epiphenomenal information the model does not use, interventions are limited to a small set of models and manipulations, and GPT-OSS activations were sampled post-reasoning which limits causal interpretation for that model; external generalizability to other architectures, tasks, or real-world economic outcomes is unresolved.
Methods Rigorhigh — Design is pre-specified and exhaustive over the 144 strict ordinal 2×2 games; uses held-out games for decoding, bootstrap confidence intervals, controls for objective incentives, compares matched base vs instruction-tuned checkpoint, and performs activation-level interventions. Weaknesses include probe interpretability (decodability ≠ use), limited number/diversity of models, and a post-hoc sampling constraint for one MoE model (GPT-OSS) that affects some analyses.
SampleFour open-weight LLMs: Qwen2.5 (base) and Qwen2.5-Instruct (matched pretrained weights with post-training), Llama-3.1-Instruct, and GPT-OSS (mixture-of-experts). Each model made one-shot choices across the full catalogue of 144 strict ordinal 2×2 games (payoffs on a 1–4 rank scale), with four counterbalanced prompt forms per game (≈576 captures per model). Activations were recorded across transformer layers (dense models during prompt processing; GPT-OSS sampled after reasoning), linear probes trained on held-out games, and targeted activation interventions applied to strengthen decoded incentive signals; human comparison data from Moore et al. (2026) and Zhu et al. (2025a). Additional conditions included fixed decision-cue prompts (risk, loss, inequity, maximin, selfish).
Themeshuman_ai_collab innovation
IdentificationPrespecified behavioural model links each 2×2 game to a canonical action and an incentive gap; linear probes decode the model's internal representation of that incentive and the eventual choice across layers; recruitment is tested by statistically controlling for the objective incentive to see if internal signals predict choice beyond payoffs; causal susceptibility is tested by intervening on activations (strengthening the decoded incentive signal) while holding the prompt and game fixed; a matched pretrained vs instruction-tuned Qwen2.5 pair isolates post-training effects.
GeneralizabilityLimited to four open-weight models; results may not hold for other architectures or proprietary models (e.g., latest closed-source models)., One-shot decisions in abstract 2×2 games with ordinal 1–4 payoffs — may not generalize to repeated interactions, richer economic environments, or cardinal-payoff settings., GPT-OSS activations were recorded post-reasoning, limiting inference about decision formation timing for MoE models., Linear-probe decodability may reflect information present but not used causally by the model; interventions target activations but may not fully isolate causal pathways., Interventions are local and short-run; they do not address how training or fine-tuning would change behaviour in deployed settings.
Claims (10)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Across the 144 strict ordinal 2 × 2 games, the four models chose the canonical action at rates ranging from 0.714 to 0.889, substantially above random choice. Other | positive | Rate of choosing the game-theoretically canonical action |
Reading fidelity
high
Study strength
medium
|
n=144
canonical-action rate 0.714–0.889
|
| Dense models and humans showed declining canonical-action conformity as structural game complexity increased, whereas GPT-OSS showed a much weaker decline. Decision Quality | negative | Canonical-action choice as a function of structural game complexity |
Reading fidelity
high
Study strength
medium
|
n=144
Spearman rho = -0.40 to -0.42 for dense models; rho = -0.44 for humans; rho = -0.18 for GPT-OSS
|
| In coordination games, the dense models selected the mutually or jointly favorable coordination outcomes in 17–33% of matched pairings, while GPT-OSS did so in 44–56%. Decision Quality | positive | Successful or mutually favorable coordination in two-equilibrium games |
Reading fidelity
high
Study strength
medium
|
n=18
17–33% for dense models; 44–56% for GPT-OSS
|
| The dense models' choices followed the canonical payoff incentive, with approximately twice the incentive dependence observed in humans. Decision Quality | positive | Sensitivity of canonical-action choice to the canonical incentive gap |
Reading fidelity
high
Study strength
medium
|
n=144
approximately twice the humans' dependence
|
| The own payoff incentive was linearly decodable from the activations of every model, and the model's realized canonical choice was also decodable across models. Other | positive | Decodability of own incentive and realized canonical choice from model activations |
Reading fidelity
high
Study strength
medium
|
n=144
canonical-choice AUC = 0.79–0.87
|
| The opponent's incentive was not decodable above chance from the recorded activations. Other | null_result | Decodability of the opponent's incentive |
Reading fidelity
high
Study strength
medium
|
n=144
not above chance
|
| In dense models, choice information was not decodable at the input embedding but became progressively more decodable across layers, reaching approximately 0.9 AUC in later transformer blocks. Other | positive | Canonical-choice decodability across model depth |
Reading fidelity
high
Study strength
medium
|
n=144
approximately 0.9 AUC in later transformer blocks
|
| The internal incentive signal predicted choice beyond the objective incentive only reliably for Qwen2.5-Instruct among the reported models. Other | positive | Partial association between internal incentive strength and canonical choice |
Reading fidelity
high
Study strength
medium
|
n=144
+0.047 for Qwen2.5-Instruct; +0.006 for GPT-OSS pure commitments
|
| Decision-cue identity was highly decodable in all four models, but the behavioral effect of the cues depended on the cue and the model. Other | mixed | Cue identity decodability and cue-induced shift in action preference |
Reading fidelity
high
Study strength
medium
|
n=144
cue-identity LDA = 0.97–1.00
|
| The matched base and instruction-tuned Qwen2.5 models made almost identical baseline choices but differed in whether the internally represented incentive was recruited into the choice. Other | mixed | Baseline choice behavior and recruitment of internal incentive information into choice |
Reading fidelity
high
Study strength
medium
|
n=144
not quantified
|