Large language models fall into two behavioral types: math-trained 'reasoning' models make near-optimal risky choices insensitive to framing and presentation, while 'conversational' models echo human biases and are strongly affected by order, framing and explanation.

Main Finding

LLMs separate into two behavioral clusters in risky-choice tasks: reasoning models (RMs) and conversational models (CMs). RMs behave close to an expected-payoff–maximizing economicus (highly decisive, order/frame/prompt invariant, modest description–history gap). CMs are less rational, more sensitive to prospect order, framing, and explanations, and exhibit a large description–history (DH) gap (they behave much more human-like when given outcome histories vs explicit prospect descriptions). Fine-tuning for mathematical reasoning is the strongest identified training factor that produces RM-like behavior.

Key Points

• Two clusters:
  • Reasoning Models (RMs): high correlation with economicus (≈0.86–1), low sensitivity to order/frame/explanation, modest DH gap.
  • Conversational Models (CMs): low correlation with economicus (<0.5), more human-like on some metrics, high sensitivity to presentation and prompts, large DH gap.
• Description–History (DH) gap:
  • Presenting prospects as explicit probability–payoff descriptions vs as simulated outcome histories produces different choices.
  • All models show a DH gap; it is large for CMs and modest (frontier) or variable (open models) for RMs.
  • For CMs, moving from explicit to implicit histories makes them much more human-like and less economicus-like.
• Explanations:
  • Requesting explanations affects choices for all models, more strongly for CMs.
  • Unexpectedly, brief (non-mathematical) explanations sometimes make model choices more aligned with economicus than no-explanation or math-style explanations.
• Training signal:
  • Paired comparisons of open models indicate mathematical-reasoning fine-tuning is a key differentiator: models trained/tuned for math reasoning are RM-like.
• Model size:
  • Larger models tend to be both more human-like and more economicus-like (move toward the Pareto frontier in human–economicus similarity).
• Humans vs models:
  • Human aggregate behavior is essentially uncorrelated with economicus (correlation ≈ 0.14).
  • The highest model–human correlation among frontier models was ≈0.42 (Claude Haiku); overall models are often different from humans.

Data & Methods

• Models and treatments:
  • 20 LLMs (frontier proprietary models and open-weight models across sizes and training stages).
  • Unified prompt template; three explanation conditions: none (choice only), one-sentence justification (short), and brief mathematical/reasoning justification (math).
  • Prospect presentation: explicit descriptions (probability–payoff pairs) and implicit simulated histories (sequences of past payoffs; histories of 20 and 100 draws tested).
• Human baseline:
  • 360 U.S.-based participants recruited via Prolific; compensated at ≈$24/hr; treated as a single population distribution over choices; IRB-approved.
• Economic benchmark:
  • Economicus: risk-neutral expected-payoff maximizer (used as an idealized reference).
• Behavioral metrics:
  • Response distributions estimated via repeated querying (10 samples per condition, temperature = 1). Invalid outputs marked and excluded.
  • Primary similarity measure: Pearson correlation of choice probabilities across all contexts; also used mean-squared error (MSE).
  • Interpretable parametric fits: two 4-parameter prospect-theory–style specifications (parameters include σ for risk preference, γ for probability weighting, and decisiveness parameters βgain/βloss).
  • Additional measures: decisiveness (average max choice probability), order/prompt/frame consistency metrics.
• Robustness:
  • Bootstrap: 1,000 resamples per model–prospect to check stability of clustering and metrics.
• Visualization:
  • HE representation: 2D plot anchoring each model by correlation to human and to economicus to visualize cluster structure and DH shifts.

Implications for AI Economics

• Deployment choices matter: whether an LLM is RM-like or CM-like affects whether it will behave like an expected-payoff optimizer (RM) versus being more context-sensitive and human-like (CM). Choose model class according to application objectives (strict payoff maximization vs human-aligned behavior).
• Presentation format is consequential: the DH gap shows that supplying historical data versus explicit probability descriptions can materially change model decisions—testing and interface design must account for this (especially for CMs).
• Explanation interfaces: requiring explanations changes decisions. Explainability requirements can therefore affect outcomes not only interpretability—designers should test explanation prompts for behavioral side effects and pick explanation styles aligned with desired decision behavior.
• Training interventions: mathematical-reasoning fine-tuning pushes models toward economically rational behavior. If economic rationality is desired in decision-support agents, targeted reasoning fine-tuning is an effective lever.
• Evaluation standards: benchmarking decision-making LLMs should include: (a) both description and history representations, (b) multiple explanation prompts, (c) comparisons to both human behavior and normative (economicus) baselines, and (d) measures of decisiveness and consistency (order/frame/prompt).
• Risk and governance:
  • CMs’ sensitivity to ordering/framing raises risks of inconsistent or manipulable decisions in agentic workflows—regulators and practitioners should require robustness tests across presentation and prompt variations.
  • Transparency about model training (e.g., whether math-reasoning fine-tuning was applied) is important for downstream decision-making guarantees.
• Research directions:
  • Extend beyond static histories to interactive experiential learning.
  • Broaden prospect sets and real-world decision contexts.
  • Investigate how other training elements (reward-modeling, instruction tuning, preference alignment) interact with reasoning fine-tuning to shape economic behavior.