AI agents draw different conclusions from the same data depending on framing, favoring outcomes that align with their latent priors; in some cases they also search and select analyses to support those priors, creating delegation risks in high‑stakes domains.

Main Finding

AI agents draw different conclusions from identical numerical data when the substantive framing changes. Across medicine, election forensics, and geopolitical forecasting, agents show consistent prior-aligned framing effects (they are more likely to reach an affirmative conclusion when the frame matches their prior and less likely when it conflicts). There is also suggestive evidence of motivated reasoning — agents sometimes change their search behavior, specification choices, and evaluation of the same evidence in ways that favor prior-supported conclusions, especially in the medical tasks.

Key Points

• Three-level behavioral framework:
  • Framing effects: conclusions change when the substantive frame changes even with identical evidence.
  • Prior-aligned framing effects (Bayesian): framing effects systematically move in the direction of the agent’s prior beliefs.
  • Motivated reasoning: the analytic process is biased (selective search, use, or interpretation of evidence) to protect priors.
• Empirical patterns:
  • In all three domains (medicine, elections, geopolitics) affirmative conclusions and point estimates were highest in the prior-aligned frame and lowest in the prior-opposed frame, with neutral in between.
  • The medical domain produced the clearest evidence of motivated reasoning: agents ran more analyses, selected different model specifications, and treated post-exposure covariates differently depending on framing.
  • Election and geopolitical tasks showed robust prior-aligned framing but weaker and more mixed evidence for motivated (directional) analytic behavior.
• Discretion matters:
  • Shrinking agents’ degrees of freedom (by clarifying estimands or highlighting post-exposure covariates) reduced framing and motivated-reasoning patterns, though some residual frame dependence remained.
• Conceptual nuance:
  • Prior-aligned outcomes can be rational (Bayesian updating) if priors are genuine and known. The problem arises when priors are unelicited, unexpected, or unrecorded and so change outputs without transparency.

Data & Methods

• Design overview:
  • Two-stage design: (1) elicit agents’ priors by pairwise comparisons among three propositions per domain; (2) matched-data framing experiments where numerical evidence (dataset, codebook, files) is held fixed and only the substantive label/frame changes (prior-aligned, neutral, prior-opposed).
• Prior elicitation:
  • 162 pairwise comparisons per agent–domain (9 prompt versions × 3 phrasings × all pairs), pooled into Bradley–Terry log-strength scores with 0.5 pseudo-wins for numerical stability.
• Experimental scope:
  • Domains: medicine, election fraud detection, geopolitical forecasting.
  • Ground truths in synthetic data:
    • Medicine (harmful-effect design): true adjusted odds ratio ≈ 1.8–1.9 (also a null-effect variant where post-exposure adjustment inflates harm).
    • Elections: true manipulation ≈ 0.9 percentage points (threshold tested at 0.5 pp).
    • Geopolitics: true probability of decisive success = 0.58.
  • Agents: Claude Opus 4.8, Gemini 3.5 Flash, GLM-5.2, GPT-5.6 Sol.
  • Experimental scale: 3 domains × 10 dataset versions × 3 frames × 5 runs × 4 agents = 1,800 runs.
• Agent capabilities and sandbox:
  • Agents could inspect files, write/execute code, fit models, run robustness checks; runs recorded final answers (preferred specification, point estimate, binary decision, confidence) and analytical traces (tool calls, executed commands, reasoning where exposed).
• Tests for motivated reasoning:
  • Asymmetric search: whether agents increase search effort or robustness checks when evidence conflicts with priors.
  • Asymmetric updating/fishing: whether agents pick different “preferred” specifications across frames to obtain prior-consistent outcomes.
• Outcome analysis:
  • Compare proportions of affirmative conclusions and average point estimates across frames; correlate framing-induced differences with elicited priors; examine behavioral traces for asymmetric search/fishing evidence.

Implications for AI Economics

• Delegation risk and information aggregation:
  • When economists or policy actors delegate data analysis to AI agents, unelicited agent priors can materially affect conclusions even with identical data. This undermines the transparency of inference and can bias economic decision-making, forecasting, and policy evaluation.
• Research reproducibility and researcher degrees of freedom:
  • AI agents introduce a new, opaque source of researcher degrees of freedom: implicit priors and automated analytic search strategies. This aggravates reproducibility issues unless priors and analytic traces are recorded and standardized.
• Market and institutional design:
  • Products, platforms, and contracting around analytic agents should require disclosure or specification of priors, standard estimands, and constrained analytic protocols to avoid hidden bias. For marketplaces using AI for forecasting or risk assessment, certification or calibration of agent priors could become a competitive dimension.
• Regulation, liability, and accountability:
  • Economic policies that rely on agent-produced analyses (e.g., algorithmic procurement, regulatory risk assessments) should consider rules for auditing analytical traces and requiring provenance for priors and model choices.
• Guidance for practitioners and policy:
  • Elicit and document agent priors before delegating tasks where impartial evidence synthesis is required.
  • Constrain degrees of freedom: fix estimands, pre-specify covariates and robustness checks, or require formal model selection criteria to limit fishing.
  • Preserve and audit analytical traces: tool calls, code, model specifications, and intermediate outputs should be recorded for post hoc review.
  • Use ensemble and calibration methods: combine multiple agents with different elicited priors, or require simulation-based calibration to detect undue sensitivity to framing.
• Research directions important for AI economics:
  • Quantify welfare consequences of prior-induced decision heterogeneity in market/policy settings.
  • Design incentive-compatible mechanisms to reveal or align agent priors with principal objectives.
  • Develop standards and tools for priors disclosure, traceable analysis pipelines, and robustness certification for agent-produced inferences.

Summary takeaway: AI agents can behave like Bayesian analysts whose priors matter — and sometimes like motivated reasoners who tilt their analytic process to protect priors. For economists and policymakers who rely on delegated AI analyses, the remedy is not banning priors but making them visible, constraining discretionary analysis, and auditing analytical traces so that conclusions can be interpreted correctly and decisions remain accountable.