LLM-based economic agents overweight idiosyncratic signals and downweight aggregates, with multiple concurrent signals crowding out marginal influence; LoRA fine-tuning dampens but does not remove these expectation biases, and updating differs between household and CEO roles.

Main Finding

LLM-based agents update expectations in systematically biased ways that can be captured by a Behavioral Kalman Filter (BKF). Across 720 controlled trials with three state-of-the-art LLMs and two agent personas (Household, CEO), the authors find (1) prior and signal weights do not sum to one (violating rational Kalman updating), (2) agents weight micro (idiosyncratic) signals more than macro signals, (3) concurrent signals interact negatively (a “cognitive discount” that reduces marginal weight of each signal), and (4) expectation-formation differs by persona (CEOs put relatively more weight on macro signals). LoRA fine-tuning reduces but does not eliminate these behavioral deviations.

Key Points

• Four consistent behavioral patterns:
  • Weights on prior + signals deviate from the rational-sum-to-one condition.
  • Micro-level shocks receive substantially higher weight than aggregate (macro) shocks.
  • A negative interaction term between micro and macro signals (βint < 0) — concurrent information sources reduce each other’s marginal impact.
  • Persona matters: CEO persona is relatively more sensitive to macro signals than Household persona.
• Quantitative highlights (Bayesian posterior means; 95% credible intervals reported in paper):
  • Household βprior ≈ 0.48–0.65; βmic ≈ 0.40–0.88; βmac ≈ 0.39–0.65; βint ≈ −0.01 to −0.03.
  • CEO βprior ≈ 0.57–0.89; βmic ≈ 0.57–1.03; βmac ≈ 0.49–0.83; βint ≈ −0.01 (sometimes ~0).
  • Interaction coefficients are small in magnitude but consistently negative for many cases, supporting the “cognitive discount” claim.
• Fine-tuning (LoRA on Qwen2.5-7B-Instruct):
  • Reduces response volatility.
  • Makes the interaction term insignificant for the Household persona but not for CEOs.
  • Does not restore the rational-sum-to-one property.

Data & Methods

• Experimental design:
  • 2 (Persona: Household vs CEO) × 2 (Micro: +5% / −5%) × 2 (Macro: +5% / −5%) factorial → 4 scenarios per persona (consistent boom, consistent bust, two conflict cases).
  • Baseline prior expectation fixed at 3.0%; shock magnitude Δ = 5.0%.
  • Models: GPT‑4o, Gemini 1.5 Flash, DeepSeek‑V3. For LoRA experiment, Qwen2.5‑7B‑Instruct.
  • 30 independent trials per model × persona × scenario; total N = 720 (3 models × 2 personas × 4 scenarios × 30).
  • Prompting: persona priming + numeric shocks; zero-shot setup; structured JSON output with numeric Updated Expectation and a Rationale (post-hoc Chain-of-Thought style field).
  • Generation temperature = 0.7 and model state reset between trials.
• Theoretical framework:
  • Behavioral Kalman Filter (BKF): extends Kalman Filter by (i) prior discount α (<1 allowed) and (ii) subjective covariance between micro and macro signals via ρ (non-zero allowed), capturing signal interference.
  • Reduced-form mapping: Bayesian linear regression of updated expectation on prior, micro shock, macro shock, and micro×macro interaction: xt|t = βprior·xt|t−1 + βmic·smic + βmac·smac + βint·(smic·smac) + ε.
  • Estimation: Bayesian linear regression with weakly informative priors (β ∼ N(0, 10^2 I)), MCMC to obtain posterior means and 95% Highest Density Intervals.
• Diagnostics / outputs:
  • Descriptive statistics (means and SDs) across scenarios and models.
  • Posterior estimates of βprior, βmic, βmac, βint map to BKF parameters α and perceived signal covariance.

Implications for AI Economics

• Modeling AI agents as economic actors:
  • LLMs do not implement textbook Bayesian/Kalman updating by default; they exhibit predictable behavioral deviations. Calibrating ABMs or policy simulations that use LLM agents requires incorporating these biases (e.g., BKF-style priors and subjective signal covariances).
• Persona and context matter:
  • Identity priming (Household vs CEO) systematically alters signal weights. When deploying LLM agents in simulations or automated decision systems, agent-role specification will change aggregate dynamics.
• Multiple signals are not additive in practice:
  • The negative interaction implies diminishing marginal value of additional signals in agent updating. Models that assume additive, independent information channels may overstate the informativeness of multi-source news.
• Fine-tuning helps but is insufficient:
  • LoRA reduces variance and some interaction effects but fails to recover fully rational updating. Reliance on simple fine-tuning to “fix” behavioral biases is unlikely to be enough for high-stakes forecasting or policy analysis.
• Practical recommendations:
  • Validate LLM-based agents’ expectation-updating behavior against BKF-like reduced forms before using them in economic simulations.
  • If using LLMs for forecasting/agent-based policy experiments, explicitly model and calibrate α (prior discount) and ρ (subjective signal correlation).
  • Consider training/regimen interventions beyond light fine-tuning (e.g., structural objective changes, multi-step Bayesian training) to approximate normative updating.
• Research avenues:
  • Extend to dynamic multi-period updating (recursions), richer shock distributions and magnitudes, more personas (firms by industry, investor vs consumer), and larger model families.
  • Structural estimation of full BKF (rather than reduced-form) and experiments on interventions (different fine-tuning regimes, RLHF objectives, ensemble methods) to mitigate biases.

Limitations to note - Shock design is limited to ±5% and a single baseline; generality across shock sizes and multi-step learning is untested. - Only three main LLM architectures were used (plus one for LoRA); results could vary with other models/versions. - Reduced-form regression proxies the BKF parameters — a full structural identification would be more demanding but could yield sharper causal inference.

If you want, I can: - Produce a one-page figure-ready summary (key tables/coefficients highlighted) for presentations. - Suggest specific calibration values for BKF parameters to use when integrating LLM agents into an agent-based macro model.