Embedding literature-derived causal chains as explicit Chain-of-Thoughts makes LLM agents behave more like human respondents in consumer-expectation data and yields more realistic macro dynamics in an agent-based simulator; improvements are meaningful but tempered by model sensitivity and potential data-leakage concerns.

Main Finding

LLM-driven agents used in macro agent-based models (ABMs) often fail to reproduce human microeconomic decision-making: they show weak responsiveness to macroeconomic trends, severe collapse of behavioral heterogeneity, and (in some models) reliance on memorized training data. The paper introduces the Heterogeneous Shock-Response Causal Transmission (HSRCT) framework — an interpretable, literature-grounded mediator–moderator causal graph plus a path-searched Chain-of-Thought (CoT) injection — that meaningfully improves micro-level alignment with human survey responses (from the Survey of Consumer Expectations) and yields more realistic macro-level simulations in the EconAgent testbed.

Key Points

• Diagnosis (using SCE panel data, Mar 2016–Dec 2023):
  • Trend responsiveness: Most LLMs fail to track human temporal response patterns to macro indicators (unemployment, CPI, federal funds rate); GPT-5 performs best but still imperfectly.
  • Heterogeneity collapse: LLM responses concentrate around “average” replies (mode collapse), producing much lower variance than the human population.
  • Data leakage: Including explicit dates in prompts boosts performance for models with pre-training cutoffs (e.g., GPT-3.5) in the pre-cutoff period (indicating memorization). GPT-5 appears to internalize mappings making leakage harder to detect.
• HSRCT intervention:
  • Constructs a three-layer causal graph L0 (Shocks) → L1 (Mediators) → L2 (Decisions).
  • Mediators (L1) are generated by LLMs, edges carry sign (+/−), mediation strength, and a textual mechanism explanation.
  • Each edge is verified against academic literature via automated Google Scholar queries (SerpApi) and pruned if unsupported.
  • Moderation: For each edge, top-2 demographic moderators are identified and assigned moderation strengths (scalar in [−1,1]) to model heterogeneity.
  • Inference: shocks are activated with direction and intensity; a profile-conditioned, strength-guided path search scores candidate paths by cumulative strength S(P)=σ_A·w(A,B)·w(B,C) with w(i,j)=s_mediation·∏(1+δ_m); the strongest causal path P* is turned into an explicit causal CoT and injected into the agent prompt to produce decisions.
• Empirical results:
  • Micro-level: HSRCT increases Spearman correlations with human trends and restores response distribution variance (lower KL divergence), reducing mode collapse.
  • Macro-level: When plugged into EconAgent ABM, aggregated dynamics better replicate historical economic trends beyond reproducing stylized facts.
• Practical choices & evaluation details:
  • Models evaluated: GPT-3.5 Turbo (chosen as backbone to limit leakage), GPT-5, Gemini 2.5 Flash, Claude 3.5 Haiku.
  • Tasks: Spending Forecast (probability of household spending change over next 12 months) and Offer Acceptance (acceptance probability for sub-expected salary offers).
  • Prompt inputs: respondent demographics, past 4 months of macro indicators, previous responses/memory, task question; run 5 times per prompt and average; temperature/top-p = 1.0.
  • Two-stage validation: (1) reproduce micro behavior on SCE; (2) generalize in dynamic macro simulation (EconAgent).

Data & Methods

• Data:
  • Survey of Consumer Expectations (SCE) panel: ≈1,100 U.S. household heads, 24 waves (Mar 2016–Dec 2023); curated ~26.6k spending samples and ~26.8k labor samples.
  • Macro indicators: unemployment rate, CPI, federal funds rate (preceding 4 months).
• Models and prompting:
  • Evaluated multiple LLMs; used “w/ date” vs “w/o date” prompt variants to probe leakage.
  • Averaged over 5 stochastic queries; high sampling (temp/top-p=1.0) to encourage behavioral diversity.
• HSRCT construction:
  • L0: predefined shocks (interest rate, inflation, unemployment).
  • L1: mediators generated by LLM (e.g., liquidity constraints, consumer confidence).
  • L2: decision nodes (task-specific).
  • Edge attributes: sign, mediation strength in [0,1], mechanism explanation.
  • Literature verification: automated search of top-5 relevant papers per edge (SerpApi/Google Scholar); edges without support pruned.
  • Moderation: choose top-2 profile attributes per edge; assign moderation strengths δ ∈ [−1,1] across attribute strata.
• Path selection & decision generation:
  • Shock activation returns signshock and intensity σ ∈ [0,1].
  • Effective edge weight w(i,j)=s_mediation(i,j)·∏(1+δ_m) over relevant moderators.
  • Compute cumulative strength S(P) and direction D(P) via sign propagation; choose P* maximizing S(P).
  • Construct a causal CoT from P* and put it into the prompt to bias mechanistic reasoning.
• Evaluation metrics:
  • Spearman correlation between aggregated LLM responses and human trends.
  • Distribution comparisons (KL divergence) to measure heterogeneity.
  • Macro-level fit of simulated series against historical trends.

Implications for AI Economics

• Need for micro-foundations: Valid macro ABM outputs require that micro agents’ decision mechanisms align with empirical human behavior; reproducing stylized facts alone is insufficient for reliable policy analysis.
• Interpretable, theory-grounded interventions help: Embedding mediator–moderator causal structure, verified against literature, makes agent reasoning mechanistic and interpretable (vs. opaque pattern-matching).
• Heterogeneity matters: Modeling demographic moderation explicitly is crucial to avoid mode collapse and to capture differential sensitivities across groups — important for distributional policy analysis.
• Leakage caution: Evaluations must control for training-data leakage; date- or event-based prompt cues can mask memorization and give a false sense of alignment. Using pre-cutoff models (or other leakage controls) helps diagnose true reasoning capacity.
• Practical trade-offs:
  • Benefits: Improved micro alignment and macro realism, better interpretability for policy contexts.
  • Costs/limits: HSRCT relies on LLMs to generate causal structures and on external literature retrieval; this adds computational/engineering complexity and potential sources of error (mis-specified mediators or mis-assessed literature support).
• Directions for future work:
  • Broader validation across countries, alternative surveys, and more behavioral domains (e.g., savings, labor supply dynamics).
  • Robustness checks under adversarial or out-of-distribution shocks.
  • Automated uncertainty quantification for mediator strengths and moderation parameters.
  • Integrating empirical parameter estimation (e.g., from causal inference on panel data) to supplement LLM-generated strengths and reduce model-dependence.

Limitations noted by the authors: reliance on LLMs for causal graph construction and literature parsing (introduces model-induced errors), evaluation concentrated on U.S. SCE tasks and selected shocks, and potential residual leakage in advanced models (e.g., GPT-5) that complicates attribution of performance to genuine reasoning.