Base language models better reproduce population-level diversity when sampled as individual respondents, while post-trained models are superior at directly estimating aggregate distributions — suggesting different model choices for text-generating simulation versus distributional estimation.

Main Finding

The paper distinguishes two tasks for using LLMs to simulate public opinion—emulation (generating individual responses that aggregate into a population distribution) and estimation (directly predicting the population distribution)—and shows a clear tradeoff: base (pretrained) models are better emulators (produce sampled, text outputs whose aggregated distribution and demographic structure more closely match human surveys), while post-trained/instruct-tuned models are better estimators (produce more accurate, well-calibrated distributional predictions when asked directly).

Key Points

• Emulation vs Estimation
  • Emulation: model generates open-text responses as if an individual respondent; population distribution emerges from sampling and mapping responses to survey categories.
  • Estimation: model is prompted to return a probability distribution (e.g., JSON) over the survey answer options directly.
• Empirical result summary
  • Base models outperform matched post-trained variants on emulation error (both TVD and Wasserstein) across all evaluated models and demographic conditions.
  • Post-trained models outperform base models on estimation error and approach or beat uniform baseline consistently; frontier model Claude Opus 4.6 achieved the best estimation accuracy.
• Demographic structure
  • Base models preserve demographic pairwise structure better (Spearman ρ ≈ 0.61–0.75, p < 0.02) but compress inter-group distance to ≈70% of human magnitude.
  • Post-trained models show weaker alignment (ρ ≈ 0.35–0.59) and tend to exaggerate inter-group differences (~2×), consistent with persona/stereotype collapse.
• Mechanism / interpretation
  • Post-training concentrates the model into a single “Assistant” persona that can be good at metacognitive verbalization (estimating distributions) but reduces output diversity needed for faithful emulation.
  • Base models retain a broader mixture of personas in generation, yielding more heterogeneous sampled outputs that better reflect within-group spread.
• Robustness & methodology
  • Authors favor open-response emulation (with an LLM judge mapping free text to categories; κ = 0.66 vs expert annotator) to avoid positional bias found in first-token extraction. First-token analyses in appendix corroborate main findings.
  • Post-trained emulation exhibits mode/persona collapse (8–14× higher bigram similarity), not solved by higher sampling temperature.
• Limitations noted by authors
  • Unclear whether the emulation–estimation tradeoff persists at frontier base-model scales (frontier base variants not publicly available).
  • Temporal mismatch: evaluation uses Pew 2019 survey data while model pretraining spans up to 2024–25, so models may reflect temporally blended opinions.
  • Study confined to survey-opinion items; multi-turn or richer behavioral tasks remain to be tested.

Data & Methods

• Dataset: 59 four-option economic-opinion items from Pew Research Center American Trends Panel Wave 54 (fielded Sep 16–29, 2019). Refusals dropped and distributions renormalized.
• Demographic conditions: marginal (unconditioned) + six groups (Democrat, Republican, Very Liberal, Very Conservative, Upper-income, Protestant).
• Models evaluated: three matched base/post-trained pairs (Qwen3-14B, Olmo-3-7B, Olmo-3.1-32B) plus Claude Opus 4.6 as a frontier estimation reference.
• Emulation pipeline:
  • Open-response interviewer-participant dialogue; demographic conditioning via a preceding turn (base) or system instruction (post-trained).
  • Responses mapped to survey options by an LLM judge; agreement with expert annotator Cohen’s κ = 0.66.
  • Sampling with temperature (post-trained higher temp to increase diversity); aggregated to empirical distribution.
• Estimation pipeline:
  • Prompted with multiple-choice items and asked to return a probability distribution (JSON).
  • Post-trained models evaluated with a single greedy sample (stable); base models averaged across multiple samples due to variability.
• Metrics:
  • Total Variation Distance (TVD) — distributional divergence.
  • Wasserstein distance — respects ordinal proximity of answer options.
  • Also assessed bigram similarity to detect mode collapse and Spearman correlations over pairwise inter-group TVD for structural alignment.
• Key quantitative outcomes:
  • Base models consistently had lower emulation Wasserstein and TVD than matched post-trained models.
  • Base model inter-group distances ≈ 70% of human; post-trained ≈ 200% (exaggeration).
  • Post-trained models had much better estimation error; estimation error decreased with model scale.

Implications for AI Economics

• Choose model type by downstream objective
  • If task requires generated respondent text (synthetic survey respondents, qualitative responses, agent-based simulations, multi-turn interactive agents), use base/pretrained models because they better preserve within-group variance and demographic structure in sampled outputs.
  • If task requires accurate aggregate forecasts (polling estimates, point forecasts of market shares or opinion proportions), use post-trained/instruct-tuned models which produce better calibrated distributional estimates when asked directly.
• Practical applications in economics and policy
  • Market research and demand estimation: use post-trained estimators for point predictions of population shares, willingness-to-pay categories, or sentiment proportions.
  • Policy counterfactuals and scenario testing with synthetic agents: prefer base models to capture distributional heterogeneity within demographic groups, reducing risk of overstating polarization or exaggerating subgroup differences.
  • Agent-based models and simulated populations: base models are preferable when you need diverse generated behaviors; however, validate aggregated statistics against real data and consider calibration using estimation-mode outputs if needed.
• Measurement & inference cautions
  • Persona collapse in post-trained models can bias subgroup heterogeneity estimates and exaggerate differences, which would distort welfare or inequality analyses if uncorrected.
  • Compressed inter-group differences in base models could understate true heterogeneity, biasing subgroup-targeted policy evaluations toward conservatism.
  • Temporal misalignment between model training data and current populations can mislead forecasting; economic researchers should either use temporally aligned datasets or treat model outputs as temporally blended priors requiring adjustment.
• Methodological recommendations
  • Use both paradigms where feasible: generate synthetic respondents with base models (emulation) and cross-check or recalibrate aggregated predictions with post-trained estimators.
  • Prefer open-response emulation (with validated mapping) over first-token probability extraction to avoid positional and label-ordering confounds when the goal is to generate interpretable text.
  • Report both TVD and Wasserstein (or other metrics sensitive to ordinal structure) when comparing model outputs to empirical distributions.
  • Validate any deployed synthetic-population-based decision tools against recent, real survey data and stress-test subgroup-sensitive outcomes to detect persona collapse or compression.
• Ethical and policy considerations
  • Simulated demographic responses risk reinforcing stereotypes—measure and preserve opinion spread rather than only modal behavior.
  • Deployment of synthetic respondents for policy design, pricing, or market segmentation must include fairness and validation checks; misrepresenting subgroup opinions can lead to harmful or inequitable decisions.
• Research directions relevant to AI economics
  • Investigate hybrid pipelines: use base models for text generation and diversity, with post-trained models for calibration, or develop post-training methods that preserve generative diversity while improving estimation.
  • Study whether the emulation–estimation tradeoff persists at larger base-model scales or with different post-training regimes (e.g., persona-preserving fine-tuning).
  • Explore temporal adaptation methods (continual fine-tuning or modular disentanglement) to ensure models reflect contemporary economic conditions for forecasting tasks.