Large language models cannot reliably simulate intersectional survey respondents: two-feature personas usually reduce to one dominant identity and often ignore race and religion, meaning synthetic samples misrepresent intersectional opinion structure.

Main Finding

Large language models do not reliably simulate intersectional demographic identities. When prompted to answer as members of two- or three-feature demographic subgroups, LLMs typically “collapse” the persona onto one (or at most two) identity dimensions rather than integrating the multiple identities additively as real respondents do. This failure is large, robust across eight models and multiple prompting/readout paradigms, and systematically downweights race and religion—the strongest real drivers of opinion.

Key Points

• Scope and scale
  • Evaluation against every real intersectional subgroup (n ≥ 20) in 15 waves of Pew’s American Trends Panel (ATP).
  • 15.7 million simulated response distributions in the primary pipeline; 21.1 million across all conditions; eight LLMs tested (7B open models through current frontier models).
• Core quantitative results
  • For two-feature personas, a single feature’s simulated bias explains the model’s pair output better than the additive combination in 75–82% of subgroups (raw win rates across models).
  • Calibrated “collapse index” (rescaling wins between additive-truth and collapse-truth endpoints) places models about 0.83–0.95 of the way to pure collapse (e.g., GPT-4o-mini ≈ 0.86–0.87).
  • Humans: pair identities are approximately additive. Real subgroups' weights center near full weight on both features (median (α, β) ≈ (0.95, 0.95), total α+β ≈ 1.83).
  • Models: pair biases concentrate on a one-identity budget (median α+β ≈ 0.98), typically dominated by one feature (median dominant share ≈ 0.88).
  • Real intersections grow more distinctive with intersection (noise-corrected squared distinctiveness rises ~2.5× from one to three features).
• Systematic patterns
  • Model hierarchies of which dimensions matter are wrong and compressed: LLMs tend to flatten steering magnitudes across dimensions, exaggerating some weak signals and often failing to amplify the strongest real drivers.
  • Race and religion—among the strongest real predictors in ATP—are disproportionately ignored in multi-feature conditioning; party and gender are over-retained relative to their true importance.
• Robustness
  • The collapse survives: aggregate-count readouts, individual-sampling readouts, and log‑prob/logit readouts; alternate prompts, role-frames, and chain-of-thought instructions do not meaningfully fix composition.
  • Results persist under stricter ground-truth cell size filters (n ≥ 200) and across model families and prompt variants.
• Diagnostic baselines and calibration
  • The paper uses three methodological safeguards that materially affect interpretation:
  • Human sampling-noise floor to avoid spurious findings from small-real-cell noise.
  • Null calibration for similarity metrics (a random unrelated dimension already attains median cosine ≈ 0.84), so absolute similarities are not interpretable without a baseline.
  • Split-sample confirmation across waves to ensure replication.

Data & Methods

• Data
  • Ground truth: respondent-level microdata from 15 waves of the Pew American Trends Panel (U.S. English, closed-form items).
  • Demographic dimensions: seven dimensions (e.g., race, religion, party, gender, age, income, education), yielding one-, two-, and three-way intersectional cells with at least n ≥ 20 respondents (larger-n subsets used for robustness).
• Measurement constructs
  • Steering vector (sg): sg = pg − ppop (how subgroup answer distribution deviates from population).
  • Bias vector (eg): eg = ˆpg − pg (difference between modelled and real subgroup answers).
  • Pair-composition contest: for a two-feature pair AB, compare eAB (realized pair bias) against two hypotheses:
    • Additive hypothesis: eA + eB (sum of single-feature biases).
    • Collapse hypothesis: the best single-feature bias (either eA or eB).
  • Scoring by cosine similarity; wins are calibrated using synthetic additive-truth and collapse-truth cells built from measured single-feature biases plus empirically derived noise.
• Calibration and noise controls
  • Synthetic endpoints give baseline win rates when truth is additive (best-single still wins ~40.3% due to noise/post-hoc max) and when truth is pure collapse (best-single wins ~84.3%). Observed rates are rescaled between these endpoints to form the collapse index.
  • Null baselines for cosine similarity computed (random unrelated dimension median cosine ≈ 0.84).
  • Split-sample and cluster-bootstrap procedures used to produce confidence intervals and avoid overfitting to specific waves.
• Model and elicitation variants
  • Eight models spanning 7B open-weight families to frontier models (examples include GPT-4o-mini, GPT-5.5, Claude variants, Gemma-2-9B).
  • Readout paradigms: aggregate histogram of 1,000 simulated respondents; repeated single-roleplay draws (≈100 personas per cell); logit-based probability readout (OpinionQA style).
  • Prompting variations: neutral/system framings, chain-of-thought, explicit instructions to integrate identities—none materially changed collapse behavior.

Implications for AI Economics

• Synthetic respondents are unreliable for intersectional inference
  • Economists should not assume LLMs can substitute for probability sampling of small intersectional subgroups. Synthetic surveys will likely under-represent joint identity effects and misattribute or ignore key drivers (notably race and religion).
  • Policy analysis, welfare estimates, or distributional predictions that depend on interaction effects across demographics (e.g., minority subgroups with particular political affiliations) risk systematic mismeasurement if based on LLM-generated synthetic samples.
• Market research and demand estimation
  • Using LLMs to cheaply “create” rare consumer personas (e.g., Black conservative women in a local market) will likely yield outputs dominated by a single salient identity, biasing heterogeneity and interaction estimates used in segmentation, targeting, or willingness-to-pay studies.
• Fairness, bias audits, and regulatory impact assessments
  • Audits that rely on silicon sampling to probe intersectional harms will understate intersectional effects and may miss the dimensions where harms concentrate (race, religion). Regulators and practitioners should treat silicon-sampling audits as incomplete unless rigorously validated against ground truth.
• Econometric practice and model-based simulations
  • LLM-based simulation can be informative for marginal (single-dimension) analyses only with caution. For interaction effects, researchers should prefer real cross-tabulated data, hierarchical models (e.g., MAIHDA), or explicitly estimated interaction terms calibrated on real samples.
• Practical recommendations for practitioners and researchers
  • Do not deploy LLM synthetic respondents as a substitute for real intersectional survey data without validation against joint empirical distributions.
  • If LLMs are used for exploratory work, report their limitations: validate which dimension the model is actually using, show collapse indices, and quantify uncertainty relative to noise baselines.
  • Adopt the paper’s suggested audit standards when evaluating silicon sampling: (1) control for human sampling noise; (2) calibrate similarity metrics against nulls and synthetic endpoints; (3) confirm findings on split samples/waves.
  • Where real intersectional data are unavailable, invest in targeted probability sampling or in statistical modeling approaches that combine limited real data with structural assumptions rather than relying purely on LLM outputs.
• Broader economic inference caution
  • Because the collapse is model- and prompt-invariant and seen across architectures, the issue is structural to present LLM conditioning behavior. Economic conclusions that depend critically on multi-way heterogeneity should not rest on unvalidated LLM simulations.

If you want, I can: - Extract a short checklist you can use when considering LLM-based silicon sampling for a project. - Produce a one-page slide summarizing the paper’s figures and most important quantitative benchmarks for a presentation.