0 cumulative citations
View corpus contextLLMs used as synthetic survey respondents add no individual-level information beyond demographic lookups and consistently exaggerate demographic effects, inflating segment gaps and leading to frequent targeting errors across U.S. and cross-cultural surveys.
When Synthetic Users Fail: A Cross-Domain Benchmark of LLM-Simulated Human Survey Responses
Under demographic prompting and the tested survey-simulation protocols, LLMs do not outperform simple demographic baselines at predicting individual human survey answers and systematically overstate how predictive demographics are, inflating between-group differences and misleading segment-targeting decisions.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Large language models (LLMs) are increasingly used as synthetic users, stand-ins for human respondents whose simulated answers feed product, policy, and market decisions. We ask when this substitution is valid and when it fails, and package the answer as an evaluation framework for intelligent synthetic-user systems. A single protocol, run across four models spanning two families and an 8B-to-frontier capability range, is applied to two independent domains of real human-response data: U.S. general social attitudes (General Social Survey) and cross-cultural values (World Values Survey). Every model is benchmarked against a suite of non-LLM baselines fit on held-out human data. Under demographic prompting and the survey-simulation protocols we test, two failures replicate across both domains, all four models, and both families. First, at the individual level no LLM beats even the strongest baseline; on cross-cultural values every model falls well below it, and the gap survives distance-aware and proper scoring. Second, models systematically over-determine demographics, treating identity as far more predictive of attitudes than it is among real people, a distortion present for nearly every question-group combination and robust to a coding-invariant measure. Neither failure is remedied by a larger, more capable model. A decision-impact analysis shows why this matters in practice: on a segment-targeting task the models inflate between-segment gaps two to fourfold, would direct a team to the wrong segment in half of U.S. and most cross-cultural cases, and manufacture segment splits that do not exist in real people. We make the cross-domain benchmark and the evaluation framework available on request, so that teams can determine in advance when synthetic-user evidence is safe for decision support and when it is not.
Summary
Main Finding
LLM “synthetic users” prompted with demographics and used to simulate survey respondents fail in two systematic ways across two distinct domains (U.S. social attitudes and 63-country values), four models, and two model families: (1) they provide no individual-level advantage over simple non-LLM demographic baselines (and often perform substantially worse on cross-cultural value questions), and (2) they massively over‑determine attitudes from demographics (stereotyping), treating identity as far more predictive of answers than real humans are. These failures persist across output formats, scoring rules, and model scale (8B → frontier).
Key Points
- Two replicated failures
- Individual-level advantage: No evaluated LLM outperforms a demographic lookup table or learned demographic models on predicting individual respondents. On World Values Survey (WVS), every model is 11–22 percentage points less accurate than the demographic baseline.
- Demographic over-determination (stereotyping): LLMs exaggerate how much demographics explain variation in answers. Example: political leaning explains ~1.5% of real-world variance for some questions but the model behaves as if it explains ~67%.
- Robustness
- Failures hold across: two domains (GSS, WVS), four models (Claude Haiku 4.5, Claude Sonnet 4.6, Llama‑3.1‑8B, Llama‑3.3‑70B), two prompt formats (single-answer and distribution), two random seeds, and under distance-aware and proper-scoring metrics.
- Larger/more capable models do not fix the problems; frontier models stereotype at least as strongly as smaller ones.
- Decision impact
- On canonical segment-targeting tasks, models inflate between-segment gaps by 2–4×, would direct teams to the wrong target segment in ~50–72% of U.S. cases and in most cross‑cultural cases, and invent segment splits that do not exist in real data (up to 41% of WVS cases).
- Contributions
- A cross-domain benchmark and validation framework (available on request) that: (i) compares LLM outputs to non‑LLM baselines, (ii) reports individual vs aggregate fidelity separately, (iii) quantifies subgroup determinism with a stereotyping index and Cramér’s V, and (iv) includes decision-impact analysis.
Data & Methods
- Domains and datasets
- GSS (General Social Survey), waves 2016–2024: in-scope pool after filtering = 14,704 respondents, 85,898 (respondent, question) pairs across 10 attitude questions. Demographics: age, sex, race, degree, region, political views, party ID.
- WVS (World Values Survey) Wave 7: in-scope pool = 91,774 respondents across 63 countries, 1,426,473 (respondent, question) pairs across 16 ordinal value questions. Demographics: age group, sex, education, settlement type, country.
- Sampling: evaluation samples drawn by cell (~100 respondents per question) to get stable per-(question, group) distributions; human distributions and baselines estimated on full datasets.
- Models tested
- Two families: Claude (closed) and Llama (open). Specific models: Claude Haiku 4.5, Claude Sonnet 4.6, Llama‑3.1‑8B, Llama‑3.3‑70B.
- Two prompt styles: Style A (single-answer) and Style C (probability-distribution output in JSON). Two independent generation runs (“seeds”) for run-to-run noise.
- Non-LLM baselines (trained on held-out 50% of data)
- Question-marginal baseline (marginal distribution),
- Demographic lookup table (most common answer conditional on given demographics),
- Learned demographic models (logistic regression, random forest).
- Metrics & analysis
- Individual-level: exact-match accuracy, distance-aware metrics for ordinal scales, proper scoring (log-loss, Brier) for probability outputs; paired bootstrap CIs.
- Subgroup structure: stereotyping index (bounded measure comparing fraction of answer variation explained by a demographic attribute in model vs real humans) and coding-invariant Cramér’s V.
- Decision-impact analysis: simulated segment-targeting task assessing inflation of gaps, wrong-segment rates, and invented splits.
- Main empirical patterns reported with confidence intervals and checked for parser-invalid outputs (invalid/refused responses flagged).
Implications for AI Economics
- Use cases at risk
- Market segmentation, product feature prioritization, demand estimation, messaging pre-testing, and policy reaction forecasts that rely on LLM-simulated respondents can be systematically misleading if the simulation is used as a substitute for human data at the individual or subgroup level.
- Economic decision distortions
- Overstated between-segment differences (2–4× inflation) can cause misallocation of resources (e.g., marketing budget steered to the wrong customer segment).
- High wrong-target rates (~50–72% for U.S.; higher cross‑culturally) mean strategies derived from synthetic users can frequently prioritize suboptimal segments or messages.
- Invented splits (up to 41% in cross‑cultural data) risk fabricating market heterogeneity and driving unnecessary product differentiation, pricing, or localization efforts.
- Practical recommendations for economists and practitioners
- Always benchmark synthetic-user outputs against non‑LLM demographic baselines before using them for decision support.
- Report individual-level and subgroup fidelity separately; aggregate agreement alone is insufficient.
- Measure and report demographic over-determination (stereotyping index or similar) to detect inflated identity-driven effects.
- Run decision-impact simulations (e.g., segment-targeting) to quantify practical consequences before deployment.
- Do not assume larger models are safer synthetic users; performance must be validated empirically for the intended use.
- Prefer human data or real-world experiments for decisions requiring reliable individual-level or subgroup inferences; treat LLM synthetic users as low-cost exploratory tools only with explicit, validated limits.
- Research & policy directions
- Explore whether richer persona construction, fine-tuning on survey data, or alternative elicitation methods can reduce stereotyping and recover individual-level information.
- Standardize validation protocols in applied economics and marketing that require baseline-anchored tests and decision-impact metrics.
- Consider regulatory or governance guidance for using synthetic-user evidence in high-stakes economic and policy decisions.
Limitations (brief) - Findings are scoped to demographic prompting under the survey-simulation protocols tested; results do not automatically generalize to other conditioning methods (e.g., complex persona narratives or fine-tuned models). - Datasets are GSS (2016–24) and WVS Wave 7; other surveys or question sets may yield different patterns and should be validated with the provided framework.
Assessment
Paper Typedescriptive
Evidence Strengthhigh — The paper runs a reproducible, benchmark-style evaluation across two independent survey domains (GSS and WVS), four models spanning an 8B-to-frontier capability range, multiple prompt formats, and a suite of non-LLM baselines; it uses distance-aware and proper-scoring metrics, paired-bootstrap confidence intervals, and a decision-impact analysis, producing consistent failures across models and domains. Limitations (see below) reduce absolutist confidence but do not overturn the reported replication and statistical rigor.
Methods Rigorhigh — The authors (i) estimate human answer distributions from large held-out datasets, (ii) train and evaluate non-LLM baselines on held-out splits for a meaningful yardstick, (iii) use multiple scoring metrics (exact-match, distance-aware, proper scoring), (iv) report paired-bootstrap CIs, (v) test multiple seeds and prompt styles, and (vi) perform a decision-impact analysis to show practical consequences; shortcomings include focus on demographic prompting only, a limited set of models/families, and potential pretraining contamination not fully eliminable.
SampleTwo public human-response datasets: (1) General Social Survey (GSS) 2016–2024 waves, after filtering 14,704 respondents and 85,898 (respondent, question) pairs across 10 attitude questions; evaluation sample drawn ~993 rows (about 100 respondents per question) with demographics: age, sex, race, highest degree, region, political views, party ID. (2) World Values Survey (WVS) Wave 7, after filtering 91,774 respondents across 63 countries and 1,426,473 (respondent, question) pairs over 16 ordinal value questions; evaluation sample ~1,458 rows with demographics: age group, sex, education, settlement type, country. Human distributions and demographic baselines are estimated from the full datasets, while model evaluation uses sampled rows allocated by cell.
Themeshuman_ai_collab adoption
GeneralizabilityResults are scoped to demographic/persona-style prompting and the specific single-answer and distribution prompt formats tested; alternative persona constructions or fine-tuning could behave differently., Only four models across two families were tested (Claude Haiku/Sonnet and Llama 3.1/3.3); findings may not generalize to all model architectures, sizes, or proprietary variants not evaluated., Decision-impact conclusions concern survey-like, segmented targeting tasks and attitude/value questions; transfer to other application types (e.g., behavioral prediction, forecasting, high-stakes policy decisions) is uncertain., Potential pretraining contamination and differing language/cultural coverage could affect WVS results for some countries; non-English or non-survey modalities not evaluated., Sampling-by-cell design improves per-cell estimation but is not a random population draw, so some aspects of population representativeness may be limited.
Claims (11)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| At the individual-response level, no evaluated LLM outperforms the strongest non-LLM demographic baseline on the GSS or WVS data. Output Quality | negative | Accuracy of predicting individual human survey responses |
Reading fidelity
high
Study strength
high
|
n=4
|
| On cross-cultural value questions from the World Values Survey, every evaluated model is 11 to 22 percentage points less accurate than the demographic baseline. Output Quality | negative | Individual-level prediction accuracy for WVS value responses |
Reading fidelity
high
Study strength
high
|
n=91774
11 to 22 percentage points less accurate
|
| The LLMs' individual-level disadvantage is not limited to exact-match accuracy; it persists under distance-aware metrics and proper scoring of probability distributions. Output Quality | negative | Distance-aware prediction accuracy and probabilistic scoring of survey responses |
Reading fidelity
high
Study strength
high
|
n=4
|
| LLMs systematically overstate how predictive demographic identity is of survey attitudes relative to real human respondents. Ai Safety And Ethics | negative | Strength of association between demographic attributes and survey answers |
Reading fidelity
high
Study strength
high
|
n=4
|
| For U.S. political leaning and confidence in banks, real respondents' politics explains about 1.5% of answer variation, whereas the model behaves as though it explains up to roughly 67%. Ai Safety And Ethics | negative | Variation in survey answers explained by political identity |
Reading fidelity
high
Study strength
medium
|
n=14704
about 1.5% in real answers versus up to roughly 67% in model behavior
|
| Demographic over-determination is statistically distinguishable from chance for nearly every question-group combination in both domains. Ai Safety And Ethics | negative | Frequency and statistical strength of demographic-answer associations |
Reading fidelity
high
Study strength
medium
|
n=4
|
| Increasing model size or capability does not remedy either the individual-level fidelity failure or demographic over-determination. Output Quality | negative | Individual prediction fidelity and demographic stereotyping as a function of model capability |
Reading fidelity
high
Study strength
medium
|
n=4
|
| On a segment-targeting task, the evaluated models inflate between-segment opinion gaps by approximately two- to fourfold. Decision Quality | negative | Difference in predicted attitudes between demographic segments |
Reading fidelity
high
Study strength
medium
|
n=4
two to fourfold
|
| Using model-generated synthetic-user evidence would direct a team to the wrong target segment in roughly half of U.S. cases and most cross-cultural cases. Decision Quality | negative | Correctness of segment-targeting decisions |
Reading fidelity
high
Study strength
medium
|
n=4
50–72% of cases
|
| The models can manufacture demographic segment splits that do not exist among real people, occurring in up to 41% of cross-cultural value cases. Decision Quality | negative | False identification of segment differences |
Reading fidelity
high
Study strength
medium
|
n=91774
up to 41% of cases
|
| LLMs can reproduce population-level answer distributions reasonably well even while failing at individual prediction and subgroup validity. Output Quality | positive | Similarity between model-generated and human population-level response distributions |
Reading fidelity
medium
Study strength
medium
|
n=4
|