Large language models reproduce average survey answers but cannot stand in for individual respondents: across >400,000 participants and multiple datasets, LLMs match item means yet explain only a tiny fraction of person-specific deviations after removing item averages, revealing a widespread 'item-mean surrogacy' limitation.
Item-Mean Surrogates: Why Richer Persona Data Fail to Improve LLMs as Human Surrogates
Across multiple large datasets, LLMs reliably reproduce item-level average responses but fail to recover respondent-specific deviations or distributional shape, explaining only about 3% of de-meaned respondent variation versus a 53.6% human test–retest benchmark.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
No provider observation is available for this paper.
Missing data, not a zero citation count.
LLMs are increasingly used as human surrogates, often on the premise that richer persona data could make them substitutes or exploratory tools for specific individuals. We test this premise across four datasets covering more than 400,000 participants and more than 6,000 survey items and experimental outcomes. LLMs perform well at the aggregate level: their average responses closely align with average human responses to the same items. But this success largely reflects predicting each item's average human response. Once each item's human mean is removed, LLM predictions explain only 3.05% of the remaining respondent-specific variation, far below the 53.6% human test-retest benchmark. Richer personas, model variants, and fine-tuning do not close this gap. In variance analyses, once item means are removed, the reliable remaining signal is person-by-item. It captures how a respondent departs from the mean on a particular item and is about 8.9x larger than the stable person effect. Persona data encode the respondent, but not this item-specific deviation. LLM responses also compress human response distributions, using less spread, fewer response categories, and distorted distributional shapes. We call this pattern item-mean surrogacy. Current LLM surrogates can approximate item averages, but not the distributions or respondent-specific deviations needed to replace individual humans. We propose four empirical tests for LLM-based human-surrogate claims.
Summary
Main Finding
LLMs reliably reproduce item-level human means but fail to recover respondent-specific deviations from those means. Across >400,000 participants and >6,000 items, after removing each item’s human mean LLMs explain only a few percent of the remaining variation (pooled de-meaned R2 ≈ 3.05% in the primary Megastudy), far below human test–retest reliability (53.6%). The dominant recoverable signal is person-by-item interaction (item-specific deviations), not stable person main effects, and richer personas, model choice, or fine-tuning do not close the gap. The authors label this pattern “item-mean surrogacy.”
Key Points
- Item-mean fidelity vs. higher fidelities:
- Item-mean fidelity: LLMs recover average responses across people for each item (high correlation; Pearson r ≈ 0.78 in primary data).
- Distributional fidelity: LLMs compress variance, use fewer effective response categories, and distort distributional shape (median item SDs 50–65% of human SD across datasets).
- Individual-level fidelity: LLMs fail—pooled de-meaned R2 (human minus item mean) ≈ 3.05% in Megastudy (only ~5.7% of the 53.6% human test–retest ceiling).
- Variance decomposition (G-theory):
- Stable person main effects ≈ 4.9% of prediction-error variance.
- Item main effects ≈ 8.7%.
- Residual ≈ 86.4%, split (using test–retest) into person-by-item interaction ≈ 44.0% and transient error ≈ 42.4%.
- Person-by-item interaction is ≈ 8.9× larger than the stable person effect.
- Robustness checks:
- Persona-swap (shuffling persona assignments) shows correct persona matching contains some respondent signal, but magnitude is tiny relative to the missing person-by-item signal.
- Fine-tuning and stronger prompts sometimes improve pooled de-meaned R2 modestly (best non-fine-tuned prompt 4.44% in Megastudy), but fine-tuning often overfits seen studies and fails to generalize to held-out studies (e.g., SocSci210: fine-tuned seen 7.57% vs. held-out 0.73%).
- Leave-one-out human item-mean baseline outperforms LLMs on per-respondent correlations (human-LOO > LLM).
- Distributional mismatch beyond compression:
- Shape distortion contributes substantially (31–40% of squared Wasserstein-2 distance) in several datasets; compression alone does not explain LLM distributional differences.
- Cross-dataset patterns:
- Megastudy pooled de-meaned R2 ≈ 3.05% (up to 4.44% in some prompts).
- SocSci210 pooled ≈ 5.78% (but poor generalization to held-out studies).
- Twin-2K-500 Survey ≈ 3.87–6.21% across specs.
- ANES (small set) higher (≈ 8.7–10.8%), with signal concentrated in party/ideology items.
Data & Methods
- Datasets:
- Megastudy (primary): 1,784 respondents, 160 items (analytical rows use 1,728 respondents × 133 items after exclusions).
- SocSci210: >400K respondents, 5,998 study-specific outcomes across 210 studies.
- Twin-2K-500 Survey: 2,058 respondents, 126 items (used for test–retest benchmark).
- ANES: 4,270 respondents, 11 political survey variables (robustness).
- Scaling and metrics:
- Responses converted to POMP scale (0–100 Percent of Maximum Possible) to compare heterogeneous item scales.
- Primary individual-fidelity metric: pooled de-meaned R2 = squared correlation between human and LLM deviations from each item’s human mean (pooled across all respondent–item pairs).
- Human test–retest reliability (Twin-2K-500) used as an upper-bound ceiling for how much stable person-by-item signal could be recovered from single-occasion data (53.6%).
- Analytical methods:
- Generalizability theory (G-theory) variance decomposition to parse prediction-error variance into item main effects, person main effects, person-by-item interaction, and transient error.
- Persona-swap randomization to test whether persona assignments carry respondent-specific signal beyond chance.
- Fine-tuning experiments (e.g., GPT-4.1 on Megastudy, Socrates-Qwen2.5-14B on SocSci210) to test whether adaptation recovers more individual signal.
- Distributional comparisons using SD ratios, effective category counts, Wasserstein-2 decompositions, and shape/skewness assessments.
- Baselines: leave-one-out human item means and a compression-only rescaling baseline (human responses rescaled to LLM mean/spread) to isolate shape differences.
Implications for AI Economics
- Limits on micro-level substitution:
- LLM-generated “synthetic respondents” are unreliable for individual-level inference or applications that require predicting how a particular person will deviate from population means (e.g., personalized policy counterfactuals, individualized treatment-response simulation, targeting).
- Using LLM surrogates as drop-in replacements for human responses in microeconometric analyses risks bias because LLMs reproduce item means but not within-person heterogeneity or cross-item respondent-specific patterns.
- Risks to causal inference and heterogeneity analysis:
- Synthetic responses that compress variance and alter distributional shape can distort estimates of effect heterogeneity, treatment effect distributions, and subgroup analyses—potentially invalidating heterogeneity-driven policy prescriptions.
- Imputation of missing values or nonresponse with LLMs may underestimate variance and mask important person-by-item interactions, invalidating multiple-imputation or inverse-probability approaches that assume fidelity to individual heterogeneity.
- Practical uses where LLMs still add value:
- LLM surrogates are useful for aggregate-level tasks: estimating item means, prototype scenario exploration, stress-testing survey designs, or quick population-average simulations where individual heterogeneity is not essential.
- Cost-efficient population-level replication and exploratory work that focuses on average responses or coarse subgroup means.
- Recommended validation and best practices for AI economists:
- Do not assume aggregate agreement implies individual fidelity. Always test for item-mean surrogacy by de-meaning per item and reporting pooled de-meaned R2.
- Assess distributional fidelity (SD ratios, effective category use, shape comparisons, Wasserstein decompositions) in addition to mean matching.
- Use G-theory or similar variance decompositions and human test–retest benchmarks to identify how much recoverable person-by-item signal exists and whether the model captures it.
- Run persona-swap and leave-one-out human-mean baselines to check whether any apparent accuracy exceeds simple item-mean predictors.
- Treat fine-tuning gains cautiously—verify generalization to held-out items/studies to avoid overfitting to seen studies.
- Research and policy implications:
- Economic and policy work that depends on individual-level counterfactuals, precision targeting, or analyses of within-person tradeoffs should avoid relying exclusively on current LLM surrogates.
- Investment in modeling approaches that explicitly target person-by-item interactions (e.g., richer longitudinal/person-specific behavioral data, hierarchical models that model interactions, or hybrid approaches that combine LLM priors with observed within-person residuals) is needed before LLMs can serve as faithful digital twins for individuals.
Takeaway: current LLMs can be useful for aggregate, mean-level simulation but systematically fail to reproduce respondent-specific deviations and full response distributions—so they should not be treated as substitutes for human respondents when individual-level fidelity matters.
Assessment
Paper Typedescriptive
Evidence Strengthhigh — Large-scale, multi-dataset empirical evaluation (including a preregistered analysis), extensive robustness checks (persona swaps, fine-tuning, model variants), and use of psychometric benchmarks (test–retest reliability and G-theory) support the descriptive claims about LLM surrogate performance; results are consistent across datasets and prompting/fine-tuning variations.
Methods Rigorhigh — Appropriate and careful measurement choices (POMP rescaling), rigorous variance decomposition using generalizability theory, clear de-meaning to isolate item effects, preregistration, large sample sizes across multiple independent datasets, and targeted robustness checks (persona-swap, leave-one-out human baseline, fine-tuning seen/held-out comparisons).
SampleFour datasets: (1) Megastudy — primary analytic sample ~1,728 respondents × 133 items (18 studies) with matched LLM and human responses; (2) SocSci210 — very large robustness set with >315K respondents and several thousand item-outcomes (primary pooled rows use 315,512 respondents × 4,364 items; fine-tuning seen/held-out splits reported); (3) Twin-2K-500 Survey — ~2,058 respondents × 126 items providing test–retest benchmarks; (4) ANES — ~4,256 respondents × small set of political items (4–11 items depending on row). Pooled analyses include >108K respondent–item pairs in the Megastudy row and over 400K participants across datasets.
Themeshuman_ai_collab adoption
GeneralizabilityFindings are specific to survey-style, ordered/bounded response items and may not apply to interactive, behavioral, or task-based settings., Results focus on currently tested LLM prompting/fine-tuning regimes and specific model families; other architectures, training data, or advanced personalizing techniques might perform differently., POMP rescaling and analytic choices (e.g., which items retained) could affect magnitudes; domain-specific items (e.g., political ideology codes) required special handling., Population representativeness depends on the original panels (Twin-2K-500, SocSci210, ANES); external populations or cultures not represented may differ., Limited assessment of longitudinal generalization and dynamic adaptation (how models improve with more individualized history across many items over time).
Claims (13)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| LLM item means closely track human item means in the primary Megastudy analysis. Decision Quality | positive | Agreement between LLM and human item-level mean responses |
Reading fidelity
high
Study strength
high
|
n=1728
Pearson r = 0.78
|
| After item means are removed, LLM predictions explain only 3.05% of respondent-specific variation in the Megastudy. Decision Quality | positive | Respondent-specific response variation after removal of item means |
Reading fidelity
high
Study strength
high
|
n=1728
pooled de-meaned R2 = 3.05%
|
| The 3.05% LLM de-meaned R2 is far below the 53.6% human test-retest reliability ceiling. Decision Quality | negative | Individual-level surrogate fidelity relative to human response reliability |
Reading fidelity
high
Study strength
high
|
n=1728
3.05% versus 53.6%
|
| LLM responses have substantially narrower distributions than human responses across the Megastudy, SocSci210, and Survey datasets. Decision Quality | negative | Response-distribution spread |
Reading fidelity
high
Study strength
high
|
LLM median item-level SDs were 65%, 50%, and 57% of human SDs in the Megastudy, SocSci210, and Survey, respectively
|
| LLMs use fewer effective response categories than humans, with median category-use ratios ranging from 43% to 73% of the human value across the examined datasets. Decision Quality | negative | Breadth of response-category usage |
Reading fidelity
high
Study strength
medium
|
43% to 73% of the human value
|
| Differences in response-distribution shape account for 31% to 40% of the squared Wasserstein-2 distance between human and LLM distributions in the Megastudy, SocSci210, and Survey. Decision Quality | negative | Distributional shape similarity between human and LLM responses |
Reading fidelity
high
Study strength
medium
|
Shape accounts for 31% to 40% of squared Wasserstein-2 distance
|
| A simple leave-one-out human item-mean predictor outperforms the LLM on per-respondent correlations across items. Decision Quality | negative | Per-respondent correlation between predicted and observed responses |
Reading fidelity
high
Study strength
high
|
n=1631
paired dz = −0.55
|
| Stable person main effects account for only 4.9% of prediction-error variance, whereas stable person-by-item interactions account for 44.0%. Decision Quality | negative | Components of LLM prediction-error variance |
Reading fidelity
high
Study strength
medium
|
n=1728
4.9% stable person main effect versus 44.0% stable person-by-item interaction
|
| The stable person-by-item interaction component is about 8.9 times larger than the stable person main-effect component. Decision Quality | negative | Relative size of person-by-item versus stable person variance components |
Reading fidelity
high
Study strength
medium
|
n=1728
8.9× larger
|
| Correctly matched personas contain respondent-specific signal relative to shuffled personas, but this signal does not materially close the individual-prediction gap. Decision Quality | mixed | Respondent-specific predictive signal from persona-respondent matching |
Reading fidelity
high
Study strength
medium
|
n=1728
3.05% versus 0.028% maximum shuffled de-meaned R2
|
| Fine-tuning did not close the individual-prediction gap in the tested settings. Decision Quality | null_result | Respondent-specific predictive accuracy after removing item means |
Reading fidelity
high
Study strength
medium
|
n=1728
2.31% versus 4.44% de-meaned R2 for fine-tuned versus strongest non-fine-tuned Megastudy model
|
| The individual-prediction gap persists across the Megastudy, Survey, SocSci210, and ANES datasets, although recovery varies by dataset. Decision Quality | mixed | Generalization of respondent-specific LLM prediction across datasets |
Reading fidelity
high
Study strength
medium
|
n=315512
De-meaned R2 ranged from 0.73% to 10.77% across reported dataset settings
|
| The paper characterizes current LLM human surrogates as exhibiting item-mean surrogacy: they approximate item averages but not the response distributions or respondent-specific deviations needed for individual replacement. Decision Quality | negative | Overall fidelity of LLMs as individual human surrogates |
Reading fidelity
high
Study strength
medium
|
n=400000
|