Large language models can stand in for human subjects in low-stakes pilots and robustness checks, but their behavioral fidelity is inconsistent and error-prone; researchers should treat synthetic participants as heuristic supplements—not substitutes—and adopt standardized evaluation and ethical safeguards.

Main Finding

LLM-generated synthetic participants show promise as a low-cost, flexible adjunct for research and data-collection tasks, but across 182 reviewed studies their fidelity to human participants is modest and inconsistent. Major failure modes—cognitive misalignments, distortions, misleading believability, and overfitting/contamination—limit their suitability as direct substitutes for human samples. The authors recommend treating synthetic participants as heuristic-like tools (useful for specific supplemental roles) rather than as full replacements, and call for standardized evaluation, careful application, and ethical safeguards.

Key Points

• Scope: Systematic review of 182 studies comparing LLM-generated participants to human samples across diverse simulation types, models, and prompting/method variants.
• Four standardized failure categories affecting synthetic participants:
• Cognitive misalignments — differences in reasoning, goals, and bounded rationality compared with humans.
• Distortions — systematic biases in outputs relative to target human distributions.
• Misleading believability — outputs that look plausible but are incorrect or unrepresentative.
• Overfitting/contamination — reproduction of pre-training or fine-tuning data (stochastic parroting) and leakage from training sets.
• Heterogeneous literature: mixed results, varied methods, and fragmented evaluation metrics impede general conclusions about when LLMs reliably mimic humans.
• Fidelity gains from prompt engineering, model selection, or participant/environment modeling have been limited and context-dependent.
• Recommendation: conceptualize synthetic participants as heuristics—useful to explore ideas, stress-test designs, augment limited human data, or run low-stakes pilots, but not to replace empirical human samples without rigorous validation.
• Additional topics covered: evaluation metrics, promising augmentative (hybrid) approaches, and professional, social, and ethical concerns (e.g., transparency, consent, contamination of human-data pools).

Data & Methods

• Literature assembled via a hybrid search strategy combining electronic database queries and reference/back-citation searches to identify empirical and methodological studies involving LLM-generated participants.
• Screening and curation: rigorous quality filters were applied to select 182 studies for synthesis (details on inclusion/exclusion criteria and quality metrics are implied by the abstract).
• Synthesis approach: studies were coded against generalizable indicators and organized into a standardized taxonomy centered on the four failure categories listed above.
• Scope of methods in reviewed studies: wide variation — different LLM families and sizes, prompt engineering techniques, participant persona modeling, environment simulations, and evaluation protocols. This heterogeneity motivated the need for a unified conceptual framing rather than a single performance estimate.

Implications for AI Economics

• Use cases where LLM participants can add value:
  • Pilot testing and rapid prototyping of surveys, experiments, games, and mechanism designs to identify gross design flaws before costly human trials.
  • Generating hypotheses or counterfactual scenarios for exploratory work and scenario analysis.
  • Stress-testing economic models by enumerating extreme or rare behaviors, or creating catalogues of plausible agent responses for robustness checks.
  • Augmenting small human samples in hybrid designs (LLM + human) with careful validation to increase variety at lower marginal cost.
  • Teaching, training, and constructing illustrative examples for pedagogy and communication.
• Risks and limitations for economic research:
  • External validity and causal inference risk: cognitive misalignments and distortions can bias estimated behaviors, preferences, or treatment effects.
  • Strategic and game-theoretic settings: LLMs may misrepresent incentives, dynamic strategic thinking, and bounded rationality, producing misleading equilibria or mechanistic responses.
  • Contamination/data leakage: stochastic parroting risks reusing training-set content, which can invalidate novelty claims, leak sensitive information, or bias empirical baselines.
  • Misleading believability can produce high-confidence but incorrect inferences if synthetic outputs are not rigorously calibrated against human data.
• Research design recommendations for economists:
  • Treat LLM participants as supplementary/heuristic tools: pre-register how synthetic data will be used, and avoid substituting synthetic for human data in confirmatory causal studies without validation.
  • Validate synthetic outputs against held-out human samples, using clear metrics for behavioral fidelity relevant to the economic question (e.g., revealed preferences, strategy choice distributions, response latencies if relevant).
  • Use hybrid designs with human-in-the-loop checks, adversarial probing, and sensitivity analyses to quantify how conclusions change with synthetic vs. human inputs.
  • Monitor and mitigate contamination: track model training sources when possible, avoid tasks that require detecting rare events or copyrighted/sensitive content, and disclose use of LLM-generated participants.
  • Invest in standardized benchmarks for economic behaviors (strategic interaction, intertemporal choice, risk, social preferences) to make cross-study comparisons possible.
• Policy and ethical considerations:
  • Transparency: disclose synthetic participant use in publications and reports.
  • Consent and representation: avoid presenting synthetic results as human behavior without clear labeling; consider effects on participant pools and labor markets for research subjects.
  • Reproducibility: document prompts, model versions, seeds, and any fine-tuning to allow audit and replication.
• Bottom line for economists: LLMs can reduce costs and accelerate exploratory work, but their current limitations mean they are best used to complement—not replace—human subjects in empirical economic research unless rigorous, domain-specific validation demonstrates equivalence for the question at hand.