Instruction-tuned LLMs rarely produce genuine per-call randomness — they 'know' population distributions but 'do' deterministic answers, skewing silicon-sampling surveys; asking models to describe distributions or randomizing prompt surface features recovers much of the lost variation.

Main Finding

Instruction-tuned language models do not behave like samplers of a stated conditional distribution: when asked to produce one response per persona-call they collapse to a single deterministic output (mode collapse), even though the same models can accurately describe the target distribution in a single call. The authors name this the KNOWS/DOES split. Alignment/instruction tuning induces a degenerate sampling primitive visible in logits, and this failure undermines common "silicon sampling" pipelines that treat each model call as an independent draw from a persona-conditional distribution.

Key Points

• Categorical collapse: on a set of synthetic categorical targets (skewed, bimodal, multi-way), instruction-tuned models produce a single answer in over 94% of calls. Example: asked for a uniform integer 1–100, the model returned 42 in 78% of calls.
• Logit-level pathology: at temperature 0 the top-two logit gap can exceed 14 nats; this concentration occurs before decoding, so temperature/top-p adjustments cannot recover the intended distributions.
• Degenerate random primitive: direct prompts to "pick a uniform random X" concentrate on a format-specific favored value (e.g., coin flip → heads 85%), and forbidding the modal value moves mass to another favored value instead of spreading it.
• KNOWS/DOES split: the same model that cannot produce per-call samples can often describe the full distribution accurately in one call (e.g., JSON probabilities or a list of exemplars). Describe-pathway interventions (V8/V9) achieved very low TV distance (~0.03), while plain per-call sampling methods had high TV (~0.52).
• Alignment-induced: comparing pretrained base models to their instruction-tuned counterparts across three families (Llama-3.1-8B, Mistral-7B-v0.3, Qwen2.5-7B) shows instruction tuning substantially increases TV-to-target on sampling tasks (mean TV gaps ~0.19–0.28), implicating alignment/instruction-tuning rather than core language modeling.
• Real-world impact (OpinionQA): on 100 Pew-based OpinionQA items, standard persona-aggregation (Argyle-style silicon sampling) produced little per-call variation (57% of (persona,item) pairs returned same answer across 50 calls) and had TV-to-Pew = 0.46. The describe pathway (one call per item returning distribution) halved the gap (TV = 0.22).
• Mitigation for per-persona needs: Prompt-Perturbed Argyle (PPA) perturbs surface prompt features (option order, question phrasing, persona placement) across calls while keeping persona semantics fixed. PPA (same-cost) reduced mean TV by 0.10 (21% relative reduction; from 0.46 → 0.36) on OpinionQA.
• Chain-of-thought / algorithmic scaffolds fail: embedding a sampling algorithm (inverse CDF with internal random u) fails because the internal random u produced by the model is itself non-uniform and degenerate.

Data & Methods

• Models and families:
  • Primary behavioral experiments used gpt-4o (temp 1.0; 200 calls per condition; temp 0 for logit reads).
  • Base vs instruct comparisons used Llama-3.1-8B, Mistral-7B-v0.3, Qwen2.5-7B and their instruction-tuned variants.
  • Cross-family replication in appendices covered five additional instruction-tuned families.
• Synthetic tasks:
  • Five non-trivial categorical targets (skewed binary, mixture, three-way, bimodal, skewed 5-way) plus uniform random-number prompts in several formats (integer 1–100, coin flip, float).
  • Metric: total variation (TV) distance between empirical distribution of 200 calls and the target distribution.
  • Key synthetic results: empirical distributions concentrated on a single option in >94% of calls; top-two logit gaps 3.75–16+ nats.
• Intervention study (ten interventions V0–V9):
  • V0–V7: various per-call sampling prompts including CoT and algorithmic scaffolds (DOES interventions) — all failed (mean TV ~0.43–0.52).
  • V8–V9: describe-as-list and describe-as-JSON (KNOWS interventions) — both succeeded (mean TV ~0.03).
• OpinionQA application:
  • 100 OpinionQA items mapped to 5-point Likert; 100 personas per item (Argyle protocol); per-item describe pipeline uses a single call returning JSON population probabilities.
  • Results: Standard Argyle TV-to-Pew = 0.46; describe pathway TV = 0.22. Demographically-matched Argyle (control) produced similar TV to standard Argyle, ruling out persona mismatch as the primary cause.
• Prompt-Perturbed Argyle (PPA):
  • Perturbations: option ordering (4 variants), question phrasing (3 variants), persona position (3 variants); 36 possible combinations, sampled uniformly across 100 calls per item.
  • Outcome: same API cost as baseline; mean TV reduced by 0.10 (21% relative, p ≪ 0.001).
• Diagnostics:
  • Per-call Shannon entropy measured; logit gap reads via top-20 logprobs at temp 0; sequence-rank correlations for describe-as-list to show the lists are near-deterministic counts rather than independent samples.

Implications for AI Economics

• Silicon sampling caution: studies using LLMs as synthetic survey respondents or agent populations must not assume per-call outputs are independent draws from an intended distribution for instruction-tuned models. Aggregation of per-call responses (persona aggregation) can severely understate within-stratum variance and produce biased population estimates.
• Prefer describe-for-population estimates: when only population-level statistics are required (e.g., estimating marginal distributions, opinion shares), prompting the model to describe the distribution in a single call (JSON probabilities / counts) yields substantially better calibration to human surveys.
• For agent-based and micro-founded simulations: many economic simulations require one response per agent that conditions downstream behavior. Because instruction-tuned models often return the same deterministic response for a given persona, simulation dynamics and variance will be mismeasured. Use PPA-like prompt perturbations to induce per-call heterogeneity without extra cost, but note PPA reduces error only partially (21% relative improvement here).
• Training and mechanism implications for economic modeling:
  • Alignment/instruction tuning alters not only model behavior on safety/alignment axes but also the fundamental sampling primitives; this can bias any economic analysis that relies on synthetic heterogeneity.
  • Researchers using synthetic data should validate that the model's per-call sampling primitive is well-behaved (check per-call entropy, logit gaps, repeatability) before using generated populations as microdata.
• Policy and robustness:
  • Policy conclusions drawn from LLM-based synthetic populations (e.g., predicted public reaction, distributional impacts) may be fragile if they depend on within-group variance. Mischaracterized variance can mislead policy design or welfare analysis.
• Practical checks and mitigations for practitioners:
  • Run diagnostic probes: repeat calls per (persona,item) and compute TV/per-call entropy; check logit top-two gaps at temp 0.
  • Compare describe vs sample pipelines on target metrics; if describe outperforms sample, suspect KNOWS/DOES split.
  • Use same-cost prompt perturbation strategies (PPA) for per-agent heterogeneity; consider combining PPA with model families or base models that show less collapse.
• Directions for further research (economic relevance):
  • Quantify how mode collapse affects counterfactual simulations, equilibrium outcomes, and estimated policy elasticities in agent-based economic models.
  • Investigate training/regulation strategies that retain alignment benefits while preserving a faithful sampling primitive (e.g., explicit calibration objectives, different fine-tuning recipes).
  • Explore methods to reconstruct valid joint or conditional distributions for downstream micro-simulation when per-call sampling is unreliable.

Limitations to bear in mind: experiments focus on specific models, instruction-tuning pipelines, and categorical tasks (plus OpinionQA); results may vary across architectures, newer alignment methods, or explicit calibration in future releases. Still, the diagnostic framework (KNOWS/DOES) and practical mitigations (describe-pathway, PPA) are directly actionable for researchers using LLMs in economics.