Large language models can serve as cheap, large-scale instruments for measuring cultural norms, beliefs and narratives—by probing pretrained weights economists can approximate aggregate discourse—yet alignment, fine-tuning and sampling choices can distort those signals, so researchers should prefer base or minimally adapted models, validate against surveys, and publish provenance and prompts.

Main Finding

Large language models (LLMs) constitute a third research ambition for contemporary AI: they can be used as scientific instruments to study human behavior, culture, and moral reasoning. Trained on vast corpora of human-produced text, LLMs act as compressed, generative “condensates” of collective symbolic behavior that make large-scale patterns in discourse computationally accessible—if used with awareness of their epistemic limits and the effects of alignment/fine-tuning.

Key Points

• New ambition in AI research: beyond productivity (tools) and alignment (safety), LLMs as instruments for social and behavioral science.
• LLMs encode large-scale regularities in argumentation, justification, narration, and norm negotiation; they can surface cultural patterns at scale.
• Distinction between model types matters:
  • Base (pretrained) models retain broad cultural regularities learned from raw corpora.
  • Fine-tuned / aligned models can systematically reshape or obscure those regularities.
  • Instruct-only and modular adaptation regimes are pragmatic compromises for behavioral research (preserve pretraining knowledge while enabling safe, controllable behavior).
• Epistemic limits:
  • Model output is not direct evidence of individual human beliefs or frequencies; it reflects the training distribution, pretraining artifacts, and modeling choices.
  • Alignment and safety interventions can remove, blunt, or bias signals from pretraining corpora.
  • Sampling choices (temperature, decoding method), prompt phrasing, and tokenization affect results and representativeness.
• Methodological approaches reviewed:
  • Prompt-based experiments (analogous to survey/experiment designs).
  • Synthetic population sampling (constructing weighted prompt sets to approximate populations).
  • Comparative-historical modeling (using models to simulate or reconstruct discourse across time/space).
  • Ablation studies (identify which features/weights contribute to behaviors).
• Mapping to social-scientific traditions: computational social science, content analysis, survey research, comparative-historical inquiry—LLMs scale and automate many of these tasks but require adapted inferential logic.

Data & Methods

• Data source: pretrained LLMs that have absorbed very large, heterogeneous collections of human text (web, books, forums, etc.). The paper treats model weights as compressed summaries of those texts.
• Methods discussed and their operationalization:
  • Prompt-based experiments:
    • Design prompt families that mirror survey questions or experimental vignettes.
    • Vary framing, demographic cues, context to probe sensitivities.
    • Use probabilistic sampling of tokens to produce distributions of responses rather than single outputs.
  • Synthetic population sampling:
    • Construct sets of prompts representing demographic, geographic, ideological subpopulations.
    • Weight or sample prompts to approximate target population mixes.
  • Comparative-historical modeling:
    • Condition on temporal markers or corpora slices (when possible) to study changes in discourse over time.
    • Use models trained on era-specific corpora or fine-tuned on historical texts.
  • Ablation and modular analysis:
    • Remove or freeze modules (e.g., instruction-tuning layers) to see which components carry cultural signals.
    • Run controlled fine-tuning (instruct-only vs full fine-tune) to measure how alignment alters outputs.
• Validation and robustness:
  • Cross-model comparisons (different architectures, vintages, pretraining corpora).
  • Ground-truthing against surveys, human annotations, and external data where available.
  • Sensitivity checks for sampling temperature, decoding method, prompt phrasing, and model version.
• Practical recommendations:
  • Prefer base or minimally adapted models when the goal is cultural signal recovery.
  • If safety is a concern, consider modular approaches that preserve pretrained layers while adding constrained interfaces.
  • Publish model provenance (pretraining corpus, date, instruction-tuning procedures) and all prompt sets and sampling parameters.

Implications for AI Economics

• Measurement and inference
  • LLMs offer cost-effective, high-throughput proxies for measuring aggregate beliefs, narratives, and norms—potential substitutes or complements to traditional surveys and content analysis.
  • They can generate synthetic populations or counterfactual discourse to study likely responses to policy, messaging, or shocks at scale.
  • Economists must treat LLM outputs as samples from a model-conditioned discourse distribution, not direct samples of people; calibration to survey data and careful validation are essential.
• Forecasting and scenario analysis
  • LLMs can help generate scenario narratives, market sentiment summaries, and plausible agent responses for macroeconomic and policy stress tests.
  • Use with ensembles and cross-validation to mitigate model-specific biases.
• Policy design and regulation
  • Regulatory analysis can use LLMs to anticipate behavioral adaptations to policy changes (e.g., tax policy framing, regulation of platforms).
  • But alignment/fine-tuning policies by model providers may obscure or bias the signals regulators rely on; transparency about model training and tuning is economically important.
• Labor and productivity research
  • Cultural and normative signals embedded in LLMs can inform studies of task substitution, complementarities, and adoption costs for AI in workplaces (e.g., variation in how workers would integrate or resist AI tools across cultures).
• Market and welfare analysis
  • LLMs can surface moral preferences and normative constraints that affect demand, compliance, and firm strategies (e.g., ethical consumption trends, reputational risk dynamics).
  • Misreading model outputs as representative human preferences risks misallocating policy or firm effort—leading to welfare losses.
• Risks and caveats for economic work
  • Representativeness: training corpora are uneven across geographies, languages, and socioeconomic groups—biases propagate into inferences.
  • Alignment interventions: commercial models often undergo alignment that can remove or distort important cultural signals; economic analysis must account for these interventions when interpreting outputs.
  • Overreliance and feedback loops: using LLM-generated content to train policy or inform markets can create feedback loops that amplify model artifacts into real-world behavior.
• Recommendations for economists using LLMs
  • Use base or minimally adapted models when studying cultural patterns; if using aligned models, document the alignment regime and test its effects.
  • Validate LLM-based measures against independent survey or behavioral data; report uncertainty and sensitivity to prompts and sampling.
  • Combine LLM-generated insights with field or lab experiments for causal claims; use LLMs primarily for measurement, hypothesis generation, and large-scale descriptive work.
  • Advocate for model transparency (provenance, corpora summaries, tuning procedures) to improve reproducibility and policy relevance.

Summary takeaway: LLMs are powerful, scalable instruments for studying cultural and normative aspects relevant to economic questions, but robust inference requires deliberate methodological choices, careful validation, and attention to how alignment and sampling decisions shape the signal.