A pretrained LLM can replace the predictive signal of many survey observations for some economic outcomes but offers little value for others; a new 'equivalent sample size' metric quantifies when LLMs are effective stand-ins for domain-specific data.

Main Finding

The paper introduces a practical, interpretable metric—the LLM's equivalent sample size (ESS)—that measures how many domain-specific human observations a pretrained LLM effectively substitutes for in a particular prediction task. The authors (1) formalize ESS as the smallest training sample N such that a conventional algorithm trained on N observations matches or beats the LLM’s expected prediction error, (2) provide estimators and a one-sided confidence procedure for ESS using block-out cross-validation and a sequential testing scheme, and (3) show substantial heterogeneity in ESS across economic prediction tasks in the PSID (e.g., ~20 observations for hourly wages vs ~600 for homeownership).

Key Points

• Definition: a-equivalent sample size N* = min{N : e_a(N) ≤ e_LLM}, where e_a(N) is expected error of algorithm a trained on N samples and e_LLM is the LLM’s (fixed) prediction risk.
• Interpretation: ESS converts pretrained LLM predictive value into an intuitive data-equivalence unit—how many real observations you would need to match the LLM.
• Estimation:
  • e_LLM estimated by empirical loss of the LLM on the full sample (no fine-tuning).
  • e_a(N) estimated by block-out cross-validation: partition data into disjoint training blocks of size N, train on each block, evaluate on complement, average test losses.
  • Plug-in estimator: smallest candidate N where estimated e_a(N) ≤ estimated e_LLM.
• Inference:
  • Sequential one-sided testing across increasing N yields a one-sided confidence interval (lower bound) for N*.
  • Because hypotheses are nested (larger N cannot increase e_a(N) under mild monotonicity), the sequential scheme controls coverage without multiplicity corrections.
• Theory: new asymptotic results (fixed-training-size regime) and a CLT for the block-out cross-validated risk estimator; consistent variance estimation—these underwrite valid inference for the ESS.
• Application (PSID, 2021): LLMs encode meaningful predictive information for some variables but much less for others—demonstrating that the utility of LLMs as data substitutes is task-dependent.
• Extensions: framework adapts to counterfactual prediction and CATE estimation under unconfoundedness (via transformed-outcome risk evaluation).

Data & Methods

• Data: Panel Study of Income Dynamics (PSID) 2021 wave used to evaluate diverse prediction tasks (hourly wages, homeownership, drinking, smoking, etc.).
• LLM protocol:
  • Use of fixed pretrained (static) open-source models with documented training cutoffs to reduce data-leakage concerns.
  • Predictions elicited by templated persona-style natural-language prompts mapping covariate vectors X → predicted Y.
  • No domain-specific fine-tuning; the LLM is treated as a fixed mapping.
• Comparator algorithms:
  • Any conventional learning procedure or structural model a that maps training data to a predictor; authors emphasize using flexible, well-suited algorithms as demanding benchmarks.
• Block-out cross-validation:
  • For candidate training sizes Nk, partition n observations into Bk = floor(n / Nk) disjoint training blocks of size Nk.
  • For each block b, train on that block and evaluate on the complement (test size n − Nk); average over blocks to estimate e_a(Nk).
  • Plugin estimator of ESS picks the smallest Nk where estimated e_a(Nk) ≤ estimated e_LLM.
• Sequential hypothesis test:
  • For ordered Nk, test H0,k: e_a(Nk) ≤ e_LLM vs H1,k: e_a(Nk) > e_LLM in sequence; stop when unable to reject to obtain a one-sided CI for N*.
• Asymptotics & inference:
  • Theory relies on a fixed-training-size asymptotic regime (training size held fixed while number of blocks grows), appropriate for testing across specific N values.
  • Derive CLT for cross-validated risk and consistent variance estimator; monotonicity assumption (e_a(N) weakly decreases in N) ensures nestedness and valid coverage without multiple-testing corrections.

Implications for AI Economics

• Practical decision rule for data collection: ESS gives a concrete cost–benefit metric—if ESS is small, a pretrained LLM may substitute for collecting many human observations; if large or infinite, collecting domain data or fine-tuning is warranted.
• Task-specific guidance: Because ESS varies widely across outcomes, researchers should evaluate LLMs per task rather than assume general-purpose substitution.
• Benchmarking standard: ESS provides an interpretable standard for reporting LLM utility in empirical economics—report the number of equivalent observations rather than only accuracy metrics.
• Experimental and survey substitution: ESS can inform when LLMs are reasonable surrogates for human subjects (low ESS) and when they are not (high ESS), complementing qualitative checks about biases and prompt sensitivity.
• Causal inference: The framework extends to CATEs under unconfoundedness, enabling an assessment of whether pretrained LLMs encode useful information for treatment-effect heterogeneity—though causal validity hinges on assumptions not guaranteed by pretraining.
• Methodological contributions: The asymptotic theory for block-out CV under fixed-training-size is useful beyond this ESS measure—for inference in settings where one wants to test algorithm performance at pre-specified training sizes.
• Caveats & limitations:
  • ESS depends on the chosen comparator algorithm a, loss function, prompt template, and the specific pretrained model (and its training data cutoff). Results are not absolute measures of an LLM’s knowledge.
  • The paper treats LLMs as fixed predictors (no fine-tuning). Fine-tuning or prompt engineering could materially change ESS.
  • Assumptions include i.i.d. data and a monotone (nonincreasing) learning curve for the comparator; departures may affect validity.
  • PSID application used static open-source models to mitigate data leakage; other LLMs with broader or later training data could show different ESS patterns.

Practical recommendation: use the ESS framework as a discipline-specific diagnostic—choose a strong, flexible benchmark algorithm, compute ESS with block-out CV and sequential inference, and use the result to guide whether to rely on pretrained LLM predictions, collect new data, or invest in fine-tuning.