Split-sample IV from multiple ML/LLM outputs cleans up biased 'generated regressors' — and often reveals much larger effects: applied to German parliamentary speeches and Chinese firm political-risk measures, corrected estimates rise substantially (often 50–100%) compared with conventional regressions.

Main Finding

Prediction-generated regressors (e.g., outputs from LLMs or other ML estimators) used as right‑hand‑side variables in linear regressions can produce substantial bias from measurement error. A simple, practical fix is to create instrumental variables by repeatedly measuring the same latent feature on independent random splits of the data and using one split’s measure as an instrument for another. Under mild assumptions (mainly that measurement noise is independent across observations conditional on inputs), the split-sample IV is valid and yields consistent estimates; in simulations and two empirical re-analyses it substantially reduces bias and often increases estimated effect sizes.

Key Points

• Problem: ML/LLM-generated measures (ˆX) used as regressors are measured with error η; plugging ˆX into regressions produces endogeneity and biased estimates (not always classical attenuation).
• Proposed solution: Randomly split the dataset into independent subsamples, produce separate measures of the same latent quantity from each split, and use one split’s measure as an instrument for the other. Intuition: different splits share signal (the latent X) but have (conditionally) independent measurement noise.
• Theoretical validity: Under (i) additive mean-zero measurement error and (ii) independence of measurement noise across observations conditional on inputs, the split-sample instruments satisfy exogeneity and exclusion restrictions and the resulting IV estimator is consistent.
• Practicality: method requires no additional labeled/validation data, is simple to implement (standard 2SLS), and can be applied with existing software.
• Empirical and simulation evidence: simulations show OLS can be biased (e.g., estimates about half the true value or otherwise distorted), while split-sample IV recovers parameters closely (often within 0.01) with reasonably sized samples. Re-analyses: (1) German Parliament speech gendered-language study (Ash et al. 2025) — IV-corrected effects larger by ≈50–100% for most outcomes; (2) Political-risk and firm spending in China (Lin 2025) — IV-corrected effect remains positive, significant, and larger in magnitude.
• Relation to literature: contrasts with approaches that require labeled validation data or parametric measurement-error models (Yang et al., Burtch et al., Battaglia et al., Duan & Pelger). The split-sample IV needs no ground-truth labels but does require the independence of measurement noise across observations/splits.

Data & Methods

• Setup:
  • Observations {(Yi, Zi)} with latent one-dimensional feature Xi = f(Zi) (f may be an LLM or other ML predictor).
  • Only noisy estimate ˆXi available: ˆXi = Xi + ηi, with E[ηi | Zi] = 0 and ηi independent across i (Assumption 1).
  • Target model: Yi = β0 + β1 Xi + εi, where E[εi | Xi] = 0.
• Split-sample IV algorithm (high-level):
• Randomly split the original dataset into two (or more) independent subsamples; the split randomization must be independent of {Zi, ηi, εi}.
• For each observation i in the analysis sample, obtain a second measure ˆX_instr,i by applying the same prediction procedure to a matching observation in the other subsample (matching can use Zi and the split randomization; nearest-neighbor matching on Zi is suggested when appropriate).
• Use ˆX_instr as an instrument for ˆX in a 2SLS regression of Yi on ˆX (and residualized covariates if present).
• Optionally repeat with k splits or multiple instruments and check first-stage F-statistics.
• Theoretical results:
  • Prove instrument relevance (correlation via shared Xi) and exogeneity (instrument noise uncorrelated with error term because it comes from a different independently measured observation).
  • Establish consistency of the IV estimator and derive finite-sample properties via simulation.
• Simulations:
  • Designs include document-term/count models and embedding-based measures that mimic empirical structures.
  • Tested several measurement-error structures (classical and non-classical).
  • Found split-sample IV performs well in recovering β1 across designs, often with small finite-sample bias.
• Empirical applications:
  • Re-analysis of Ash et al. (2025): gendered speech measure from topic models on >500k speeches. IV-correction increases estimated effects on reactions and interactions (≈50–100% larger).
  • Re-analysis of Lin (2025): firm-level political risk from BERT on 418k Q&As; IV-correction yields a larger positive effect of political risk on poverty-alleviation spending.

Implications for AI Economics

• Awareness: Many empirical studies in political economy, media economics, and firm behavior use ML/LLM outputs as regressors. If researchers ignore measurement error in these generated regressors, estimated causal or associative effects can be biased — frequently underestimated (attenuated) but not always.
• Practical mitigation: The split-sample IV is a low-cost, widely applicable correction that does not require ground-truth labels. It is particularly attractive when researchers have large datasets and rely on pretrained models or stochastic LLM outputs.
• Trade-offs and diagnostics:
  • Trade-offs: splitting reduces effective sample size and can increase variance (efficiency loss) relative to estimators that exploit validation labels or parametric measurement-error structure. Use multiple splits (k-fold) or more instruments to improve strength.
  • Crucial assumption: measurement noise must be independent across observations conditional on inputs. If measurement errors are correlated across observations (e.g., via shared prompts, deterministic model outputs, network/geographic spillovers, or unaccounted-for data structure), instruments may be invalid and bias can remain.
  • Practical checks: assess first-stage strength (F-stat), try multiple independent splits, examine sensitivity to matching rules, and when possible compare with approaches that use labeled validation data for efficiency gains.
• Research directions:
  • Extend methodology to settings with correlated measurement errors (networked or spatial data).
  • Combine split-sample IV with small labeled validation sets to gain efficiency while retaining robustness.
  • Formal guidance for optimal matching/assignment between observations across splits and for choosing number/size of splits given finite-sample trade-offs.
• Bottom line for AI economics: measurement error is a material concern when using LLM/ML outputs as regressors. The split-sample IV provides an accessible, theoretically grounded tool to reduce bias and often reveals substantially larger effects than naive OLS — altering substantive conclusions in applied work.