Adding LLM-generated predictions to regression adjustment modestly sharpens randomized-experiment estimates and will not worsen unbiased estimates, with the biggest gains when rich text or unstructured data are available.

Main Finding

Including AI-generated predictions (e.g., LLM “digital twin” outputs or other black-box model forecasts) as covariates in standard regression adjustment reliably reduces variance in randomized experiments and carries a practical “do no harm” guarantee: when predictions are uninformative the adjusted estimator reverts to the unadjusted difference-in-means. No new estimators are required — calibrated regression adjustment (equivalently, adjustment for an estimated prognostic score or a linear-calibrated form of PPI/AIPW) gives unbiased (or asymptotically unbiased) treatment effect estimates with weakly smaller variance.

Key Points

• Core idea: treat AI predictions Æ_i(z) as auxiliary covariates and include them in regression adjustment. This is analogous to adjusting for an estimated prognostic score.
• Do no harm: calibrated regression adjustment never increases asymptotic variance. If corr(Y(z), Æ(z)) = 0 the design effect equals 1, so you recover the unadjusted estimator.
• Simple implementation: run OLS of outcome on treatment indicator and mean-centered predictions (optionally include treatment × prediction interaction). Use heteroskedasticity-robust SEs. The interacted specification (Lin-style) removes small O(1/n) finite-sample bias.
• Efficiency gain formula (intuition): design effect in arm z ≈ 1 − ρ(z,z)^2, where ρ is the correlation between the outcome and the AI prediction in that arm. Gains depend directly on prediction quality.
• Comparisons:
  • Model-assisted / uncalibrated PPI: can be unbiased but can inflate variance unless predictions are strong (requires high correlation threshold, e.g., ρ > 0.5 in some cases).
  • Digital-twin / imputation estimator (imputing missing potential outcomes with AI) is biased unless AI predictions are correct on average — risky in practice.
  • Regression adjustment is equivalent (in randomized experiments) to a calibrated AIPW / prognostic-score ANCOVA and to certain CUPED/CALM calibrated variants.
• Practical tips for discrete outcomes: use continuous predicted probabilities (extract log-probabilities from token distributions, average across multiple stochastic draws) rather than discretized outputs.
• LLMs are most valuable when predictive signal lies in unstructured or textual inputs; with purely structured tabular features, traditional methods (ridge, XGBoost) can match or beat LLM predictions. Combining LLM-derived features and tabular covariates often performs best.
• Empirical results: simulations and three applications (survey mega-study with digital twins, email marketing A/B test, large-scale platform experiment) show modest but real efficiency gains; gains larger when unstructured text is present. The do-no-harm property holds empirically.

Data & Methods

• Framework: finite-population randomized experiment (SATE target), allow predictions Æ_i(z) for both potential outcomes (may depend on z). Randomness only from random assignment Z.
• Estimators discussed:
  • Unadjusted difference-in-means (baseline).
  • Model-assisted bias-corrected estimator (control-variates / PPI style): unbiased but can inflate variance when predictions are weak.
  • Calibrated regression adjustment: estimate within-arm slopes β̂_z (OLS of Y on Æ(z) within arm z) or fit a single OLS Y ~ 1 + Z + Æ(Z) (+ Z×Æ(Z)). This yields the calibrated estimator with guaranteed weak variance reduction.
  • Digital-twin (imputation) estimator: impute missing potential outcome by AI predictions — biased unless Æ is correct on average.
  • Equivalences: calibrated regression = AIPW with Æ as outcome model = ANCOVA on estimated prognostic score.
• Inference: use heteroskedasticity-robust SEs; Lin-style interacted regression reduces finite-sample bias.
• Handling discrete/counted outcomes: calibrate continuous predicted probabilities; obtain probabilities via token log-probs, or average multiple model samples to get smoother scores; prefer continuous calibration over thresholding.
• Feature engineering: LLMs can featurize unstructured inputs (summaries, embeddings, categorical labels) to augment traditional covariates; include these features in the regression adjustment.
• Empirical evaluation: synthetic simulations plus three real-world applications (survey mega-study with digital twins, an email A/B test with tabular covariates, and a large-scale platform experiment). Across these, regression adjustment with AI predictions produced modest efficiency gains and preserved unbiasedness.

Implications for AI Economics

• Practical recommendation for experimenters: routinely include AI-generated predictions (and LLM-derived features for text) as covariates in regression adjustment for randomized experiments. Low implementation cost, minimal risk (do-no-harm property), and measurable efficiency gains justify adoption as standard practice.
• Efficiency & cost tradeoffs: modest variance reductions translate into smaller required sample sizes or higher power for the same sample budget. This is economically valuable for firms/researchers running many A/B tests or costly field trials — especially when experiments involve textual or other unstructured data that LLMs can exploit.
• When to expect large gains: settings where treatment-relevant signal is contained in rich unstructured inputs (messages, open-ended survey text, user-generated content). In fully structured tabular domains, gains may be limited and traditional ML can suffice.
• Design and policy applications: improved precision enables more granular heterogeneity analysis, faster iteration for product design, and more efficient allocation of testing budgets. LLM featurization can also facilitate stratification and more informed adaptive designs.
• Cautions and limits:
  • Prediction quality matters: poor predictions yield limited gains; uncalibrated uses (digital twins as substitutes, uncalibrated PPI) risk bias or inflated variance.
  • Operational costs and stability: generating large-scale LLM predictions incurs compute costs and introduces dependencies on model versions (drift, reproducibility). These economic costs should be weighed against sample-size savings.
  • Not a substitute for real data: while AI helps efficiency, substituting synthetic respondents for human data without correction risks bias; the recommended practice is hybrid adjustment, not replacement.
• Research directions for AI economics: quantify the return-on-investment of model-assisted adjustment across domains (compute cost vs. sample-size savings), study model update / drift effects on long-running test portfolios, and integrate AI-featurization into optimal experimental design and stratification policies.