Observational causal‑ML can reproduce randomized trial results — but only with careful modeling. In a large tech firm's feature rollout, estimators matched the RCT benchmark only after flexible learners, thoughtful covariate engineering and robustness checks were applied.

Main Finding

Observational causal methods can recover experimental (ground-truth) treatment effects in large, high-dimensional product-release data—but only when estimation design choices follow careful best practices. With proper trimming, cross‑fitted hyperparameter tuning, and doubly robust estimation that flexibly models both propensity and outcome functions, the authors fully recover the experimental ATE for a continuous device‑performance outcome. For a binary performance outcome, however, even the best observational strategies leave substantial residual bias (observational estimate: −54% vs experimental truth: −43%), implying missing confounding that the observed covariates do not capture.

Key Points

• Data structure: a paired dataset consisting of (a) an experimental rollout (randomized treatment assignment) and (b) an observational sample where users self‑opted into the same feature. Same outcomes and rich covariates are available in both samples, enabling direct benchmark comparison.
• Directional agreement but magnitude error: naive differences-in-means in the observational sample give the correct sign for both outcomes, but materially misstate magnitudes due to selection.
• Continuous outcome: careful application of modern methods (doubly robust estimators with flexible ML nuisance models, tuned via sample-splitting, plus overlap trimming and model averaging) removes selection bias and matches the experimental estimate.
• Binary outcome: none of the observational methods come close to the experimental benchmark; available covariates explain almost none of the binary outcome variation, suggesting important unobserved confounders.
• Estimator risk: flexible ML helps but can harm if mis‑tuned. Without proper hyperparameter tuning and sample‑splitting, doubly robust ML estimators can be as biased as the unconditional mean difference.
• Best-practice checklist reinforced:
  • Trim to enforce overlap (Crump et al. 2009).
  • Use doubly robust/AIPW estimators that combine outcome and propensity models.
  • Estimate nuisance functions with flexible ML but tune hyperparameters using out‑of‑sample validation / cross‑fitting.
  • Average across models/ensembles to reduce model uncertainty.
  • Report and check predictive performance of nuisance models (poor fit suggests limited ability to satisfy unconfoundedness).
  • Conduct sensitivity analyses (e.g., Chernozhukov et al. 2022) and consider benchmarks other than zero.
• Heterogeneity: validated CATE detection with DR‑score (Chernozhukov et al., 2024). Both experimental and observational samples indicate heterogeneity for the binary outcome (e.g., higher‑usage devices see larger declines), and preferred CATE estimators identify similar drivers of heterogeneity across samples—although selection bias persists within subgroups in the observational sample.

Data & Methods

• Data:
  • Source: Microsoft paired dataset (publicly released, masked features) covering two weeks in December 2022.
  • Units: Windows devices. Treatment = enabling a new software feature (binary). Treatment rare: observational sample 4.5% opted in; experimental assignment 2.6% treated.
  • Outcomes: one continuous and one binary device‑performance metric (both masked).
  • Covariates: rich device specs and usage summaries (anonymized/masked; some truncation/collapsing applied for release).
  • Public replication materials available at https://github.com/microsoft/Reevaluating-Causal-Estimation-Methods.
• Identification assumptions discussed:
  • Unconfoundedness (selection on observables) and overlap/common support.
  • Practical mitigation: trimming based on estimated propensity score (Crump et al., 2009) to estimate overlap‑restricted ATE.
• Estimators evaluated:
  • Outcome modeling: linear regression and two‑model (Oaxaca‑Blinder) approaches; nonparametric/ML outcome models.
  • Propensity‑based methods: propensity‑score matching, inverse propensity weighting (IPW).
  • Doubly robust methods: Augmented IPW (AIPW) combining outcome and propensity estimates.
  • Modern ML for nuisance functions: random forests, boosting, LASSO, etc., with emphasis on hyperparameter tuning and sample‑splitting / cross‑fitting.
  • Averaging/ensembling across estimators to reduce model uncertainty (Breiman-style model averaging).
  • Sensitivity analysis following Chernozhukov et al. (2022) and heterogeneity detection using DR‑score (Chernozhukov et al., 2024).
• Validation strategy:
  • Compare observational estimates directly to experimental estimates (gold standard).
  • Inspect predictive fit of nuisance models and perform sensitivity checks beyond the usual "is effect plausibly nonzero" test.

Implications for AI Economics

• Observational evaluation of AI/product features is feasible—but not automatic. When randomized experiments are impossible or costly, researchers can still obtain credible causal estimates if they:
  • Collect rich, relevant covariates that plausibly capture selection into treatment.
  • Use doubly robust estimators that flexibly model both treatment and outcome processes.
  • Tune ML hyperparameters with out‑of‑sample validation and use cross‑fitting to avoid overfitting nuisance estimators.
  • Trim to enforce overlap and be explicit that estimates apply to the overlap population (not necessarily the full population).
  • Use ensemble/model‑averaging to address model selection uncertainty.
  • Report predictive performance of nuisance models and run sensitivity analyses; passing a null‑sensitivity check is necessary but not sufficient when effect magnitudes matter.
• Practical caution: binary outcomes and rare treatments are particularly challenging—poorly explained outcomes or rare treatment status make unconfoundedness less plausible and leave residual bias even under sophisticated methods.
• For firms and policymakers evaluating AI-driven product changes, the paper implies:
  • Prefer randomized rollouts where feasible; when relying on observational data, allocate resources to measure pre‑treatment behavior and device/user features likely to drive selection.
  • Validate observational pipelines when possible by holding out randomized cohorts or using paired experimental data; use those validations to calibrate trust in observational estimates.
  • When estimating heterogeneous effects (personalization, targeted rollouts), use validated CATE methods (e.g., DR‑score, cross‑fitted CATE learners) and check whether subgroups identified from observational data align with experimental heterogeneity.
• Research implication: paired observational–experimental benchmarks (like this Microsoft release) are valuable for stress‑testing modern causal ML methods and for developing practical "design‑stage" guidelines for credible causal inference in AI evaluations.