When treatments leak, old controls fail: standard matching and DiD break down without uncontaminated controls, but forecasting-based counterfactuals — interrupted time series and ML forecasts — can recover short-run direct and spillover effects under plausible stability assumptions.

Main Finding

Forecast-based counterfactual methods (FBCMs) — e.g., interrupted time-series and machine-learning control methods — can identify meaningful aggregate treatment and spillover effects even when interference is pervasive and no uncontaminated control units exist. Control-based counterfactual methods (CBCMs) such as DiD and matching require clean (unexposed) controls or strong, often untestable, assumptions about the interaction/exposure structure; when those are absent or misspecified, CBCMs either fail or produce biased/ambiguous estimates. However, FBCMs trade the need for uncontaminated controls for assumptions about temporal stability and short-term predictability of untreated dynamics, and they generally cannot recover exposure-specific effects without some information on the exposure mapping.

Key Points

• Problem focus: interference/spillovers violate SUTVA; standard treated-vs-control comparisons confound direct and spillover effects.
• Two broad counterfactual strategies contrasted:
  • Control-based counterfactual methods (CBCMs): matching, difference-in-differences (DiD), doubly robust estimators. Rely on (observed or assumed) uncontaminated controls or correctly specified exposure mappings.
  • Forecast-based counterfactual methods (FBCMs): interrupted time-series (ITS), machine-learning control methods (MLCM). Construct counterfactuals via time-series/panel forecasts rather than cross-sectional untreated units.
• Identification insights:
  • Aggregate policy effects relative to the no-treatment/no-exposure counterfactual can be identified without specifying an exposure mapping if one can credibly forecast the untreated outcome path.
  • Exposure-specific effects (direct vs indirect at particular exposure levels) still require information on the exposure mapping and are sensitive to misspecification.
  • CBCMs identify direct and spillover effects only under assumptions like partial interference, correctly specified exposure maps, or existence of uncontaminated controls; misspecification of exposure mapping contaminates "clean" controls and biases estimates.
  • FBCMs remain viable when all units are directly or indirectly affected (no-clean-control), but their credibility hinges on temporal stability, the predictability of untreated outcomes, and is typically more defensible for short-run effects.
• Simulation evidence:
  • Uses actual U.S. county geography, queen contiguity spillovers, 200 treated counties, 5 pre + 1 post period, complex outcome dynamics (level differences, persistent covariates, nonlinearities, AR adjustments).
  • CBCM (doubly robust estimator) performance deteriorates with exposure-mapping misspecification; apparent clean controls become contaminated.
  • FBCMs produce unit-level counterfactuals independent of the assumed mapping; aggregate policy effect estimates are robust to mapping errors, while exposure-specific aggregations can still be sensitive.
• Empirical application:
  • Reanalysis of Di Tella & Schargrodsky (2004) on police protection and car theft in Buenos Aires.
  • CBCM (DiD with distance-defined exposure) works when spatial structure of spillovers is well approximated; MLCM (forecast-based) remains informative when spatial spillover structure is uncertain, highlighting the practical trade-offs.

Data & Methods

• Formal setup:
  • Balanced panel i = 1..N, t = 1..T, treatment common adoption date T0, time-invariant assignment Di.
  • Exposure S_i = Σ_{j≠i} w_ij D_j (weight matrix W encodes spatial/network interactions); effective exposure S_it = S_i 1{t ≥ T0}.
  • Potential outcomes Y_it(d, s) indexed by own treatment d ∈ {0,1} and exposure s ∈ support(S).
  • Defines key estimands: ATT, direct effect DE_t(s), indirect effect IE_t(s), total effect TE_t(s), and decompositions.
• Identification analysis:
  • Characterizes four interference scenarios (no spillovers; selective/partial interference; pervasive interference/no clean controls; within-treated interference) and derives identification conditions for CBCMs and FBCMs in each case.
  • Shows when CBCMs identify ATT/DE/IE/TE and when they fail (notably under pervasive or ill-defined spillovers).
  • Shows how FBCMs identify aggregate effects by forecasting the untreated outcome path; highlights dependence on stationarity/predictability assumptions and short-run validity.
• Simulation study:
  • Geography-based Monte Carlo with known true exposure mapping (queen contiguity) and two misspecified mappings (centroid distance thresholds).
  • Methods compared: doubly robust CBCM; two FBCMs (including MLCM).
  • Performance metrics: bias and contamination of control groups; sensitivity of exposure-specific vs full-population estimates.
• Empirical reanalysis:
  • Reapplies DiD (distance-exposure groups) and MLCM to Buenos Aires policing intervention; compares estimates and interprets differences through the lens of interference and control contamination.

Implications for AI Economics

• Common AI-economics settings where interference matters:
  • Firm-level AI adoption causing knowledge/competition spillovers across nearby firms or supply-chain partners.
  • Platform-level AI interventions (recommendation algorithm changes, pricing rules) affecting users and connected sellers/buyers.
  • Labor-market AI impacts where displaced workers move across local labor markets, producing spatial spillovers.
  • Sectoral or regional diffusion of AI-driven productivity where treated units affect peers through markets or networks.
• Practical guidance for researchers evaluating AI policies/interventions:
  • Assess plausibility of uncontaminated controls. If many units plausibly exposed (no clean control), prefer forecast-based approaches to avoid cross-sectional contamination.
  • Use forecast-based (ML) controls when (a) you have enough pre-treatment periods to validate predictive performance, and (b) untreated dynamics are stable/predictable at the relevant horizon. Emphasize short-run causal claims when long-run predictability is doubtful.
  • When exposure-specific (e.g., peer-intensity) effects are policy-relevant, collect or model exposure mappings (networks, proximities, market links) carefully — but treat mapping assumptions as potentially fragile and run sensitivity checks.
  • Combine methods: use CBCMs where credible clean controls exist and FBCMs as robustness checks (or vice versa); compare estimates to detect contamination or model-dependence.
  • Validation and sensitivity practices:
    • Pre-treatment out-of-sample forecast validation and placebo post-treatment checks.
    • Re-run analyses with alternative exposure mappings and aggregation rules; report how exposure-specific vs aggregate estimates change.
    • Simulate plausible spillover mechanisms (e.g., via geography or supply chains) to assess bias risk under CBCMs.
  • Machine-learning control methods have advantages (flexible high-dimensional forecasting) but require careful regularization, cross-validation, and stability checks to avoid overfitting and misleading counterfactuals.
• Policy interpretation:
  • If the policy objective is aggregate impact (total effect on the whole system), FBCMs can be especially valuable because they do not require correctly specified exposure mappings.
  • If regulators or firms care about localized/direct vs displaced/indirect effects (who gains vs who loses), then credible exposure information is essential — CBCMs can estimate these when valid exposure groups exist, otherwise identification requires structural or instrumental assumptions.
• Research agenda suggestions for AI economics:
  • Develop and standardize pre-treatment forecast diagnostics and robustness protocols for ML-based counterfactuals in panel settings.
  • Combine network measurement efforts (administrative or transaction data) with forecast-based methods to better identify exposure-specific spillovers.
  • Explore hybrid designs that exploit staggered adoption, high-frequency outcomes, or exogenous shocks to strengthen identification under interference.

Overall: when evaluating AI-related interventions likely to generate spillovers, choose the counterfactual strategy to match what you can credibly assume or validate. Forecast-based methods broaden identification possibilities in contaminated settings but demand rigorous temporal-validation and typically yield more credible short-run inferences; exposure-specific causal claims still require additional structure or data on how treatments propagate.