Machine predictions can backfire: a new Bayesian model shows that even a well-specified ML predictor can worsen outcomes if human priors are misaligned, meaning decision support can do more harm than good under realistic inference dynamics.

Main Finding

The paper introduces the 2-Step Agent, a computational framework that models how a Bayesian decision maker (agent) updates beliefs from an ML prediction and then uses those updated beliefs to choose interventions. Key results are: (1) a formal model of belief-updating from ML-based decision support (ML-DS); (2) a tractable analytic reduction in a linear Gaussian setting that collapses the training-data plate into sufficient statistics; and (3) simulation evidence that (i) ML-DS can improve decisions under some conditions, but (ii) even a single misaligned prior held by a fully rational agent can make ML-DS strictly harmful (worse outcomes than no decision support), even when the predictive model itself is well specified.

Key Points

• 2-Step Agent framework:
  • Step 1 (Bayesian update): the agent treats a model prediction for a new instance as compressed information about the model’s training data and updates a joint posterior over population-level parameters (e.g., data distribution, treatment policy, treatment effect).
  • Step 2 (Causal decision): the agent uses the posterior to draw a distribution over structural causal models and picks an action by aggregating Conditional Average Treatment Effects (CATEs) across that posterior.
• Important modelling choices: the agent knows the model class, signature (which covariates are used and that the model is treatment-naive), and training set size, but not the raw training data.
• Tractability result (linear Gaussian): in a linear, Gaussian, regression-without-intercept setting with k covariates, the likelihood of the model parameters can be expressed using a finite number of sufficient statistics (reducing the plate of interchangeable training datapoints). This makes the Bayesian update computationally tractable; the paper gives an explicit count of required sufficient statistics (function of k).
• Simulations (univariate case): with n = 1,000, continuous X, continuous A (but agent chooses between two fixed dosages A=10 and A=20), and linear outcome, the authors:
  • Define a default “correct” prior with small uncertainty.
  • Systematically perturb single prior means (e.g., prior mean on treatment effect, on historical treatment policy, on covariate mean) to model misalignment.
  • Compare outcomes with and without ML-DS.
• Empirical findings:
  • There are parameter regimes where ML-DS consistently improves effect estimates and downstream outcomes.
  • However, a single misaligned prior (e.g., wrong prior about the historical treatment policy or treatment effect) can lead a perfectly rational agent to update in a way that yields worse decisions and lower realized outcomes than if no prediction had been provided.
• Scope and limitations:
  • Framework focuses on learning from non-causal prediction models (predicting E[Y|X] only), not on ML models that output causal or interventional estimates.
  • Analytic tractability shown for linear-Gaussian regression; extension to more complex models requires further work.
  • Agent-side assumptions: agent is Bayesian-rational, has access to model documentation (class/signature/training size), and uses the prediction as information about latent training-data composition.

Data & Methods

• Formal setup:
  • Hist: a structural causal model (SCM) generating historical data tuples (X, A, Y).
  • D_Hist: dataset sampled from Hist used to train a predictive model f that estimates E_Hist[Y | X].
  • Agent model A_hist: a Bayesian network capturing the agent’s joint prior over population-level parameters (α_X, α_A, α_Y), per-instance noise variables (plate of interchangeable NX_i, NY_i, NA_i), treatment-effect latent NE, and the trained model parameters θ (derived deterministically from the plate).
  • Agent observes X_o (new instance) and Pred_o = f(X_o) and performs a Bayesian update P(A_hist | X_o, Pred_o).
  • Agent inference model A_inf: draws α and NE from that posterior and computes CATE for the decision problem EA_inf[Y | X_o, do(A=20)] − EA_inf[Y | X_o, do(A=10)], then applies a threshold τ to decide.
• Tractability theorem (informal):
  • In linear structural equations, continuous Gaussian noises, and linear regression predictive model (no intercept), the likelihood of θ depends only on population-level variables and a polynomial number of sufficient statistics (collapsing the plate).
  • This reduction enables exact/tractable posterior computation in that setting.
• Experimental details (representative univariate experiment):
  • Data-generating Hist: X ~ Normal(α_Xµ, α_Xσ), A = 0.125 X + noise, Y = 12 − 0.1 X + NE * A + noise.
  • Predictive model: least-squares linear regression without intercept predicting Y from X (θX).
  • Agent priors (default “correct” agent): centered on true values with small variance (examples: α_Xµ ~ N(80,0.1), NE ~ N(1,1), etc.).
  • Interventions: agent chooses between A=10 and A=20 based on posterior-averaged CATE.
  • Experiments systematically vary single prior means to simulate misalignment and compare realized outcomes under ML-DS vs no ML-DS.
• Reproducibility: code and experiments reported as open-source (placeholder link provided).

Implications for AI Economics

• Value of predictive decision support is endogenous to agent beliefs:
  • Economic evaluations (e.g., welfare analysis, cost–benefit, adoption models) must incorporate how humans update beliefs from model outputs and how heterogeneous priors across agents can alter realized benefits or harms.
• Risk of well-specified models producing worse welfare:
  • Even with a well-specified predictive model and rational agents, deployment can decrease welfare if users’ priors are misaligned. Policy assessments and regulatory approvals should not assume model accuracy alone guarantees net benefit.
• Design and deployment recommendations:
  • Improve transparency about training-data composition and historical policies (document historical treatment policies, sample composition, and training objectives) so agents can form better-aligned priors.
  • Supplement predictions with information intended to calibrate priors (e.g., cohort summaries, confidence intervals about training policies), or provide causal/what-if outputs when appropriate.
  • Consider structured onboarding or training to align priors across practitioners, especially in high-stakes settings (medicine, criminal justice, welfare).
  • Use experimental evaluation designs that account for provider learning dynamics (e.g., two-stage randomization or provider-level randomization) to avoid confounding from heterogeneous prior updates.
• Modeling and research implications for economists:
  • Incorporate belief-updating dynamics in models of technology adoption and diffusion, accounting for heterogeneous priors and Bayesian learning from outputs (not just from outcomes).
  • When estimating the welfare effect of ML-DS, include counterfactuals of no-prediction and heterogeneity in agent priors; robustness checks should consider worst-case prior misalignments.
  • The tractable reduction in linear-Gaussian settings offers a pathway for quantitative welfare simulations; extensions to realistic non-linear/black-box models and empirical calibration of human priors are important next steps.
• Regulatory implications:
  • Standards like the EU AI Act that emphasize human oversight should consider guidance on documenting historical treatment policies and on measures to mitigate harms from misaligned human priors.
  • Certification/evaluation protocols for ML-DS should test not only model accuracy but also the interaction between model outputs and representative decision-makers’ prior beliefs.

Summary: The 2-Step Agent formalizes how predictions act as signals about training data and shows that learning from ML-DS is fundamentally a belief-revision process that can improve or worsen outcomes depending on prior alignment. For economists and policymakers, this highlights the importance of modeling endogenous human learning, ensuring transparency about model provenance, and designing interventions (training, documentation, experimental trials) to align priors before large-scale ML-DS deployment.