Predictive models can erode their own reliability: the more a model changes behavior, the weaker the guarantees on its generalization, with worst-case scenarios where sampled users deceptively validate predictions while the broader population contradicts them; in some settings retraining on performatively distorted samples can improve guarantees, illustrated using German job‑training assignment data.

Main Finding

Performative Learning Theory (PLT) embeds performative predictions into statistical learning theory and proves generalization bounds when predictions change the data-generating process. The paper shows that performativity complicates generalization in two opposing ways: the population can self-negate predictions (worst-case, min–max behavior) while the training sample can deceptively self-fulfill them (min–min behavior), producing an empirical echo chamber. Under generic Wasserstein-sensitivity assumptions on the performative transition map, PLT derives finite-sample Wasserstein- and expectation-based generalization bounds for scenarios where performativity acts on the sample, on the population, or on both. A notable and counterintuitive implication is that retraining on performatively distorted samples can improve generalization guarantees. The theory is illustrated on administrative German job-seeker data (1975–2017; raw >60M rows), showing the trade-off between changing the world and reliably learning from it.

Key Points

• Problem scope

  • Extends prior performative prediction (PP) work by studying learnability from finite samples when performativity affects the sample, the population, or both.
  • Poses four research questions (RQ1–RQ4) about excess risk and cumulative/inferential gaps under different retraining and performativity setups.

• Conceptual insights

  • Two antagonistic phenomena:
    • Self-negating performativity (population reacts to erase predicted effects): modeled as min–max risk functional in Wasserstein space (worst-case / distributionally robust).
    • Self-fulfilling performativity (sample reacts to confirm predictions): modeled as min–min risk functional (empirical echo chamber / favorable learning).
  • Fundamental trade-off: more performative change (stronger world-changing interventions) increases difficulty of reliably learning population properties from samples that themselves are reacting.
  • The performative response rate m/n (number of units in sample that change behavior divided by sample size) is a key observable that feeds bounds and estimators.

• Theoretical contributions

  • Formal PLT framework that defines risk, empirical risk minimization (ERM) and repeated (empirical) risk minimization (RRM / RERM) in stateful performative settings.
  • Finite-sample generalization bounds under mild assumptions (compact parameter space, strong convexity of loss, Lipschitz/Wasserstein sensitivity of transition map).
  • Casting of worst- and best-case performative effects into min–max and min–min learning problems and using tools from empirical process theory and Wasserstein duality to bound expectation differences.
  • Corollary showing retraining on performatively distorted samples can tighten bounds.

• Practical example domains

  • Routing apps (strategic/endogenous performativity — drivers reroute when shown congestion predictions).
  • Public employment services (exogenous/treatment performativity — assignment to job training alters unemployment risk). The latter is used for empirical illustration.

Data & Methods

• Theoretical methods

  • Assumptions (subset of prior PP literature):
    • Condition 3.1: loss ℓ(z, θ) is γ-strongly convex in θ (uniqueness of minimizer).
    • Condition 3.2: transition map Tr is (ε, p)-jointly Wasserstein-sensitive: Wp(Tr(d, θ), Tr(d′, θ′)) ≤ ε Wp(d, d′) + ε ||θ − θ′||2 for ε > 0 and 1 ≤ p ≤ max{2, ν/2}.
    • Condition 3.3: ℓ is Lℓ-Lipschitz in z and κ-Lipschitz in gradients w.r.t. θ.
    • Θ is compact and convex; Y, X bounded.
  • Tools used:
    • p-Wasserstein distance and its concentration (Fournier & Guillin-type bounds).
    • Kantorovich–Rubinstein duality to relate Wasserstein distances to expectation differences.
    • Empirical process and distributionally robust learning theory to characterize min–max/min–min risk functionals.
  • Key technical lemmas (representative):
    • Lemma (Wasserstein concentration): Wp(bd0, d0) ≤ β0 with high probability, exponential tail in n.
    • Lemma 3.5 (In-sample performative shift bound): If at most m sample units change per iteration, then Wp(bd0, bdT) ≤ [(ε(1 + La))^(T−1) / (ε(1 + La) − 1)] · (m/n)^{1/p} · DZ, where La is the Lipschitz constant of the minimizer mapping G, DZ = sup_z,z' ||z − z'||2.
  • From these Wasserstein bounds, expectation/generalization-gap bounds are derived for the RQ scenarios (RQ1–RQ3).

• Empirical illustration

  • Data: administrative labor market records from the German Federal Employment Agency (1975–2017; raw dataset >60M rows).
  • Application: model-assisted assignment of job trainings based on predicted long-term unemployment risk; performed experiments/analyses to estimate performative response rates and demonstrate the bound behavior (e.g., Figure 1.3 shows how generalization-gap bound grows with performativity and components of the bound).
  • The illustration maps theoretical quantities (ε, m/n, La, T) to empirical estimates to visualize the change-vs-learn trade-off.

Implications for AI Economics

• Evaluation and external validity

  • Testing ML systems on limited, visible samples can systematically misestimate performance when deployment causes population-level reactions. Sample-based results may be biased by self-fulfilling dynamics (echo chambers) and not transfer to the wider population.
  • Practitioners should explicitly account for performative response rates (m/n) and sensitivity ε when extrapolating from experiments or pilots to full rollouts.

• Policy, regulation, and welfare

  • Predictive-driven interventions (e.g., targeting of training, insurance pricing, routing, lending) can change distributions and create social externalities; bounds like these quantify how such interventions reduce learnability and increase uncertainty about outcomes.
  • Regulators and institutions should monitor not only downstream outcomes but also how the act of prediction modifies behavior—audits should check for performative feedback loops.

• Experimental design & deployment

  • Randomized/controlled rollouts that hide predictions from parts of the population (or use double-blind designs) help identify performative effects and estimate response rates, improving calibration of generalization bounds.
  • Retraining strategies can sometimes improve population generalization: retraining on performatively distorted samples, rather than simply retraining on untouched data, can reduce some sources of uncertainty—this suggests adaptive retraining schedules and carefully designed online learning policies.

• Robustness & mechanism design

  • The min–max perspective connects performative learning to distributionally robust optimization; when worst-case self-negating responses are plausible, robust objectives and conservative deployments are warranted.
  • Conversely, if sample self-fulfillment dominates (min–min), optimistic inferences can be misleading; mechanism designers should guard against incentives that induce favorable but unrepresentative sample responses.

• Limitations and open directions relevant to economics

  • The bounds rely on Wasserstein-sensitivity and strong convexity assumptions; real-world response dynamics may violate these or be more complex (multi-agent strategic equilibria, long-range dynamics), which can make bounds loose.
  • Tr(·, ·) is unknown in practice; estimating ε and the form of Tr is challenging—econometric strategies (RCTs, instrumental variables, structural models) are complementary to PLT for identification.
  • Extensions needed: tighter bounds under stronger structural assumptions, finite-sample inferential theory for stateful processes (RQ4 extension), and richer agent-behavior models (strategic, boundedly rational, network effects).

Summary recommendation for AI economists: treat predictions as policy instruments—explicitly model and measure how predictions change behavior, estimate performative response rates during pilots, and incorporate PLT-style bounds and robustness when scaling predictive systems that affect outcomes.