Allowing everyone to game AI decision-support systems can entrench inequality: simultaneous recourse shifts selection thresholds so early winners set the benchmark and direct improvement, amplifying initial disparities and creating persistent performance gaps.

Main Finding

When actionable recourse (feasible feature changes recommended by an AI) is made available in competitive, fixed‑quota settings (select top ρ fraction), the resulting strategic interactions produce endogenous, closed‑loop dynamics that often amplify initial advantages. The top performers determine both the success threshold and the improvement direction; as many rejected candidates simultaneously invest to follow that direction, the benchmark shifts, making upward mobility harder. The model yields two kinds of stable outcomes: structural equilibria (selection depends only on immutable features so recourse is impossible) and effort‑suppressed equilibria (actionable direction exists but rejected agents optimally choose zero effort). Both mechanisms can generate persistent social stratification and limits to upward mobility.

Key Points

• Model setup
  • Population of n candidates with feature vectors xi ∈ Rd.
  • Designer uses a linear score s_i = w⊤xi and selects top ρ fraction (ρn).
  • Selection expressed as a maximization of an upper‑tail CVaR (CVaRupρ) of scores.
  • Regularization (quadratic, parameter λ) imposed to make w well‑posed.
• Optimization structure and geometric interpretation
  • CVaRupρ(w) = minη η + (1/(ρn)) Σ (w⊤xi − η)+; dual yields capped‑simplex weights αi with 0 ≤ αi ≤ 1/(ρn), Σαi = 1.
  • At optimum αi ∈ {0, 1/(ρn)} (nondegenerate), so w⋆ ∝ centroid of selected top‑ρ subset: w⋆ = (1/(λρn)) Σ_{i∈I} xi.
  • The classifier’s weight vector is therefore proportional to the centroid of the selected upper tail.
• Actionable recourse and cost
  • Features partitioned into actionable JA and immutable JN; recourse only allowed in JA.
  • Minimal quadratic recourse (if w⋆_A ≠ 0):
    • Optimal action: a⋆_i = (Δi / ||w⋆_A||^2) w⋆_A, where Δi = η⋆ − w⋆⊤xi (competitive margin).
    • Minimal cost: c⋆_i = Δi^2 / (2 ||w⋆_A||^2).
  • If w⋆_A = 0, recourse is structurally infeasible (classifier insensitive to actionable coordinates).
• Dynamic closed‑loop model (recourse as strategic interaction)
  • Discrete time Xt → designer solves CVaR problem → obtains (w⋆t, η⋆_t) and actionable direction d_t = Π{JA}(w⋆_t) → rejected agents optimally choose effort along d_t → population updates Xt+1 = Φ(Xt).
  • Candidate effort modeled with bounded moves toward a ceiling g on an actionable coordinate and a cost combining quadratic term and a logarithmic barrier to enforce the ceiling:
    • Ct_i(γ) = (k_i/2) γ^2 − θ_i log(Δ^t_i − γ), with Δ^t_i remaining gap; first‑order condition yields unique γ⋆ (closed form root).
  • Update recursion (componentwise): x_{t+1,i} = x_{t,i} + 1{i∈R_t} γ^t_{i⋆} ˜d_t, where ˜d_t is normalized actionable direction and R_t are rejected indices.
• Equilibria and classification of fixed points
  • Recourse equilibrium X⋆ satisfies Φ(X⋆) = X⋆. Characterization: for every rejected i, γ_{i,X⋆} d⋆ = 0.
  • Two equilibrium regimes:
    • Structural equilibrium: d⋆ = 0 (actionable projection vanishes) and some remain rejected → selection driven purely by immutable features.
    • Effort‑suppressed equilibrium: d⋆ ≠ 0 but γ_{i,X⋆} = 0 for all rejected i (effort is not profitable due to costs/barriers).
  • Social stratification defined by persistent immutable gaps: if d⋆ = 0 and R⋆ ≠ ∅, group separation remains supported by immutable features and cannot be contested by recourse.
• Mechanism for amplification of disparities
  • The top group sets both threshold η and direction d; rejected agents must follow a narrow, top‑driven direction.
  • Since cost grows (quadratically and via barrier), those farther from threshold face disproportionate costs; simultaneous adjustments shift the benchmark and can widen intergroup gaps rather than close them.

Data & Methods

• Nature of the work: theoretical, mathematical modeling and analysis (no empirical dataset).
• Key mathematical tools and assumptions:
  • Linear scoring rule and selection of top ρ fraction; representation via an “upper‑tail” CVaR (CVaRupρ).
  • Quadratic regularization of scoring vector w (parameter λ) to prevent unbounded scaling.
  • Duality and capped simplex characterization of α weights (0 ≤ αi ≤ 1/(ρn), Σαi = 1), leading to selection as centroid maximization.
  • Partition of features into actionable JA and immutable JN; projection Π_{JA}(·) used to define actionable direction.
  • Closed‑form minimal recourse under quadratic cost when movement allowed in a subspace.
  • Dynamical system: discrete time closed‑loop recursion Xt+1 = Φ(Xt) combining designer optimization and candidate best‑responses.
  • Candidate effort cost modeled as sum of quadratic term and logarithmic barrier (parameters k_i, θ_i, valuation β_i); first‑order optimality yields explicit update γ⋆ (root of quadratic).
  • Fixed‑point analysis to define recourse equilibria and classify regimes (structural vs effort‑suppressed).
• Key assumptions/limitations:
  • Linear classifier (scoring direction), selection by upper quantile.
  • Deterministic updates and full information of designer’s w (direction is communicated).
  • Feature space compactness, distinct scores for some derivations, and specific functional forms (quadratic and log barrier).
  • Homogeneous selection fraction ρ and population size n are exogenously fixed.
  • No stochasticity, no endogenous designer adaptation across rounds beyond computing w from current X.

Implications for AI Economics

• Performativity and endogenous selection
  • Deploying recourse recommendations in competitive contexts changes the data-generating distribution: performative feedback can shift thresholds and scoring directions endogenously.
  • Policy or model evaluation that ignores strategic, simultaneous responses under competition will mispredict long‑run outcomes.
• Inequality amplification and persistent stratification
  • When selection depends on immutable dimensions (or when actionable projection vanishes), recourse cannot remedy rejections — leading to locked‑in advantages.
  • Even when actionable directions exist, simultaneous pursuit can raise the benchmark so that absolute gains translate to no relative improvement — resembling arms‑race dynamics and “involution.”
• Design trade‑offs and governance levers
  • Actionability strength (||w_A||) is a diagnostic: low actionable weight makes recourse expensive/ineffective. Designers choosing w matter for permeability and fairness.
  • Regularization (λ), choice of ρ (selection tightness), and which coordinates are actionable versus immutable are powerful levers that change equilibrium outcomes.
  • Transparency about actionable directions can be double‑edged: it helps individuals but can coordinate investments that raise thresholds.
• Welfare and economic efficiency
  • Socially unproductive competition: if w_A aligns with dimensions that are costly to improve but not socially valuable, system incentivizes wasteful effort.
  • Heterogeneous costs/valuations (k_i, θ_i, β_i) imply unequal capacities to respond, so recourse can exacerbate preexisting inequality.
• Policy interventions suggested by the analysis
  • Ensure actionable dimensions reflect socially valuable skills (align w_A with desirable outcomes).
  • Introduce fairness constraints or diversity quotas to prevent selection purely on immutable axes.
  • Limit the informational signal about the precise improvement direction or randomize selection when competition is wasteful.
  • Subsidize recourse for disadvantaged groups, reduce barriers (lower θ_i), or directly expand capacity (increase ρ) to reduce destructive arms‑race effects.
  • Design classifier/regulator objectives to internalize social costs of competitive recourse (penalize over‑investment directions, or optimize steady‑state social welfare rather than immediate CVaR).
• Directions for empirical work and further research
  • Extend to stochastic dynamics, continuous time, and designer learning/adaptation.
  • Empirical calibration and validation: measure whether real competitive systems (admissions, hiring platforms) display predicted benchmark drift and stratification.
  • Study alternative scoring rules (nonlinear, causal) and institutional mechanisms (multi‑dimensional selection, lotteries, subsidies).
  • Explore strategic concealment or manipulation of recourse signals by designers and information design as a policy instrument.

Summary takeaway: Allowing actionable recourse in isolation—without considering competition, capacity constraints, and endogenous adjustments—can unintentionally harden inequality and lock in advantages. Designers and policymakers need to account for strategic, population‑level dynamics when deciding what recourse to provide, which features to make actionable, and how to regularize selection rules.