Improving algorithmic ranking can worsen distributional fairness under scarcity: as resources become scarcer, modest gains in model fidelity can sharply magnify selection gaps between groups, a pattern the authors derive analytically and observe in simulations and two public-sector datasets.

Main Finding

The paper identifies and formalizes the "Accuracy Trap": in resource-scarce allocation settings, incremental improvements in an algorithm’s rank-discrimination fidelity can exponentially amplify between-group disparities. The authors derive the scaling law D(t, ρ) ∝ exp(t · ρ · Δ), where D is the ratio of selection probabilities between an advantaged and disadvantaged group, t is the scarcity-induced selection threshold (moves into the extreme tail as supply shrinks), ρ ∈ [0,1] is rank-discrimination fidelity, and Δ is the structural gap in latent risk between groups. Under scarcity (large t), scarcity and accuracy interact multiplicatively, so modest increases in fidelity can produce very large relative disparities.

Key Points

• Accuracy Trap mechanism: noisy/imprecise ranking introduces randomization around the threshold that acts as an unintended equity buffer. As fidelity (ρ) increases, this buffer collapses and selection concentrates on the group with higher latent scores, increasing disparity.
• Main analytic result (Gaussian regime): ln D(t, ρ) = tρΔ + O(1) as t → ∞, so D ∝ exp(tρΔ). Equivalent power-law form holds in log-normal/heavy-tailed regimes: D(T, ρ) ∝ T^(ρΔ).
• Sensitivity: ∂D/∂ρ ≈ tΔ · D, so the effect of improving fidelity scales with scarcity (t) and the structural gap (Δ).
• Debiasing limits: upstream debiasing and calibration may reduce Δ but cannot eliminate the multiplicative amplification from accuracy × scarcity; debiasing sets Δ but does not remove the trap.
• Discretion/noise trade-off: administrative discretion or algorithmic noise can have a structural equity-preserving effect (by randomizing selection near the cutoff), separate from case-level harms of discretionary bias.
• The phenomenon is distributionally robust: demonstrated in Gaussian and log-normal regimes.

Data & Methods

• Formal model:
  • Two groups A and B with latent risk Z_A ∼ N(μ_A,1), Z_B ∼ N(μ_B,1). Structural gap Δ = μ_A − μ_B > 0.
  • Observed score: Y = ρ Z + sqrt(1 − ρ^2) ε, ε ⟂ Z, giving Y_g ∼ N(ρ μ_g, 1).
  • Selection uses a threshold t corresponding to the top σ fraction (σ → 0 → t → ∞).
  • Relative disparity D(t,ρ) = P(Y_A > t) / P(Y_B > t).
• Analytic derivations:
  • Mill’s ratio yields the asymptotic exponential scaling in the Gaussian tail.
  • Log-normal (heavy-tailed) derivation gives a power-law version: D(T, ρ) ∝ T^(ρΔ).
• Empirical validation:
  • Monte Carlo simulation: 20,000 individuals (10k per group), μ_A = 0.8, μ_B = 0.3, ρ varied 0.2–1, 25 iterations per (σ,ρ).
  • Child welfare case study (Children’s Aid Society Toronto): N = 583 families, n = 37,201 case notes; family-level risk R_llm derived from an LLM classifier (Meta Llama-3.1-8B) → mean risk Inner Toronto = 0.443, Outer Toronto = 0.385 ⇒ Δ ≈ 0.058. Post-hoc noise injection applied: Y = ρ R_llm + sqrt(1−ρ^2) ε; bootstrap (1,000 resamples) to estimate ln D.
  • Breast cancer (SEER) study: gradient-boosting model predicting 5-year breast-cancer-specific mortality; groups Non-Hispanic Black (n = 16,820) vs Non-Hispanic White (n = 118,662); mean risks 0.418 vs 0.325 ⇒ Δ ≈ 0.093. Post-hoc noise injection and 25 simulations per (σ,ρ).
• Key empirical patterns:
  • In all three settings (simulation, child welfare, SEER), ln D increases monotonically with scarcity (1−σ). Increasing ρ steepens that curve, i.e., higher fidelity creates larger disparities in high-scarcity regimes.
• Robustness/limitations noted by authors:
  • Structural gap Δ is modeled as exogenous (but represents residual structural differences after upstream interventions).
  • Noise injection is a post-hoc probe of fidelity; it is not a deployment of a different algorithm.
  • Smaller sample sizes in tails (esp. child welfare) add variability in empirical estimates.

Implications for AI Economics

• Rethink benefits of accuracy improvements in scarce settings: standard valuation of predictive improvements via accuracy gains can be incomplete or misleading when allocation is rationed. Investments that raise ρ can raise aggregate efficiency but may produce large, predictable redistributive effects that worsen relative outcomes for disadvantaged groups.
• Regulatory and policy design:
  • Fairness interventions that only target model-level metrics (debiasing/calibration) will not by themselves prevent large allocation inequalities under scarcity. Policies must consider supply constraints and allocation rules (thresholds, quotas, ties, randomization).
  • Consider explicit institutional interventions: increase supply (expand capacity), change allocation objectives (e.g., equity-weighted allocation, affirmative reserves, or targeted subsidies), or introduce randomized tie-breaking near the cutoff to preserve an equity buffer.
  • Evaluate algorithms by allocation outcomes, not only by standard predictive fairness metrics. Regulatory impact assessments should simulate tail selection under realistic scarcity and fidelity scenarios.
• Cost–benefit and welfare analysis:
  • Economic appraisal of predictive models should internalize distributional externalities that arise when higher accuracy concentrates benefits (or burdens) across groups. Social welfare calculus must include how accuracy interacts with scarcity and Δ.
  • The marginal social value of reducing Δ (through upstream social policy or debiasing) may be large because it reduces the exponent in the trap; but reducing Δ may be costly relative to reallocating supply or changing allocation rules.
• Organizational design and discretion:
  • Administrative noise/discretion can function as a structural equity-preserving mechanism. Policymakers must balance the case-level harms of discretionary bias against its aggregate effect in maintaining nonzero selection probabilities for disadvantaged groups.
  • Where feasible, controlled randomization or calibrated relaxation of thresholds may be preferable to maximal precision that mechanically amplifies disparities.
• Modeling and market implications:
  • For markets or platforms that ration scarce goods (organ transplants, specialized treatments, public housing, etc.), designers and regulators should model tail behavior and consider multi-dimensional intervention (accuracy, allocation policy, capacity).
  • Firms and public agencies investing in better predictive models should anticipate distributional consequences—these may alter political economy feedbacks, demand for services, and regulatory responses.
• Research directions for AI economics:
  • Extending analysis to multi-group settings, dynamic allocation with feedback, endogenous Δ (where algorithmic allocation affects future group positions), strategic behavior and gaming, and optimal policy design (quota vs randomization vs targeted capacity expansion).
  • Empirical measurement strategies for Δ and t in operational systems to forecast the strength of the Accuracy Trap before deployment.

Overall, the paper reframes fairness/accuracy trade-offs in allocation problems: under scarcity, accuracy is not neutral with respect to distribution — it multiplies structural differences into potentially exponential disparities. Policy responses must therefore combine model-level work with institutional redesign and supply-side solutions.