Explainability rules and box‑ticking human‑review requirements can make automated lending legally defensible but socially exclusionary, concentrating reliance on safe proxies and defensive rejection of marginal borrowers. Regulators should move from artifact‑centred checks to outcome‑facing supervision, enforceable contestability rights, and standards that ensure human intervention is substantive rather than ceremonial.

Main Finding

Explainability and human-overwatch requirements in automated credit scoring can produce “legible injustice”: paper‑reviewable but systematically exclusionary lending. When compliance is judged by auditable artefacts rather than actual outcomes, procedural transparency and oversight can channel models and operations toward legally comfortable yet socially stratified practices. To prevent this, regulation must shift from artifact‑centred checks to outcome‑facing supervision, meaningful intervention standards, and an enforceable right of contestability.

Key Points

• Problem diagnosis

  • Governance has focused on explainability and human oversight as remedies for automation risks, implicitly assuming procedural transparency secures substantive fairness.
  • Monitoring compliance via auditable artefacts (explanations, documented human reviews) can produce decisions that are reviewable on paper but exclusionary in effect.

• Three incentive‑driven mechanisms that produce legible injustice

• Proxy concentration: Pressure for interpretability pushes models toward familiar, legally safe variables that serve as proxies for social disadvantage, concentrating reliance on stratifying predictors.
• Defensive rationing: Accuracy–interpretability trade‑offs combined with asymmetric liability and supervisory exposure incentivize tighter thresholds or rejection of marginal/non‑standard applicants to avoid perceived legal or reputational risk.

• Review theatre: Human‑in‑the‑loop requirements ossify into ritualized workflows that create documentary evidence of review without preserving genuine discretion or remedial action.

• Comparative legal reconstruction

  • Uses the EU model as focal point and contrasts with US fair‑lending law and the UK Consumer Duty to show how explainability/intervention duties play out differently across borrower, governance, and supervisory audiences.

• Proposed remedies

  • A six‑element standard for meaningful intervention: competence, authority, independence, informational sufficiency, evidential engagement, and auditable records — distinguishing substantive review from perfunctory sign‑offs.
  • A right of contestability: decision‑salient notices, low‑friction challenge channels, and credible escalation routes so borrowers can meaningfully dispute automated decisions.
  • Outcome‑facing supervision instruments: distributional dashboards, diagnostics of proxy usage, and metrics that evaluate the effectiveness of interventions rather than just the presence of explanations.

Data & Methods

• Methodological approach: legal and regulatory analysis combined with incentive‑theory reasoning.
  • Reconstructs statutory and regulatory duties (EU digital/consumer/credit frameworks, US fair‑lending regimes, UK Consumer Duty) and maps how explainability and intervention obligations are interpreted by different audiences (borrower, firm governance, supervisors).
  • Uses conceptual models to identify how compliance incentives produce three mechanisms of legible injustice.
  • Normative design: formulates the six‑element standard, contestability rights, and supervision metrics as policy instruments.
• Evidence: primarily doctrinal analysis and theoretical argumentation rather than original empirical testing; draws on institutional incentives and known regulatory practices to motivate prescriptions.

Implications for AI Economics

• Model selection and feature engineering

  • Regulatory emphasis on interpretability can bias model choices toward simpler, more legally legible predictors, potentially increasing reliance on proxies that correlate with disadvantage—altering the trade‑off between predictive performance and distributional equity.
  • Economists modeling firm behavior should incorporate regulatory compliance costs and liability asymmetries as selection forces shaping algorithm design.

• Credit allocation and financial inclusion

  • Defensive rationing can reduce credit supply to marginal borrowers even when predictive capacity exists, exacerbating exclusion and altering credit market segmentation.
  • Outcome‑facing metrics (distributional dashboards, intervention‑effectiveness) can help detect and reverse exclusionary equilibria, but require regulators to measure market‑level outcomes rather than checklists.

• Regulatory design and enforcement economics

  • Procedural transparency that produces auditable artefacts may create perverse compliance equilibria (review theatre); enforcement should instead target outcomes and the effectiveness of human interventions.
  • The six‑element standard and contestability rights change the enforcement technology: regulators and courts will need to assess competence, independence, and evidential engagement, raising the administrative cost but improving corrective power.

• Measurement and evaluation

  • New metrics are needed: distributional impact indicators, proxy‑concentration diagnostics, and intervention‑effectiveness measures. These facilitate cost‑benefit and welfare analyses of automated lending systems and regulatory interventions.
  • Empirical research should quantify how interpretability requirements influence feature selection, rejection thresholds, and lending outcomes across demographic groups.

• Market structure and incentives

  • Firms facing liability asymmetry and supervisory focus on artifacts may centralize model development around compliance templates, reducing innovation and entrenching stratifying predictors.
  • Policy that ties transparency to corrective enforcement rather than legitimacy can realign firm incentives toward substantive fairness, potentially reshaping competitive dynamics (e.g., firms that credibly demonstrate equitable outcomes may gain market advantage).

Overall, the paper argues that AI economics must account for how regulatory form (artifact‑centred vs outcome‑facing) interacts with firm incentives to shape both model behavior and real‑world distributional outcomes.