A single optimal permutation makes permutation-based feature importance faster, more stable and more accurate, while a new "Systemic Variable Importance" metric exposes how shocks propagate through correlated inputs — revealing hidden dependence on protected attributes in credit and household finance models.

Main Finding

Replacing the usual B independent random permutations per feature with a single, deterministic permutation (a rank or index cyclic shift by ⌊n/2⌋) yields a non‑random, faster, and more stable permutation‑based variable‑importance estimator that (under broad conditions) reduces estimator variance and often achieves lower MSE than the classical Breiman Monte‑Carlo permutation estimator. The paper also introduces a Systemic Variable Importance extension that explicitly models feature correlations for model stress‑testing and fairness auditing.

Key Points

• Three notions of variable importance are clarified: population VI, model‑class VI, and model‑instance (direct) VI. The paper focuses on model‑instance (direct) VI, where ignoring feature dependence is intentional.
• Classical permutation VI samples B random permutations per feature; this creates finite‑sample variance Var(g)/B and non‑determinism (seed dependence).
• Proposal: use one deterministic permutation that maximally perturbs ranks — specifically a cyclic rank shift by k = ⌊n/2⌋ (or an index shift approximation).
  • Proposition 1: the cyclic shift by ⌊n/2⌋ maximizes the minimum circular displacement of ranks (an optimality criterion).
  • Proposition 2: gives a component‑wise MSE dominance condition. The deterministic estimator ˜Ij has lower MSE than the B‑rep Monte Carlo estimator ˆIB,j iff its squared bias is less than the Monte‑Carlo squared bias plus σ^2_j/B; intuitively, ˜Ij wins when its excess bias is smaller than the Monte Carlo estimator’s standard error.
• Practical diagnostics: compare ||˜I − ˆI_B||^2 to ||ˆσ^2||^2 / B to decide whether the single‑permutation estimator is preferable under a given compute budget B.
• Algorithmic/design choices:
  • Deterministic permutation: rank‑shift (O(n log n)) is optimal; index‑shift (O(n)) is a fast suboptimal approximation that performs similarly in simulations.
  • Scoring metric: MAE (default), but experiments use MSE of prediction differences for comparability with classical baselines; for classification the change in negative Brier score (MSE of class probabilities) is used.
  • Final VI scores are normalized to sum to one.
• Empirical validation: nearly 200 scenarios including real household finance and credit‑risk case studies. Reported benefits include improved bias–variance tradeoffs and better recovery of ground‑truth VI in challenging regimes (small n, high p, low signal‑to‑noise).
• Systemic Variable Importance: an extension that accounts for input correlations to quantify how shocks/perturbations propagate through correlated features, useful for stress‑testing and revealing hidden reliance on protected attributes (fairness auditing).
• Computational gains: single deterministic permutation is at least an order of magnitude faster than classical 10‑repetition permutation importance and competitive with single random‑permutation scoring, with zero Monte‑Carlo variance (deterministic output).

Data & Methods

• Definition: Direct Variable Importance (DVI) = average change in model predictions after replacing feature x_k by its permuted version x'_k, measured with a metric d (MAE/MSE/RMSE of prediction differences), then normalized to sum to one.
• Deterministic permutation choices:
  • Rank shift by ⌊n/2⌋ (optimal by Proposition 1).
  • Index shift by ⌊n/2⌋ (computationally cheaper approximation).
• Theoretical analysis:
  • Formal variance/bias decompositions and MSE comparison between the deterministic estimator and Monte‑Carlo averaged estimator (Proposition 2 and Corollary for vector MSE).
  • Asymptotic remarks: single permutation can also dominate asymptotically if its bias decays sufficiently fast relative to Monte‑Carlo variance.
• Empirical setup (summarized from paper):
  • Nearly 200 simulation/real‑data scenarios, including correlated feature designs to stress interdependence.
  • Real‑world case studies in household finance and credit risk focusing on fairness and systemic dependence.
  • Baselines: classical Breiman‑style permutation importance (multiple random permutations) implemented in standard libraries (e.g. scikit‑learn).
  • Evaluation metrics: MSE of prediction differences (primary for comparability); MAE default also discussed; classification uses change in negative Brier score.
• Computational complexity:
  • Rank‑shift requires sorting (O(n log n)); index shift is O(n).
  • Single deterministic permutation avoids repeated model evaluations for multiple random permutations, yielding large runtime reductions.
• Practical check recommended by authors: compute the Monte‑Carlo variance estimates ˆσ^2_j and test whether ||˜I − ˆI_B||^2_2 < ||ˆσ^2||^2_2 / B to decide whether deterministic is likely better for that budget.

Implications for AI Economics

• Faster, deterministic VI lowers the operational cost of model explanation and auditing, enabling:
  • More frequent and cheaper audits of deployed economic/financial models (credit scoring, consumer finance, macroprudential models).
  • Reproducible importance scores for regulatory reporting and model governance (no seed‑driven variability).
• Improved stability in small‑sample, high‑dim, and low‑SNR regimes is especially relevant in economic applications where data are limited or noisy (e.g., firm‑level studies, niche credit products, microdata).
• Systemic Variable Importance offers a principled, computationally tractable way to:
  • Assess how shocks to one input feature propagate through correlated inputs—useful for stress‑testing credit portfolios, systemic‑risk assessment, and scenario analysis.
  • Detect hidden reliance on protected attributes (gender, race) via propagation through correlated proxies—important for fairness auditing and regulatory compliance.
• Practical governance: the deterministic method lowers the barrier to routine model instance VI reporting (easier to standardize across institutions), which could improve market transparency and comparability of model risk assessments.
• Caveats and limitations for economic use:
  • The direct VI intentionally ignores feature dependence; this is appropriate for measuring reliance in a fixed fitted model but can mislead if one wants conditional or causal importance (i.e., model‑class or population VI).
  • Deterministic VI can have bias relative to the Monte‑Carlo expectation; the paper gives a testable criterion for when the deterministic estimator is likely superior, so practitioners should compute the permutation variance diagnostic before wholesale adoption.
  • Further validation is needed across more model classes, macroeconomic forecasting contexts, and in live regulatory workflows; the paper’s favorable results are strong but not universal.
• Overall: the method makes robust, reproducible, and cheap model‑instance importance analysis more practical in economic and financial settings and provides a new tool (Systemic VI) tailored to stress‑testing and fairness concerns that hinge on correlated inputs.