Class-conditional conformal prediction plugs a dangerous blind spot in imbalanced, high-stakes decisions — minority-class coverage jumps by ~61.7 percentage points versus marginal CP — and, when paired with cost-aware human deferral, cuts expected decision costs and identifies when review pays off.

Main Finding

Mondrian (class-conditional) conformal prediction plus a cost-controlled abstention rule restores valid coverage for rare, high-cost classes on severely imbalanced tabular problems and—when combined with a human-in-the-loop deferral policy—reduces expected decision costs. Marginal (global) conformal prediction can catastrophically under-cover minority classes (sometimes <1% coverage); Mondrian CP fixes that (average minority-coverage gain 61.7 percentage points, p < 10^−80). The paper also derives dataset-specific break-even review-cost thresholds under which deferral to human experts is economically preferable.

Key Points

• Scope and scale
  • Benchmark over 15 real-world imbalanced tabular datasets from multiple domains (finance, healthcare, industrial safety, fraud, environment, etc.).
  • Extensive experimental grid: 7 base classifiers, 3 probability-calibration techniques, 10 random seeds → 3,150 experimental runs.
• Failure of marginal CP
  • Marginal conformal predictors guarantee only global (marginal) coverage; on imbalanced data they can over-cover the majority class while severely under-covering the minority (examples down to <1% minority coverage).
• Mondrian (class-conditional) CP
  • Computes per-class non-conformity quantiles and enforces coverage 1−α within each class, restoring minority coverage across datasets and models.
  • Non-conformity score used: S(X,y) = 1 − P̂(Y = y | X).
• Operational rule and abstention
  • Prediction sets are {0}, {1}, {0,1}, or ∅. The operational decision rule: execute automated label when the set is a singleton; abstain (DEFER to human) when set is multi-label or empty.
• Cost-sensitive framework
  • Asymmetric cost matrix with C_FN (false negative), C_FP (false positive), and C_rev (human review cost); human reviewer model includes error rate ε_hum.
  • Authors show cost-controlled abstention (deferring ambiguous cases identified by Mondrian CP) reduces expected cost compared to fixed-threshold classifiers, confidence-based rejectors, and risk-controlled rejectors under realistic review budgets.
• Quantitative results
  • Mondrian CP yields an average +61.7 percentage-point improvement in minority coverage vs marginal CP (highly significant).
  • The benchmark identifies dataset-specific break-even human-review cost C*_rev below which deferral reduces expected cost.

Data & Methods

• Datasets
  • 15 OpenML tabular datasets spanning finance (default, credit, fraud), healthcare (mammography, sick, diabetes), industrial (aps_failure, seismic_bumps), environment (ozone, oil_spill), and others. Sample sizes range from ~937 to ~284k; minority-class prevalence ranges from 0.17% (fraud) up to ~30% (credit_g), producing extreme imbalance ratios (e.g., 580:1 for fraud).
• Models & calibration
  • Evaluated 7 base classifiers (representative families: gradient-boosted trees, linear models, probabilistic models — the paper lists HistGradientBoosting explicitly) and 3 probability-calibration methods (the paper references standard techniques such as Platt scaling and isotonic regression among the calibrators used).
• Conformal methods
  • Marginal CP: single calibration quantile q_marg computed over all calibration samples; prediction set Ĉ_marg(X) = { y : S(X,y) ≤ q_marg } with marginal guarantee P(Y ∈ Ĉ_marg(X)) ≥ 1 − α.
  • Mondrian CP: separate calibration quantiles q_y for each class y computed on class-specific calibration subsets; prediction set Ĉ_Mondrian(X) = { y : S(X,y) ≤ q_y } with class-conditional guarantee P(Y ∈ Ĉ_Mondrian(X) | Y = y) ≥ 1 − α for every y.
  • Non-conformity score used: S(X,y) = 1 − P̂(Y = y | X). APS / RAPS variants are noted as set-size control techniques but the principal comparison is marginal vs Mondrian CP.
• Cost-controlled abstention
  • Decision rule: if |Ĉ(X)| = 1 execute that label; otherwise defer (human review). Empty sets (∅) and multi-label sets ({0,1}) are both treated as deferral triggers (capturing atypical or ambiguous cases).
  • Expected per-instance cost for the conformal + abstain system: E[L_conf] = (1 − π1) · C_FP · FPR_conf + π1 · C_FN · FNR_conf + r_abs · C_rev where r_abs = P(|Ĉ(X)| ≠ 1).
  • Break-even human-review cost (below which deferral is beneficial versus point classifier at Bayes threshold τ = C_FP / (C_FN + C_FP)): C_rev = [ (1 − π1) C_FP (FPR_τ − FPR_conf) + π1 C_FN (FNR_τ − FNR_conf) ] / r_abs (Equation cited and derived in the paper.)
• Evaluation metrics
  • Coverage (global and per-class), prediction-set sizes, abstention rate, automated FPR/FNR on singleton predictions, and expected decision cost accounting for C_FP, C_FN, C_rev, and human error ε_hum. Statistical significance reported (e.g., p < 10^−80 for coverage gains).
• Baselines compared
  • Standard thresholded point classifiers at Bayes-optimal cost threshold, confidence-based rejectors, and risk-controlled rejectors (conformal risk control style).

Implications for AI Economics

• Economic value of class-conditional uncertainty quantification
  • Distribution-free, class-conditional coverage guarantees protect rare but costly outcomes that marginal methods miss; this reduces tail losses that dominate expected cost in high-stakes settings (fraud detection, clinical screening, industrial failure).
• Decision to invest in human review
  • The derived break-even C_rev gives an actionable economic rule: measure r_abs, FPR_conf, FNR_conf for your system and compute C_rev to decide whether human review capacity is justified. If actual per-case review cost < C*_rev, deferral yields net cost savings.
• Resource allocation and pricing
  • Organizations can use the benchmarked relationships (abstention rate vs coverage vs expected cost) to size human-in-the-loop staffing, set pricing for manual review, or determine acceptable review throughput given expected cost reductions.
• Regulatory and fairness considerations
  • Restoring minority-class coverage reduces disparate under-protection of rare groups or events; regulators and auditors can prefer class-conditional conformal guarantees when fairness and protection of vulnerable subgroups matter.
• Operational caveats and sensitivity
  • Results depend on realistic human-review models (ε_hum, variability in C_rev). The break-even threshold is dataset- and model-specific; practitioners must estimate these terms empirically in their operational environment.
  • Trade-offs remain between prediction-set size (more abstentions increase human burden) and automated error rates; cost-sensitive tuning of the conformity level α and choice of calibration set are practical levers.
• Strategic deployment
  • In markets or applications where false negatives are extremely costly and the minority class is rare, deploying Mondrian CP with targeted deferral can produce large ROI compared with attempting to push more automation via threshold tuning alone.
  • The distribution-free nature of CP reduces dependence on well-calibrated probability estimates, lowering model-risk premiums in decision pipelines and potentially simplifying compliance/assurance for high-stakes deployments.

If you want, I can (a) extract the exact experimental results per dataset (coverage, abstention rate, C*_rev) into a compact table, or (b) produce a short checklist for deploying Mondrian CP + cost-controlled abstention in an operational risk-management workflow. Which would be most useful?