Machine learning improves which shipments to inspect only when delay severity is learnable and calibrated: in three real supply-chain datasets ML beats severity-only ranking but only outperforms simple value-sorting in the DataCo logistics set (+10.1pp at a 10% review budget), while it underperforms value sorting in SCMS (-5.5pp) and Olist (-4.9pp); the paper advises auditing R^2 and calibration under leakage-controlled rolling-origin evaluation before deployment.

Main Finding

Machine-learned ranking by predicted delay severity times known shipment value (M1) outperforms severity-only ranking but does not reliably beat a no-model baseline that simply inspects the highest-value shipments first (VALUE_ONLY). Across three public supply‑chain datasets, M1 clears a demanding operational gate (paired 95% bootstrap CI for Capture@k(M1) − Capture@k(VALUE_ONLY) strictly > 0 at k = 10%) only in the DataCo logistics dataset, not in SCMS (procurement) or Olist (e‑commerce). Nested-CV cost‑sensitive retraining (M2b_tuned) provides no stable improvement over M1.

Key Points

• Operational objective: with a fixed review budget k (e.g., top 10% of shipments), maximize captured exposure defined as exposure = delay severity (days) × known value.
• Rankers compared (examples): VALUE_ONLY (v), SEVERITY_ONLY (predicted delay d̂), PROB_ONLY (P(late)), BINARY×VALUE (P(late)×v), M1 (d̂×v), M2b_tuned (cost‑weighted retrained d̂×v).
• Evaluation protocol (conservative / leakage-controlled):
  • Rolling‑origin temporal folds (5 outer folds).
  • Training-only preprocessing and hyperparameter sharing across rankers.
  • Paired 1,000-sample bootstrap for confidence intervals on Capture@k differences.
  • Gate criterion: M1 must have strictly positive paired 95% CI versus VALUE_ONLY.
• Main numeric result at k = 10% (M1 − VALUE_ONLY):
  • DataCo: +10.1 percentage points (pp), 95% CI ≈ [9.67, 10.42] pp — Gate cleared.
  • SCMS: −5.5 pp, 95% CI ≈ [−11.8, 1.7] pp — Gate not cleared.
  • Olist: −4.9 pp, 95% CI ≈ [−7.3, −2.9] pp — Gate not cleared.
• Mechanism: success correlates with severity “learnability” and calibration.
  • DataCo: Spearman ρ ≈ 0.386, R² ≈ 0.270, calibration bias ≈ +0.01 days → meaningful ordering + good calibration.
  • SCMS & Olist: ρ small (0.149, 0.075), R² ≈ −0.02, negative calibration bias → weak signal; VALUE_ONLY often superior.
• Cost‑sensitive retraining (nested CV weight tuning) did not reliably improve Capture@10% and was sometimes worse.
• Practical effect sizes: on DataCo, switching from VALUE_ONLY to M1 raises captured exposure at 10% from ~22% to ~32% (≈45% relative increase). On SCMS and Olist M1 captures less exposure than VALUE_ONLY.

Data & Methods

• Datasets (public):
  • SCMS (procurement): 10,324 rows, late rate ≈ 11.5%.
  • DataCo (logistics): 180,519 rows, late rate ≈ 57%.
  • Olist (Brazilian e‑commerce): 96,470 rows, late rate ≈ 8.1%.
• Target and exposure:
  • Delay severity d_i = max(actual − scheduled, 0) in days.
  • Exposure e_i = d_i × v_i (multiplicative by design).
• Modeling:
  • Severity regressor: XGBoost (same hyperparameters across M0, M1, M2b_tuned).
  • Binary classifier for P(late) using the same leakage-controlled features.
  • M2b_tuned applies cost‑proportionate sample weights w_i(α) = 1 + α v_i normalized, α chosen by nested rolling‑origin CV (α ∈ {0,1,2,5,10}).
  • Target transformation (raw or log(1+d)) selected by pooled RMSE within rolling folds.
• Evaluation:
  • Capture@k = fraction of total realized exposure captured by top k% of ranked shipments; Lift@10% = Capture@10% / 0.10.
  • Leakage control: no post‑delivery fields in features; fold‑local preprocessing.
  • Uncertainty: paired bootstrap (1,000 resamples) to generate 95% percentile intervals.
• Robustness & limitations noted by authors:
  • Results conditional on multiplicative exposure definition (d×v).
  • Only three heterogeneous datasets — not a universal threshold.
  • Retrospective data and available pre‑delivery signal limitations; drift requires ongoing re‑audit.

Implications for AI Economics

• Decision-focused evaluation matters: predictive accuracy (AUC/RMSE) can be a poor proxy for operational value. Economists and managers should evaluate ML via the downstream constrained decision (fixed budget Capture@k) and against simple, costless baselines.
• Opportunity cost of deployment: deploying ML can improve or worsen realized exposure capture relative to a free VALUE_ONLY rule. The expected net benefit must account for:
  • Gain in captured exposure (e.g., DataCo ≈ +10 pp at 10% budget).
  • Implementation, monitoring, and operational costs of ML pipelines.
  • Risk of model degradation under drift — requires ongoing audits.
• Diagnostic checklist (operational recommendation):
• Define exposure and fix intervention budget k.
• Compute VALUE_ONLY Capture@k as incumbent baseline.
• Audit severity rank informativeness (Spearman ρ, R²) and calibration on rolling‑origin folds.
• Adopt M1 only if paired 95% CI for (M1 − VALUE_ONLY) is strictly positive at k.
• Re-audit under drift; do not assume cost‑sensitive retraining will help without nested‑CV evidence.
• Generalization to other economic applications: the “value‑only gate” applies whenever exposure factorizes into known magnitude × uncertain risk (fraud screening, preventive maintenance, collections). In those contexts, simple sorting by known magnitude is a low‑cost baseline that ML must outperform under temporally valid evaluation.
• Research agenda for AI economics:
  • Characterize thresholds of severity learnability (ρ, R²) and calibration that predict gate clearance across more domains.
  • Study alternative exposure specifications (non‑multiplicative costs, threshold/contractual penalties) and richer cost‑sensitive learners (quantile/distributional models, asymmetric loss functions).
  • Quantify full cost–benefit including operationalization and monitoring costs to support deployment decisions.

Concise takeaway: retain VALUE_ONLY as a permanent benchmark, audit learnability and calibration with temporally valid evaluation, and deploy ML prioritization only when it demonstrably and robustly increases exposure capture under the real constrained decision.