Better-looking forecasts do not always cut costs: a decision-focused simulator for automotive spare parts finds that gains in MAE/RMSE often fail to improve inventory cost or service, and models with similar accuracy can yield very different operational outcomes.

Main Finding

Improvements in conventional forecasting accuracy (MAE, RMSE, R2, IAE) do not reliably translate into better operational outcomes (total inventory cost, fill rate) for intermittent spare‑parts demand. The authors present an open simulation framework that closes the loop between demand forecasting and inventory decision-making and demonstrate that models with similar statistical errors can produce markedly different cost–service trade-offs. In the experiments, a classical intermittent-demand method (Croston) yielded the lowest total inventory cost despite being worse on standard accuracy metrics than modern ML models.

Key Points

• Decision-centric evaluation: The paper argues forecasts should be judged by their downstream operational impact (KPIs such as total cost and service level) rather than only by pointwise statistical accuracy.
• Software framework: A modular, closed-loop simulator linking (i) synthetic spare‑parts demand generation, (ii) a flexible forecasting module (any model), and (iii) an inventory cost simulator driven by chosen inventory control policies.
• Synthetic demand focused on real-world spare‑parts features: intermittent/lumpy demand, seasonality, heterogeneous failure dynamics, and concept drift (sudden and slow changes in fleet size).
• Inventory DES: a discrete-event cost simulator computes holding, ordering, rush, transportation, and badwill costs, using a standard periodic-review safety-stock policy (normal-demand assumptions).
• Empirical result: Across 48 three-year synthetic scenarios, Croston’s method produced the lowest aggregate inventory cost (≈7.4×10^5) while ML models like XGBoost, despite best MAE (≈0.43), produced higher costs (XGBoost ≈8.2×10^5). Random Forest was competitive on both accuracy and cost (≈7.5×10^5).
• Demand irregularity: Most generated series were highly intermittent (high ADI) and variable (CV^2), and R^2 values were often negative—indicating strong prediction difficulty.
• Practical implication: Model choice should account for asymmetric and context-dependent cost consequences of forecast errors (e.g., underforecast vs overforecast).

Data & Methods

• Framework components:
  • Synthetic demand generator (Discrete Time Simulation, daily timestep) with hierarchical entities: Dealer → Truck → Part.
  • Forecasting module: accepts arbitrary models; experiments used XGBoost, Random Forest, SVR, ARIMA, Croston, SBA, and TSB. Models were hyperparameter‑tuned to minimize MAE and trained on the first 2 years to forecast the 3rd year.
  • Inventory/cost simulator: Discrete-Event Simulation (DES) implementing a standard inventory policy (safety stock, reorder points) and cost accounting (holding, order, rush, transport, badwill).
• Demand generator details:
  • Failure modeled with survival/hazard functions: Exponential, Weibull, Log‑logistic, Gompertz. Median lifetime tm used as interpretable parameter.
  • Seasonality modeled with radial basis functions over day-of-year; parts and regions (northern/southern) can have different seasonal profiles.
  • Concept drift: sudden (abrupt dealer fleet increase) and slow (gradual growth).
  • Outputs: daily demand time series per (dealer_id, part_type).
• Experimental design:
  • 48 scenarios: 4 drift conditions × 12 parameter configurations (4 ranges of trucks per dealer × 3 ranges of median lifetime).
  • Simulation horizon: 3 years (T = 1,095 days). Forecasting horizon: year 3.
  • Forecast evaluation metrics: MAE, RMSE, R^2, IAE (intermittent demand accuracy).
  • Operational KPIs: total inventory-management cost and fill rate (service level).
  • Simplifying assumptions in cost sim: equal unit cost/weight/lead time across parts, single unlimited-capacity supplier, no batching/partial shipments, fixed inventory-policy parameters across parts.
• Reproducibility: open-source code and experimental results available (GitHub link in paper).

Implications for AI Economics

• For operational problems (inventory, spare parts), statistical forecast accuracy is an insufficient objective. Economic evaluation must be decision-aware: choose models by their impact on relevant KPIs (cost, service, environmental externalities).
• Practical model selection: prefer forecasting approaches that minimize operational cost under the inventory policy and cost structure in use. Simpler, specialized methods for intermittent demand (e.g., Croston variants) can outperform complex ML models on operational KPIs despite worse pointwise accuracy.
• Model development: incorporate asymmetric loss and decision costs into training (cost‑sensitive or end‑to‑end decision-aware learning) rather than optimize symmetric pointwise errors alone.
• Policy and managerial guidance:
  • Use closed-loop simulation to evaluate the marginal value of forecast improvements before costly data-collection or model-deployment investments.
  • Align forecast outputs (e.g., predictive distributions, bias characteristics) with inventory rules (safety stock formulas, reorder policies) — e.g., biased forecasts may be compensated by safety-stock calibration.
  • For sustainability goals, decision-centric forecasting can quantify how improved predictions reduce rush shipments and overproduction.
• Research directions for AI economics:
  • Extend closed-loop evaluation to multi-echelon inventory, stochastic lead times, and endogenous supplier capacity constraints.
  • Study the economic value of alternative information sources (telematics, maintenance logs) within the same decision-centric framework.
  • Develop learning algorithms trained with inventory-cost (or other downstream-KPI) objectives and evaluate trade-offs between statistical accuracy and operational value.
• Limitations to note:
  • Results are based on synthetic demand and simplifying inventory/cost assumptions (identical part costs/lead times, normal-demand safety-stock formulas). Real-world heterogeneity and multi-location interactions may change relative model performance.
  • The inventory policy was held constant; jointly optimizing policy and forecast model could yield different conclusions.

Overall, the paper operationalizes the link between forecasting and inventory decisions, providing an open simulation environment for researchers and practitioners to evaluate forecasting methods by the economic outcomes that actually matter.