Planning that enforces zero failures produces reliably failure-free schedules; reinforcement learning often accepts occasional failures to cut costs and struggles to reach zero-failure behavior even at high penalty values, with action masking improving but not closing the gap.

Main Finding

Planning (finite-horizon search with a hard zero-failure constraint) and reinforcement learning (DQN/PPO optimizing discounted expected cost) produce systematically different maintenance policies in multi-asset scheduling. Planning enforces zero-failure schedules (when feasible) and is insensitive to failure-penalty magnitude; RL treats failures as soft costs and trades preventive maintenance against tolerable failures, yielding lower cost under low-penalty regimes but persistent non-zero failures even when penalties are large. Lightweight fixes for RL (reward shaping, action masking) reduce but do not eliminate this reliability gap. The paper also delivers a controlled benchmark unifying environment, cost model, and evaluation for comparing approaches.

Key Points

• Objective formulation drives behavior:
  • Planner uses a hard constraint that forbids any failure in the planning horizon → zero-failure policies (if feasible), independent of failure penalty cf.
  • RL minimizes discounted expected cost → may accept occasional failures if maintenance cost and discounting make prevention suboptimal.
• Empirical pattern:
  • Under low cf, RL often outperforms planner in total cost by accepting rare failures.
  • Under high cf (safety-critical), planner still produces zero-failure schedules while RL policies often continue to incur some failures.
• Discounting and horizon mismatch:
  • Discount factor and finite lookahead create a structural bias in RL toward deferring maintenance when asset lifetimes exceed the effective discount horizon, increasing failure risk.
• Lightweight reliability mechanisms:
  • Reward shaping (amplifying failure penalty during training) had negligible effect on failure rates at evaluation.
  • Action masking (forbidding no-op when any asset falls below a threshold) reduced failures but substantially increased total cost, and did not fully close the gap to planning.
• Practical recommendation:
  • Use planning when strict zero-failure reliability is required or deployment horizons are short and dynamics are well modeled.
  • Use RL when occasional failures are acceptable, uncertainty favors learned policies, and long-run operational efficiency is prioritized.
• Benchmark contribution:
  • The authors release a unified protocol (environment, cost model, evaluation) for comparing planning and learning in maintenance, enabling reproducible controlled studies.

Data & Methods

• Environment / system:
  • Multi-bearing maintenance problem with N = 6 run-to-failure bearings operating in parallel and a single shared maintenance crew (at most one bearing serviced per time step).
  • Two post-maintenance dynamics: oracle (reset to original recorded run) and estimated (sampled from fitted log-normal lifetime distribution).
  • Observation mappings: continuous normalized remaining useful life (RUL) + failure flag, and a discrete four-state health categorization (healthy, degrading, critical, failed).
• Dataset:
  • Run-to-failure bearing dataset (Nectoux et al. 2012) with six trajectories; lifetimes: 515, 797, 871, 911, 1637, 2803 recording windows (10 s per window).
  • Episodes simulated with 8000 steps (longer than observed lifetimes to allow failures if no maintenance).
• Cost model:
  • Per-step operating cost per active bearing cop = 1.0.
  • Fixed maintenance action cost cm = 25.0.
  • Failure penalty cf varied across {100, 500, 1000, 5000, 10000} to span low to high risk regimes.
  • Total cost summed over episode: operating + maintenance + cf × newly failed bearings.
• Planning baseline:
  • Dijkstra / uniform-cost search over a finite-horizon H = 20 joint state space with a hard infinite cost assigned to any state containing a failure (i.e., zero-failure constraint). Receding-horizon execution: execute first action then replan.
  • Rule-based threshold policy also included as a heuristic baseline.
• RL baselines:
  • DQN (value-based) and PPO (policy-gradient), 2-layer MLPs (64 units), ReLU; observation = normalized RUL + failure flag for each bearing.
  • Training protocol: DQN 1500 episodes, PPO 1000 episodes, 3 seeds, discount γ = 0.99 (default), replay and PPO hyperparameters as standard.
• Reliability mechanisms tested for RL:
  • Reward shaping: amplify failure penalty during training only.
  • Action masking: forbid no-op when any non-failed bearing’s normalized remaining life < δ (δ = 0.25), applied during training and eval.
• Evaluation:
  • Compare total cost and failure counts across cf regimes, oracle vs estimated post-maintenance dynamics, and methods.

Implications for AI Economics

• Trade-off between reliability and cost is fundamentally shaped by objective specification:
  • When operators value strict reliability (large implicit or explicit penalty for failures), planning formulations with hard constraints better align incentives and guarantee zero-failure behavior (when feasible). This yields predictable operational costs that are insensitive to the numeric failure-penalty parameter.
  • When operators accept occasional failures to reduce maintenance spending (risk-tolerant regime), RL can produce lower expected operational costs by optimizing the trade-off under uncertainty.
• Economic decision-making must account for horizon and discounting effects:
  • Discounted RL may underweight very long-run failure risks, biasing toward deferred maintenance; correspondingly, choice of discount factor (or switching to average-reward / risk-sensitive formulations) materially affects economically optimal policies.
• Practical method selection depends on organizational objectives and constraints:
  • Use planning for short-term deployments, well-modeled dynamics, and strict safety/regulatory constraints.
  • Use RL for long-running operations with stochastic dynamics and where learning from data can capture complex interactions and yield cost efficiencies—provided acceptable failure rates.
• Policy design and governance:
  • Numeric penalties (cf) alone may not suffice to enforce reliability under learning-based methods. Organizations must decide whether to encode reliability via constraints (planning or CMDP) or accept probabilistic guarantees from RL.
  • Lightweight fixes (shaping, masking) have trade-offs: masking raises costs; shaping may have limited impact. For high-assurance contexts, invest in constrained optimization methods (CMDPs, shielding) or hybrid approaches.
• Economic value of benchmarks:
  • The paper’s unified benchmark enables apples-to-apples comparisons and can inform procurement and technology selection decisions by quantifying cost–reliability trade-offs across methods.
• Limitations that affect economic interpretation:
  • Small-scale dataset (six bearings) and planner horizon cap limit direct scalability claims; computational cost of planning and training/maintenance of learned policies must be included in full economic analyses.
  • The planner’s advantage depends on feasibility of zero-failure schedules and accuracy of degradation models; in highly stochastic or poorly modeled settings, RL’s robustness to model misspecification may outweigh planners’ guarantees.

Overall, for AI economics and operations planning: explicitly choose the decision paradigm according to acceptable failure risk, model fidelity, planning/compute budgets, and time horizon; consider constrained or risk-aware RL formulations or hybrid planning+learning pipelines when both reliability and long-run efficiency matter.