A new differentiable DRL policy that enforces hard combinatorial inventory constraints cuts simulated supply‑chain costs by up to 9.8% on literature benchmarks and by about 3.2% on an ASML industry instance, outperforming echelon base‑stock and rolling‑horizon stochastic programs.

Main Finding

The paper introduces a differentiable policy architecture that enforces hard, interdependent action constraints for sequential decision problems (e.g., capacitated multi-echelon inventory planning). It combines a neural-network decision rule, a convex quadratic-program (QP) projection onto the continuous feasible set, and a dual-informed integer mapping with straight-through surrogate gradients. The approach trains end-to-end via pathwise gradients (given a differentiable simulator), attains near-optimal performance on small instances (<1% optimality gap), and yields economically meaningful cost reductions versus state-of-the-art heuristics and rolling-horizon stochastic programming on larger and industry-scale benchmarks (up to ~9.8% and 3.22% improvements reported).

Key Points

• Problem class: finite-horizon MDPs with combinatorial (integer) actions and interdependent linear constraints split into state-dependent constraints Ax ≤ b_t and static constraints Cx ≤ k; continuous relaxation is A_rel(s_t).
• Policy architecture (three modules):
• Neural network outputs continuous action targets z (differentiable decision rule).
• QP projection maps z to the continuous feasible set A_rel(s_t). A QP (strongly convex projection) is used to ensure unique, smooth optimizer and stable implicit gradients. Projection sensitivities are shown to be nonexpansive (helping gradient stability).
• Dual-informed integer mapping converts projected continuous actions to feasible integer actions using the projection’s dual variables to prioritize rounding; surrogate straight-through estimator (STE) supplies backward gradients.
• Theoretical guarantees:
  • Bounded error relative to exact integer projection (both in distance and decision outcome).
  • Completeness: every feasible integer action is reachable from a positive-measure set of target actions (avoids unreachable frontier actions that naive rounding can cause).
  • Frontier condition: when the target overshoots in all coordinates the mapping returns an action that is limited in at least one coordinate (allowing pushing against binding constraints).
• Training: end-to-end via pathwise (deterministic) gradients when a differentiable simulator is available (they build on HDPO-style low-variance gradient signals).
• Empirical results:
  • Small 3-item instances: average optimality gap < 1% without instance-specific tuning.
  • 10-item benchmark (Tempelmeier & Derstroff style): up to 9.75% cost savings vs state-of-the-art echelon base-stock policies; at least 7.7% better than rolling-horizon multi-stage stochastic programming.
  • ASML industry-scale 14-item benchmark: average cost reduced by up to 3.22% vs best-known forecast-based echelon base-stock policy; realized-cost distribution is first-order stochastically dominated.
  • Ablations: removing projection sensitivities causes training divergence; replacing dual-informed mapping with naive floor rounding increases cost up to 15.8%.
• Where benefits concentrate: tightly capacitated networks, high demand variability, and shared resources across echelons — precisely the regimes where MILP/heuristics struggle most.
• Practical design advantages: constraints are enforced via a projection module specified by the model (formulation-driven), so adding/changing constraints does not require reengineering the learning architecture. Duals provide interpretable prioritization signals.

Data & Methods

• Formal setting: finite-horizon constrained MDP with integer actions x_t ∈ N_0^n satisfying Ax_t ≤ b_t (state-dependent) and Cx_t ≤ k (static); transitions s_{t+1} = Γ(s_t, x_t, ξ_t). The continuous relaxation A_rel(s_t) replaces integrality with nonnegativity.
• Projection choice: quadratic-program projection (minimize squared distance to target z subject to linear inequalities of the continuous relaxation). QP chosen over LP to avoid degeneracy and discontinuous optimizer behavior; implicit differentiation of KKT conditions yields analytical gradients.
• Integer mapping: uses dual variables from the projection to rank/prioritize rounding into integer decisions that satisfy constraints; guarantees on bounded deviation from exact integer projection and reachability of integer feasible space.
• Surrogate gradients: straight-through estimator (STE) for passing gradients through the discrete mapping during backpropagation.
• Training protocol: pathwise gradients through differentiable simulator (HDPO-style), sampling trajectories to compute low-variance gradient estimates; no per-state MILP solving during training/deployment (only projection QPs + integer mapping).
• Scalability: exploits decomposition/locality of constraints common in inventory networks (resource constraints localized to machines/facilities/BOMs), allowing parallelizable smaller QPs and tractable integer mapping for larger instances.
• Benchmarks and evaluation: small instances with known optima, literature benchmark networks (10-item), and a realistic industry-scale ASML instance (14-item) under multiple demand scenarios; baselines include optimized echelon base-stock policies (forecast-based) and rolling-horizon multi-stage stochastic programming.

Implications for AI Economics

• Operational cost impact: Demonstrates that DRL methods can yield nontrivial, economically meaningful savings (several percent reductions at industry scale; larger gains in harder settings). Even modest percentage improvements can translate into substantial dollar savings across large operations.
• Replacing/augmenting MILP in stochastic settings: The approach offers a tractable alternative to per-period MILPs or expensive rolling-horizon stochastic programs in stochastic, constrained settings where MILPs scale poorly. It keeps the benefits of formulation-driven constraint specification while enabling scalable, data-driven policies.
• Where to prioritize adoption: strongest value when constraints are tight, demand is volatile, and resources are shared — exactly the complex environments where heuristic policies perform poorly and manual tuning is costly.
• Interpretability & diagnostics: Use of projection duals provides economically interpretable priority signals (e.g., marginal value of resources), which can aid managerial understanding and trust compared with black-box policies.
• Practical caveats & adoption constraints:
  • Differentiable simulator requirement: pathwise gradient training needs a simulator that is differentiable w.r.t. actions and (often) states. Building or adapting simulators may be nontrivial in practice; if unavailable, alternative gradient estimators or surrogate approaches would be needed.
  • Approximateness of integer mapping: the integer mapping is proven to be bounded-close to exact integer projection and complete, but it is not exact optimization; thus, worst-case performance guarantees are weaker than exact MILP optima.
  • Training complexity: while deployment requires only QPs and rounding heuristics, training can still be compute-intensive and data-hungry; practitioners should assess compute vs benefit trade-offs relative to existing planning tools.
  • Robustness and generalization: as with learned policies broadly, out-of-distribution demand or structural changes may degrade performance; combining learned policies with fallback rule-based safety/feasibility checks is prudent.
• Research and policy implications: This work shows that differentiable optimization layers permit enforcement of hard operational constraints within DRL, suggesting a broader role for such hybrid architectures in economic applications where constraints and combinatorics matter (inventory, workforce scheduling, energy systems, financial regulations). Economists and operations researchers should consider integrating dual information from learned policies into managerial decision-support and using such methods to explore counterfactual operational investments (e.g., where increasing capacity yields largest marginal returns according to the learned duals).

Suggested next steps for practitioners/economists: - Pilot on a representative segment where capacity is tight and demand stochasticity is high. - Ensure or develop a differentiable simulator; otherwise explore variants using policy gradients with constraint-aware surrogates. - Monitor worst-case constraint satisfaction during rollout and keep conservative safeguards for high-stakes operations. - Leverage projection duals as operational KPIs for capacity planning and as inputs to hybrid human–AI workflows.