Machine-learning surrogates that predict plant measurements closely can still mislead real-time optimization: in a vinyl-acetate benchmark the structured model lost ~30% of profit and the black-box over 50% when used for RTO, and even noise-free retraining with stochastic optimizers can create phantom optima.

Main Finding

Data-driven models that fit plant measurements very accurately — including physics-structured hybrids that enforce mass balances and thermodynamics — can still fail to produce the correct economic optimum when used inside real-time optimization (RTO). Such models can (a) produce many spurious ("phantom") optima instead of the plant's unique, well-conditioned optimum and (b) become biased by the stochastic training optimizer itself (weights can drift away from a correct solution even with noise-free data and initialization at the true weights). Consequently, standard predictive validation does not guarantee reliable economic performance; decision-oriented benchmarks and objective-aware model checks are required before deployment.

Key Points

• Case study: a vinyl acetate monomer (VAc) production flowsheet (reactor + separators, 7 species, 13 streams) with a unique, well-conditioned steady-state economic optimum:
  • Optimal decision vector u* = [15.54, 11.14, 17.15, 0.57, 0.6, 427.1 K] (feed rates, recycle fraction, temp)
  • Optimal profit ℓ* = $292.87/s and production N10V = 13.13 mol/s
  • Scaled reduced Hessian at optimum is negative definite and well-conditioned (condition number ≈ 415) → strict local maximum, no other local optima found by multi-start search
  • Profit margin ≈ 7.4% of revenue
• Models tested:
  • Structured (hybrid) model: mechanistic mass/thermo laws retained; unknown reaction rates replaced by neural-network closures. Training formulations:
    • rmeas: train networks directly on rate measurements (r1,r2) — diagnostic / reference
    • cmeas_rinit: train on concentrations but initialized from rmeas weights
    • cmeas: train on concentrations only
  • Black-box model: neural ODE approximating full ODE right-hand-side (no hard physical constraints)
• Primary empirical outcomes:
  • Both structured and black-box models reproduced concentration measurements accurately and showed little prediction variance across random initializations, passing standard validation checks.
  • Despite good predictive fits, the economic optima obtained by optimizing these trained models differed substantially from the plant optimum. The trained models yielded many phantom optima while the plant had a single optimum.
  • Profit losses in worst cases (graphical abstract): structured-model loss ≈ 30% of plant profit; black-box model loss > 50%.
  • Even with noise-free data and initialization at weights that reproduce the plant optimum, stochastic gradient training (SGD-type optimizers) can drift to parameter values yielding much worse RTO outcomes — i.e., the identified model can be an artifact of the training optimizer as well as the data.
• Interpretation:
  • Good predictive accuracy on available measurements (and even low predictive variance) is insufficient to guarantee that the model captures decision-relevant derivatives/curvature that determine economic optima.
  • Rate measurements (when available) are the most informative for recovering kinetics; training on rates is a useful diagnostic/baseline.

Data & Methods

• Ground truth: a detailed reactor model (isothermal CSTR) with overall reaction rates r1, r2 and a simplified separator (flash, absorber, distillation) flowsheet forming a DAE system; for experiments the DAE was reduced to an ODE for transient training rollouts.
• Economic objective: steady-state profit function combining revenue from vinyl acetate and processing/material costs; RTO problem solved with CasADi/IPOPT.
• Training data:
  • Simulated plant trajectories under step changes in decision variables (fresh feeds, recycle fractions, reactor inlet T).
  • Full-state concentration measurements available in training (y = x).
  • A variant includes direct rate measurements (r1,r2) used in the rmeas case.
• Model classes & training formulations:
  • Structured/hybrid: mechanistic balances & EOS enforced; NN closures for r1, r2. Trained by minimizing multi-step rollout error in concentrations (eq. 9) or (for rmeas) direct rate-fitting loss (eq. 10).
  • Black-box neural ODE: NN approximates RHS f(x,u) and trained by multi-step rollout concentration loss (eq. 11).
  • Loss normalization by measurement std. Multi-step rollout horizon used to reduce compounding one-step errors.
• Evaluation:
  • Standard predictive validation (fit to training/validation trajectories), plus multi-start RTO using the trained models to obtain optimized steady-state setpoints.
  • Comparison of optimized profit (model-based RTO) vs plant profit when those setpoints are applied to the plant model.
  • Diagnostic experiments isolating the effect of the training optimizer: initializing at weights that recover the plant optimum and retraining stochastically to observe drift.
• Additional details (hyperparameters, optimizer comparisons) provided in supplementary material.

Implications for AI Economics

• For economic decision tasks (RTO, optimal control, plant economics), predictive accuracy alone is an inadequate model acceptance criterion. Models must be validated for decision-relevant properties (gradients, curvature, uniqueness of optimum).
• Deployment risk: data-driven RTO models that pass standard validation may still prescribe significantly suboptimal (or unsafe) operating points, causing substantial profit loss and operational risk.
• Practical recommendations for practitioners and AI-in-industry teams:
  • Use decision-oriented benchmarks (like the VAc benchmark developed) that require models to recover plant optima, not just to fit data.
  • Where possible collect and use the most informative measurements (e.g., reaction-rate measurements) — they are directly informative about closures and reduce ambiguity.
  • Augment training objectives to be decision-aware:
    • Include economic-gradient and curvature matching terms in the loss, or train end-to-end on the downstream optimization objective (RTO-aware training).
    • Penalize models that create spurious optima in the feasible region (e.g., via adversarial search during training).
  • Enforce hard physical constraints (conservation laws, EOS) where feasible to reduce infeasible extrapolations; but note that even physics-constrained hybrids can fail to recover the correct optimum.
  • Use modifier-adaptation or gradient-correction schemes in closed-loop to guarantee convergence to the plant optimum — but be aware these schemes assume the model has reasonable curvature at the plant optimum.
  • Treat the stochastic training optimizer as a potential source of bias: run multiple optimizer seeds, use deterministic or carefully-tuned optimizers for decision-critical deployments, monitor parameter drift, and validate stability of RTO outcomes under retraining.
  • Adopt decision-focused validation metrics: evaluate models by the profit loss when optimized setpoints are applied to the plant, not only by trajectory RMSE.
  • Insist on recovery-of-optimum tests before any plant trials: models should be required to recover known optima on representative benchmarks.
• Research directions: develop training/regularization methods that preserve decision-relevant derivatives, design metrics and benchmarks for decision-oriented ML in process systems, and better understand optimizer-induced identification artifacts in stochastic training.

Limitations and scope: results are demonstrated on a simulation benchmark (VAc flowsheet) with controlled experiments; behavior may vary with architectures, datasets, and process complexity, but the qualitative risk—that excellent predictive fit need not imply correct economic optima—generalizes to other process RTO applications.