2 cumulative citations
View corpus contextPurpose-trained 8B AI models repair infeasible supply‑chain optimization models to operationally rational solutions far more often than off‑the‑shelf APIs — 81.7% versus a best API of 42.2%; failures cluster in solver interaction and in domain-specific rationale, requiring different fixes: targeted training for solver workflow and explicit, solver‑verifiable checks for rationality.
OptiRepair: Closed-Loop Diagnosis and Repair of Supply Chain Optimization Models with LLM Agents
Purpose-trained 8B-parameter models resolve infeasible multi-echelon supply-chain LPs to both solver feasibility and inventory-theory rationality 81.7% of the time, substantially outperforming off-the-shelf API models (best API 42.2%, average 21.3%), with the biggest shortfall for API models in iterative solver interaction during feasibility repair.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Supply chain optimization models frequently become infeasible because of modeling errors. Diagnosis and repair require scarce OR expertise: analysts must interpret solver diagnostics, trace root causes across echelons, and fix formulations without sacrificing operational soundness. Whether AI agents can perform this task remains untested. We decompose this task into two phases: a domain-agnostic feasibility phase that iteratively repairs any LP using IIS-guided diagnosis, and a domain-specific validation phase that enforces five rationality checks grounded in inventory theory. We test 22 API models from seven families on 976 multi-echelon supply chain problems and train two 8B-parameter models with self-taught reasoning and solver-verified rewards. The trained models reach 81.7% Rational Recovery Rate (RRR) -- the fraction of problems resolved to both feasibility and operational rationality -- versus 42.2% for the best API model and 21.3% on average. The gap concentrates in Phase 1 repair, where API models average 27.6% recovery rate versus 97.2% for trained models. Two gaps separate current AI from reliable model repair: solver interaction, as API models restore only 27.6% of infeasible formulations; and operational rationale, as roughly one in four feasible repairs violate supply chain theory. Each gap requires a different intervention -- targeted training closes the solver interaction gap, while explicit specification as solver-verifiable checks closes the rationality gap. For organizations adopting AI in operational planning, formalizing what 'rational' means in their context is the higher-return investment.
Summary
Main Finding
OptiRepair introduces a two-phase, closed-loop framework that combines solver-in-the-loop diagnosis with domain-specific operational checks to repair infeasible multi-echelon supply-chain LPs. Two 8B-parameter models trained with iterative self-taught reasoning and solver-verified rewards achieve a Rational Recovery Rate (RRR) of 81.7% on the OptiRepair-SC benchmark (976 problems), versus 42.2% for the best off-the-shelf API model and 21.3% on average. The authors identify two distinct failure modes for general-purpose LLMs—poor solver interaction and lack of operational rationale—and show that different remedies are required for each.
Key Points
- Pipeline decomposition
- Phase I (domain-agnostic): iterative diagnosis and repair of infeasible LPs using Gurobi IIS feedback (up to 20 turns).
- Phase II (domain-specific): validation via a Rationality Oracle enforcing five supply-chain checks (e.g., base-stock structure, bounded bullwhip, inventory allocation, cost consistency).
- Benchmark and error taxonomy
- OptiRepair-SC: 976 multi-echelon problems with controlled injections of 10 common modeling errors (ME-1 … ME-10) across demand, timing, capacity, coefficients, indices, and costs.
- Metrics and results
- Primary metric: Rational Recovery Rate (RRR) — fraction of problems restored to solver feasibility and passing operational checks.
- Trained 8B pipeline: 81.7% RRR (Qwen3-8B: 81.7%; Llama-3.1-8B: 80.3%).
- Best API (GPT-5.2 / Gemini 2.5 Pro): 42.2% RRR; mean across 22 API models: 21.3%.
- Phase I gap: API average Phase-1 recovery 27.6% vs trained models 97.2% — indicates general LLMs struggle with solver interaction and iterative repair.
- Phase II gap: ~25% of feasible repairs from models violate operational rationality; codifying domain checks reduces this issue.
- Training approach
- Iterative self-taught reasoning (STaR) with solver-verified rewards (deterministic verification via Gurobi) to bootstrap models for both phases.
- Key insight
- Domain specialization (task-specific training + formalized, solver-verifiable rationality checks) delivers larger practical gains than model scale alone or prompt engineering.
Data & Methods
- Problem generator and instances
- Serial multi-echelon LPs (2–5 echelons, 12–24 periods).
- Typical instance sizes: 70–400 decision variables, 60–350 constraints.
- Baseline instances verified feasible and optimal before error injection.
- Error injection
- Ten error types (ME-1 … ME-10) representing realistic modeling mistakes (demand inflation, lead-time errors, impossible balances, capacity mis-specification, cost-structure errors, bullwhip forcing, coefficient/sign/index errors, redundant constraints).
- A saboteur module ensures the corrupted instance is infeasible (or optimal-but-irrational for cost errors) and that a ground-truth fix exists.
- Two-phase MDP formulation
- Phase I state: (problem description, model code, solver status, IIS contents, history, step count).
- Action space: diagnostic queries (GET_IIS, CHECK_SLACK), repair actions (RELAX/DROP/UPDATE constraint/bounds/coeff), and SUBMIT; transitions deterministic via Gurobi re-solve.
- Phase I success: solver returns Optimal within 20 steps.
- Phase II: domain oracle applies five supply-chain rationality checks (base-stock behavior, bullwhip bounds, inventory allocation, cost consistency, order smoothing) up to 3 turns.
- Models and baselines
- Trained models: two 8B-parameter agents (one per phase) trained with STaR + solver-verified rewards.
- Baselines: 22 API models across 7 families (including GPT-5.2 and Gemini 2.5 Pro).
- Evaluation
- Test set: held-out subset (e.g., 284-problem test for central table) and larger 976-instance benchmark.
- Primary outcomes: Phase I recovery rate, Phase II rationality pass rate, overall RRR.
Implications for AI Economics
- Investment priorities: domain formalization > raw scale
- The paper provides strong empirical evidence that investments in codifying domain constraints and training models on solver-verified interactions produce larger operational gains than relying on larger, general-purpose LLMs or prompt engineering. For firms deciding where to allocate AI budgets, funding the creation of solver-verifiable operational oracles and domain-specific training data yields high returns.
- Labor and productivity effects
- OptiRepair targets a time-consuming expert task (IIS interpretation + repair + operational validation). Reliable automated repair could materially reduce OR analyst hours per model revision, accelerating planning cycles and lowering operational costs—provided the repairs are verified against domain checks to avoid “mathematically valid but operationally meaningless” fixes.
- Risk management and compliance
- The distinction between mathematical feasibility and operational rationality highlights a key risk: naive automated repairs can produce solutions that violate business constraints or safety margins. Organizations must adopt explicit, auditable rationality checks (solver-verifiable) to make AI-driven model repair acceptable for production use, audits, and regulatory scrutiny.
- Cost structure and deployment trade-offs
- Specialized moderately sized models (8B) fine-tuned with solver interactions can outperform large API models, suggesting potentially lower inference and integration costs for in-house deployments. However, building the generator, error taxonomy, and rationality oracle entails upfront engineering and domain-expert effort—an investment that scales with problem complexity and the breadth of LP families to support.
- Product-market implications
- There is a commercial niche for domain-specialized repair agents (supply chain, energy, logistics) that integrate tightly with solvers. Vendors offering packaged solver-verified oracles and fine-tuned agents could displace generic LLM-based assistants for operational optimization workflows.
- Organizational recommendation (practical)
- Before adopting AI for operational planning, organizations should (1) formalize what “rational” means in their context and encode it as solver-verifiable checks; (2) prioritize targeted agent training on solver interactions (IIS + repair loop); (3) maintain human oversight for ambiguous or high-impact repairs; and (4) instrument and log solver-agent interactions for auditability.
- Limitations and open economic questions
- Generalizability: results are for LP-based multi-echelon supply-chain models—other OR domains (integer programs, stochastic programs, non-linear constraints) may require different oracles and training regimes.
- External validity: OptiRepair uses generated instances and simulated errors; real-world complexity, noisy data, and business constraints could change the cost-benefit calculus.
- Incentives and governance: who bears liability for faulty repairs (model vendor, internal AI team, or downstream planners) remains an open governance question with economic implications.
If you want, I can: - Extract the precise five rationality checks and their formal definitions from the appendix, - Produce an executive one-page recommendation for procurement/IT teams considering adoption, or - Sketch a cost-benefit estimate template (engineering + training vs. analyst-hours saved) tailored to your organization.
Assessment
Paper Typedescriptive
Evidence Strengthhigh — Large-scale empirical evaluation (976 multi-echelon LP problems) comparing 22 off-the-shelf API models and two purpose-trained 8B-parameter models, with objective, solver-verified outcome measures (feasibility and five operational rationality checks). Results show large, consistent performance gaps across clearly defined phases, reducing concerns that findings are due to noise or single-case anomalies.
Methods Rigorhigh — The study decomposes the task into two explicit phases, uses IIS-guided diagnosis for Phase 1 and five domain-grounded, solver-verifiable rationality checks for Phase 2, evaluates many baseline API families, trains models with solver-verified reward signals, and reports concrete metrics (Rational Recovery Rate, per-phase rates). The evaluation appears systematic and reproducible, though full assessment would require access to dataset generation procedures and training/validation splits.
SampleBenchmark of 976 multi-echelon supply-chain linear programs (LPs) suffering infeasibility; evaluation includes 22 commercial/open API models from seven families and two internally trained 8B-parameter models trained with self-taught reasoning and solver-verified rewards; outcomes measured by solver feasibility and five inventory-theory-based rationality checks.
Themeshuman_ai_collab productivity
GeneralizabilityLimited to linear-program formulations (LPs); results may not extend to mixed-integer or nonlinear models, Benchmark focuses on multi-echelon inventory/supply-chain problems; other operational domains may differ, API model performance may depend on prompt design and API versions used; rapid API changes could alter baselines, Trained-model results depend on the specific training regimen and model size (8B); scaling laws or different architectures may change outcomes, Relies on IIS-based diagnosis and the chosen five domain checks — alternative diagnosis or rationality definitions could affect conclusions
Claims (11)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Supply chain optimization models frequently become infeasible because of modeling errors. Error Rate | negative | frequency of infeasible optimization models |
Reading fidelity
medium
Study strength
medium
|
not reported
|
| Diagnosis and repair require scarce OR expertise: analysts must interpret solver diagnostics, trace root causes across echelons, and fix formulations without sacrificing operational soundness. Hiring | negative | need for specialized operations research expertise to repair models |
Reading fidelity
high
Study strength
medium
|
not reported
|
| We decompose this task into two phases: a domain-agnostic feasibility phase that iteratively repairs any LP using IIS-guided diagnosis, and a domain-specific validation phase that enforces five rationality checks grounded in inventory theory. Task Allocation | positive | ability to repair LP feasibility and enforce domain rationality |
Reading fidelity
high
Study strength
speculative
|
not reported
|
| We test 22 API models from seven families on 976 multi-echelon supply chain problems and train two 8B-parameter models with self-taught reasoning and solver-verified rewards. Adoption Rate | neutral | empirical evaluation coverage (models tested and problems solved) |
Reading fidelity
high
Study strength
high
|
n=976
|
| The trained models reach 81.7% Rational Recovery Rate (RRR) -- the fraction of problems resolved to both feasibility and operational rationality -- versus 42.2% for the best API model and 21.3% on average. Developer Productivity | positive | Rational Recovery Rate (RRR): fraction of problems resolved to feasibility and operational rationality |
Reading fidelity
high
Study strength
high
|
n=976
81.7% RRR (trained) vs 42.2% (best API) vs 21.3% (API average)
|
| The gap concentrates in Phase 1 repair, where API models average 27.6% recovery rate versus 97.2% for trained models. Developer Productivity | negative | Phase 1 (feasibility) recovery rate |
Reading fidelity
high
Study strength
high
|
n=976
API average 27.6% vs trained 97.2%
|
| Two gaps separate current AI from reliable model repair: solver interaction, as API models restore only 27.6% of infeasible formulations; and operational rationale, as roughly one in four feasible repairs violate supply chain theory. Developer Productivity | negative | solver-interaction repair success; operational rationality compliance |
Reading fidelity
high
Study strength
high
|
n=976
27.6% solver-interaction success; roughly 1 in 4 rationality violations
|
| API models restore only 27.6% of infeasible formulations (solver interaction gap). Developer Productivity | negative | fraction of infeasible formulations restored by API models |
Reading fidelity
high
Study strength
high
|
n=976
27.6%
|
| Roughly one in four feasible repairs violate supply chain theory (operational rationale gap). Output Quality | negative | fraction of feasible repairs that violate domain rationality checks |
Reading fidelity
high
Study strength
high
|
n=976
roughly one in four
|
| Targeted training closes the solver interaction gap, while explicit specification as solver-verifiable checks closes the rationality gap. Training Effectiveness | positive | effect of targeted training and explicit checks on repair success and rationality compliance |
Reading fidelity
medium
Study strength
medium
|
n=976
|
| For organizations adopting AI in operational planning, formalizing what 'rational' means in their context is the higher-return investment. Organizational Efficiency | positive | relative return on investment of formalizing domain rationality vs other interventions |
Reading fidelity
high
Study strength
speculative
|
not reported
|