A new benchmark built from 223 real-world mixed-integer programs reveals that large language models that excel on toy tasks struggle to translate realistic, industrial-scale optimization requirements into correct formulations, exposing practical failure modes invisible at small scale.

Main Finding

The paper introduces MIPLIB-NL, the first NL-to-Optimization benchmark constructed by reverse-generating natural-language problem specifications from real industrial mixed-integer linear programs in MIPLIB 2017. Using a structure-aware reverse construction pipeline (loop/constraint-family recovery, expert-driven NL generation, and iterative semantic validation), the authors produce 223 one-to-one reconstructions that preserve the original mathematical content. State-of-the-art LLM-based NL-to-Opt systems that perform well on existing toy benchmarks suffer substantial performance degradation on MIPLIB-NL, revealing failure modes and overestimation of current capabilities.

Key Points

• Motivation
  • Prior NL-to-Opt benchmarks are toy-sized or synthetic and flatten industrial compositional structure (indexing, constraint families), leading to inflated estimates of LLM capability.
  • Real industrial MILPs often have loop-based indexing and huge scale (103–106+ variables/constraints) absent from earlier datasets.
• Dataset (MIPLIB-NL)
  • 223 instances reverse-generated from MIPLIB 2017 (.mps files).
  • Each instance is a faithful one-to-one reconstruction preserving mathematical structure and numerical properties.
  • Instances are provided in a unified model–data separation format so NL descriptions remain compact while models can expand to industrial scale.
  • The dataset exhibits a heavy upper tail of instance sizes — many far larger than prior benchmarks.
• Construction pipeline (three stages)
• Structural abstraction: expert-driven recovery of indexed variable groups and repeated constraint families from flat MPS algebra (loop-based scaffold).
• Structure-preserving Opt→NL generation: experts convert the scaffold into deterministic NL blueprints and readable problem narratives tied to the recovered structure.
• Semantic validation: independent NL-to-Opt reconstructions, human–LLM interaction, and expert audits ensure semantic sufficiency and exact equivalence when scaffolds are expanded.
• Structural findings
  • Common industrial loop structures recovered include nested loops, subset-indexed loops, temporal/recursive coupling, sliding-window aggregations, pairwise (complete-graph) instantiation, and extra-dimension replication.
  • Treating families of constraints (loop templates) as primitives is crucial; atomic constraint types alone are insufficient.
• Evaluation results
  • Broad evaluation of recent LLM-based NL-to-Opt systems (prompting, agentic frameworks, fine-tuned models) shows large performance drops on MIPLIB-NL versus existing datasets.
  • This gap implies previous benchmarks masked industrial-scale failure modes (indexing errors, missing constraint families, structural hallucinations).
• Contributions summarized
• Identifies dataset design as a central bottleneck for NL-to-Opt evaluation.
• Proposes a structure-aware reverse generation methodology to derive NL instances from real MILPs.
• Releases MIPLIB-NL (223 industrially grounded instances).
• Demonstrates systematic performance degradation of current methods on industrial-scale problems.

Data & Methods

• Source
  • Base models: MIPLIB 2017 mixed-integer linear program instances (provided in solver/ algebraic formats like .mps).
• Structural abstraction methodology
  • Experts analyze MPS-level signals (naming regularities, incidence structure) and MIPLIB metadata to hypothesize loop-based index sets and constraint families.
  • Drafted scaffolds are expanded back to algebraic rows and checked for exact equivalence against original MPS to remove subjectivity.
  • A taxonomy of recovered scaffolds is produced (see nested, subset-indexed, temporal coupling, sliding-window, pairwise, replication).
• Opt→NL generation
  • For each validated scaffold, deterministic NL blueprints map variables, objectives, and constraint families into a natural-language problem description. Optional linguistic polishing retains strict mathematical fidelity.
  • Application contexts (domain narratives) are attached using MIPLIB cues where possible; ambiguous instances are given plausible contexts.
  • A strict model–data separation schema (stored artifacts described in appendix) keeps NL compact while supporting data-driven expansion.
• Semantic validation
  • Independent reconstructors (human/LLM) attempt to rebuild the optimization model from the NL + data spec.
  • Iterative human–LLM interaction and expert review verify that reconstructed models are semantically equivalent to originals.
  • Instances failing semantic sufficiency or verification are refined or discarded.
• Dataset scale and format
  • 223 fully validated instances with loop-based scaffolds and corresponding NL descriptions plus data artifacts. Many instances reach industrial scales (thousands to millions of algebraic rows when expanded).
• Evaluation
  • Benchmarking uses accuracy/Pass@1-style metrics and structural equivalence checks (beyond matching objective values) to detect hallucinations and missing constraints.
  • Wide range of LLM-based systems tested (prompting, fine-tuned models, domain-specific agents), showing clear drops versus toy datasets.

Implications for AI Economics

• For research and practice
  • Realistic stress-testing: MIPLIB-NL enables evaluation of LLM-based modeling tools on industrial-scale, structurally-rich optimization problems common in economics (supply chains, energy market clearing, production planning, portfolio optimization).
  • Reveals current limitations: High-level LLM competence on toy problems does not guarantee correctness on large, indexed economic models. Models commonly hallucinate structure (missing index dimensions, omitted constraint families), which in economics could produce severely misleading policy or allocation recommendations.
  • Need for structure-aware methods: Economic modeling systems should incorporate explicit treatment of indexed variable groups and constraint families (loop scaffolds), not just atomic constraint templates.
• For deployment and risk management
  • Verification is essential: Deployments that use LLMs to generate optimization models for economic decision-making must include symbolic verification (expand scaffolds, check equivalence, run solver validations) and human expert audits to prevent high-cost errors.
  • Evaluate beyond objective values: Benchmarks and deployment checks should test structural equivalence, feasibility, indexing correctness, and constraint completeness—not only objective outcomes—to avoid silent structural failures.
• For model development
  • Dataset-driven improvement: Researchers should use MIPLIB-NL for fine-tuning or structured prompting so models learn industrial scaffolds (nested indices, temporal coupling, sliding windows).
  • Hybrid architectures: Combining LLMs (for narrative-to-structure mapping) with symbolic programs/DSLs that enforce indexed constructs and deterministic expansion may yield more reliable NL-to-Opt systems for economics.
  • Metrics & training targets: Train models to produce explicit loop scaffolds (index sets + family templates) as intermediate outputs; use scaffold expansion and equivalence checking as a training/validation signal.
• Future opportunities
  • Domain specialization: Extend MIPLIB-NL-style reverse generation to domain-specific MILP collections (energy markets, transportation, financial networks) to provide targeted benchmarks for economics subfields.
  • Stochastic/nonlinear models: Extend methodology beyond MILPs to stochastic programs or nonlinear formulations used in economic modeling.
  • Tooling & auditing: Develop off-the-shelf verification tools and prompt templates that encourage structural completeness (e.g., request explicit index definitions, loop constructs), and incorporate MIPLIB-NL as a standard evaluation suite for regulatory-grade validation.

Suggestions for practitioners in AI economics - Use MIPLIB-NL to benchmark any LLM-based modeling tool before applying it to high-stakes economic decision-making. - Require automated expansion + equivalence checks and human expert sign-off for any generated optimization model. - Favor hybrid LLM + symbolic code pipelines that output explicit indexed scaffolds and deterministic expansions. - Prioritize dataset-driven fine-tuning on industrial-scale, loop-structured instances to reduce structural hallucinations.

If you want, I can: - extract specific examples of loop scaffolds and representative MIPLIB-NL instances relevant to supply-chain or energy-economic models; or - propose an evaluation checklist and verification pipeline tailored for deploying NL-to-Opt systems in economic decision-making.