Large language models can do local calculations in optimization problems but cannot reliably build and enforce global constraints; integrating solvers helps with arithmetic yet automated constraint formulation remains the main failure mode as tasks scale.

Main Finding

OPT-ENGINE is an extensible, complexity-scalable benchmark for LLM-driven optimization modeling. Using it, the authors show that (1) Pure-Text Reasoning (PTR, chain-of-thought) breaks down rapidly as problem scale/constraint density grows, (2) invoking external solvers (Solver-Integrated Reasoning, SIR) preserves accuracy at scale, and (3) the central remaining bottleneck for current SIR pipelines is automated, correct formulation of constraints (not solver computation or objective perturbations). External computational tools help with local arithmetic but do not reliably enforce global feasibility when formulation is wrong.

Key Points

• OPT-ENGINE: benchmark taxonomy of 10 canonical OR problems:
  • LP family (5): inventory, portfolio allocation, production, transportation, pollution control.
  • MIP family (5): traveling salesman (TSP), knapsack, bin packing, job-shop scheduling, min-cost network flow.
• Generation pipeline (four stages):
• Numeric instance generation (G) with controllable difficulty parameters θ (e.g., Ncities).
• Canonical problem construction via templates (M).
• Problem augmentation / rephrasing by an LLM agent (R) to produce diverse narratives.
• Instance validation (V): LLM judge + rule-based verifier; resample/repair until consistent with ground-truth solver outputs.
• Evaluation protocol:
  • Ten distinct instances per problem at each complexity level.
  • Solution considered correct if relative error < 1e-3.
  • Report avg@10 (mean success rate over 10 instances).
• Empirical comparisons:
  • Models tested include DeepSeek-V3.2, GPT-5.1, and Qwen3-4B variants (Instruct and RL-enhanced).
  • SIR (autoformulate then call a solver) maintains high accuracy or small degradation as scale increases.
  • PTR (end-to-end textual chain-of-thought) shows severe accuracy collapse as complexity grows.
  • Small models sometimes show PTR > SIR at tiny problem sizes, but this reverses with scale; improving code-generation (via RL training) raises SIR performance substantially.
  • Execution rates (ability to produce runnable solver code) vary widely across models and correlate with SIR success.
• Failure-mode analysis for PTR:
  • Token-length and trace analysis show models reduce explicit reasoning steps near a complexity threshold (a precursor to collapse).
  • Error decomposition distinguishes arithmetic mistakes from structural/constraint formulation errors: external calculators fix arithmetic but not incorrect constraints.
• Conclusion: SIR is the practical route to industrial-scale optimization with LLMs; the key research focus should be reliable, accurate auto-formulation of constraints and form-to-solver fidelity.

Data & Methods

• OPT-ENGINE generation mechanics:
  • G : D × Θ → I produces numeric problem instances; infeasible draws are resampled.
  • M : I × T → SC maps numeric instance to canonical statement.
  • R : SC × L → SR uses an LLM to generate multiple rephrasings/narratives.
  • V (validator) uses LLM judging plus rule-based checks to ensure numerical and structural consistency; invalid augmentations are re-run.
• Complexity control: each class exposes interpretable parameters (e.g., Ncities for TSP, Nassets for portfolio) so problem size and constraint density are tunable.
• Experimental setup for algorithmic comparison:
  • For each instance, obtain LLM reasoning trace z(1..m).
  • PTR: final trace contains numeric answer ˆy(PTR); SIR: final trace contains solver-code executed to get ˆy(SIR).
  • Success criterion: |ˆy − y|/(|y| + 1e-6) < 1e-3.
  • Ten instances per level; report avg@10.
• Models and interventions:
  • Frontier models (DeepSeek-V3.2, GPT-5.1) and smaller Qwen3-4B series.
  • Qwen3-4B-RL is an RL-fine-tuned variant to improve code generation / execution rate (RLVR).
• Analyses:
  • Scaling curves across problems and models.
  • Token-usage and trace pattern study for PTR collapse signatures.
  • Error decomposition into infeasibility (constraint violations) vs arithmetic inaccuracies.
• Reproducibility: code and data released (GitHub link in paper).

Implications for AI Economics

• Practical adoption guidance for economists and operations researchers:
  • Use solver-integrated pipelines (SIR) when dealing with medium-to-large-scale optimization tasks. PTR-based, chain-of-thought outputs are brittle for policy, allocation, and large-scale resource-optimization decisions.
  • Always validate LLM-produced formulations (constraints, variable definitions) with rule-based checks and run deterministic solvers to obtain final numeric optima.
• Risk and robustness considerations:
  • PTR can produce plausible-looking but infeasible or suboptimal formulations; relying on raw textual outputs for economic decisions risks silently violating constraints (e.g., budget/feasibility limits).
  • Automated constraint mis-formulation is the main practical failure mode; this has downstream economic consequences (misallocated funds, invalid schedules, regulatory non-compliance).
• Research and tool development priorities for AI applied to economics:
  • Focus funding and effort on improving automated, verifiable constraint formulation (structure-aware LLMs, constraint-aware fine-tuning, checking/repair agents).
  • Invest in hybrid systems that pair LLMs for language and model assembly with deterministic solvers for computation and final validation.
  • Develop and adopt benchmark-driven evaluation (e.g., OPT-ENGINE) for economic/OR workflows to test generalization across complexity and linguistic variation.
• Policy and governance:
  • For economic policy-support tools that incorporate LLMs, require solver-backed verification and transparency of constraint/specification generation; log formulation steps for auditability.
  • Encourage standards for minimum validation (e.g., solver-recomputed optimality and feasibility checks) before deployment in high-stakes economic decisions.
• Short actionable checklist for practitioners:
  • Prefer SIR architectures for production optimization tasks.
  • Use benchmarked pipelines like OPT-ENGINE during model selection and stress-testing.
  • Automatically validate constraint integrity (syntactic and semantic) before trusting solver outputs.
  • Treat PTR outputs as exploratory or prototyping aids only, not as final solutions for large/critical optimization models.

If you want, I can extract the concrete quantitative scaling plots or summarize the per-problem scaling behavior (e.g., at what sizes PTR collapses for TSP vs. knapsack) from the paper's figures and tables.