An agentic LLM workflow, LEAN-LLM-OPT, can auto-formulate large-scale optimization models and matches or beats prior methods on new Large-Scale-OR and Air-NRM benchmarks; in a Singapore Airlines revenue-management trial it delivers leading performance across tested scenarios.

Main Finding

LEAN-LLM-OPT is an agentic workflow framework that lets LLMs reliably auto-formulate large-scale optimization (LP/MILP) models from natural-language problem descriptions plus external datasets. By decomposing modeling into (1) problem classification, (2) dynamic workflow generation using reference problems, and (3) workflow-guided model generation with auxiliary data tools, the system achieves strong modeling accuracy on medium-to-large problems (≥20 variables, many ≥100). Instantiated with GPT-4.1 and an open-source 20B model, it outperforms competing LLM approaches on the new Large-Scale-OR benchmark and a Singapore Airlines revenue-management use case.

Key Points

• Problem addressed: building large-scale optimization models is labor-intensive; standard prompting/fine-tuning approaches weaken as input size and heterogeneity grow.
• Architecture: three cooperating LLM agents
  • Classification agent: identifies the problem type (e.g., NRM, resource allocation).
  • Workflow generation agent: dynamically constructs a structured, step-by-step workflow for that query, including reference problems and which dataset components to focus on.
  • Model generation agent: follows the workflow, uses customized tools to extract/process data, and outputs the mathematical formulation plus solver-ready Python code.
• Design insight: offload mechanical, repetitive data handling to tools and use the LLM’s flexibility for unstructured reasoning and decomposition, reducing the burden on any single monolithic prompt or fine-tuned model.
• Benchmarks and results:
  • Introduced two datasets: Ref-Data (96 reference instances) and Large-Scale-OR (101 testing instances, with 50% ≥100 variables).
  • LEAN-LLM-OPT (GPT-4.1 and gpt-oss-20B) attains >76% overall accuracy on Large-Scale-OR, outperforming ORLM, Gemini 3 Pro, GPT-5.2 and others.
  • Performs strongly on other benchmarks (NL4OPT, IndustryOR, Mamo).
  • Singapore Airlines case (Air-NRM): achieves top accuracy on fare-type capacity allocation (Air-NRM-CA, 15 instances) and small optimality gaps on joint network planning (Air-NRM-NP, 21 instances).
• Ablation: both agentic workflow construction and the auxiliary data-handling tools materially improve accuracy; removing either degrades performance.
• Practical artifacts: authors release code and data (GitHub link in paper).

Data & Methods

• Datasets:
  • Ref-Data: 96 curated instances (78 small, 18 large) from textbooks, Kaggle, and industry cases; labeled with problem type and expert formulations; used as references during workflow generation and classification.
  • Large-Scale-OR: 101 testing instances spanning medical, e-commerce, supply-chain, etc.; emphasizes medium (20–99 vars, 26%) and large (≥100 vars, 50%) model sizes; each instance includes problem description, datasets, ground-truth formulation, and optimal value.
  • Air-NRM benchmarks: Air-NRM-CA (15 instances) and Air-NRM-NP (21 instances) built for Singapore Airlines SBLP use case.
• Evaluation metrics:
  • Modeling accuracy (agreement with ground-truth formulation) and optimality gap (for solution quality where applicable).
• Agentic workflow mechanics:
  • Workflow generation agent selects reference problems and synthesizes a sequence of subtasks (e.g., identify decision variables, map datasets to parameter matrices, write constraints).
  • Model generation agent executes each step, invokes specialized tools for dataset parsing and mechanical transformations (so LLM token usage focuses on reasoning), and emits optimization formulation + solver code.
• Models evaluated:
  • Proprietary: GPT-4.1 (primary high-performing backbone), comparisons with Gemini 3 Pro, GPT-5.2.
  • Open-source: gpt-oss-20B (demonstrated viability for lower-cost instantiation).
• Comparative experiments and ablation tests demonstrate robustness across multiple benchmarks and show reduced degradation on long/higher-volume inputs relative to baseline prompting/fine-tuning approaches.

Implications for AI Economics

• Productivity and cost effects
  • Labor displacement/substitution: automating the formulation step can reduce demand for routine optimization-modeling labor (junior analysts), while increasing demand for oversight, validation, and higher-level modeling expertise.
  • Lower bar to entry: firms without deep optimization teams can deploy bespoke LP/MIP models faster, potentially compressing first-mover advantages in analytics-intensive domains.
  • Training vs. orchestration trade-off: LEAN-LLM-OPT suggests an alternative to expensive large-scale fine-tuning—coordinate smaller, reusable assets (reference problems + tools + agents)—which lowers adoption costs and accelerates diffusion.
• Firm heterogeneity and competition
  • Capability diffusion may narrow gaps between analytics-rich incumbents and smaller competitors, changing competitive dynamics in logistics, revenue management, supply chains, and service operations.
  • Strategic value shifts from model construction to data quality, feature engineering, and integration—firms excelling at curated datasets and validation pipelines will retain advantage.
• Market for AI tools and services
  • Demand growth for modular tooling (data parsers, verifiers, model-checkers) and for human-in-the-loop validation services. Audit, verification, and robustness testing become commercial services.
  • Open-source instantiations (e.g., gpt-oss-20B) enable lower-cost deployment, raising concerns about commoditization of optimization advice and price competition in consulting.
• Policy, governance, and risk
  • Model correctness and economic risk: incorrectly formulated optimization models can propagate costly operational mistakes (e.g., capacity misallocation). Regulators and firms may need standards for validation, explainability, and fallback human oversight.
  • Labor market transitions: policies for retraining and role respecification (from model-building to supervision/evaluation) may be warranted in sectors with large optimization teams.
• Research directions for economists
  • Empirical studies on adoption rates across firm sizes and sectors, and productivity impacts after deploying agentic LLM pipelines.
  • Welfare analysis: consumer and producer surplus changes when optimization capability becomes cheaper and more widespread.
  • Labor-demand modeling: how wages and employment composition evolve for analysts, OR specialists, and related occupations.
  • Market-structure modeling: how diffusion affects competition, entry/exit, and price-setting where optimization gives operational edge (airlines, logistics, retail).

Limitations and caveats to note - Focus is on LP/MILP; extension to nonlinear, stochastic, or dynamic programs needs further validation. - Dependence on curated Ref-Data and workflow templates may limit zero-shot generalization to entirely novel problem families. - Hallucination and correctness guarantees remain concerns—human validation and formal checks are still necessary in high-stakes deployments. - Computational and latency costs of multi-agent orchestration were not deeply discussed; real-time or very large-scale deployments may require engineering trade-offs.

If you want, I can: - Extract concise quantitative results (tables of accuracy by model and benchmark) from the paper for quick inclusion in a slide or brief. - Propose empirical study designs for measuring firm-level productivity effects of adopting LEAN-LLM-OPT–style tools.