LLMize lets language models solve messy optimization problems by expressing constraints and heuristics in plain English, but it cannot beat classical solvers on standard benchmarks; its strength lies in accessible, domain-specific tuning where formal mathematical models are impractical.

Main Finding

LLMize is an open-source Python framework that turns large language models (LLMs) into black-box proposal generators for iterative numerical optimization. By encoding problem descriptions, constraints, and prior solution–score pairs in natural language prompts, LLMize uses in‑context learning to produce and refine candidate solutions. It implements three strategies (OPRO-style iterative prompting, a hybrid LLM–evolutionary algorithm HLMEA, and a hybrid LLM–simulated annealing HLMSA). The framework is model- and problem-agnostic and is most useful for complex, domain-specific black‑box problems where constraints or heuristics are hard to formalize — while classical solvers still outperform it on simple, well-structured problems.

Key Points

• Core idea: treat the LLM as a proposal mechanism pθ(x | Prompt) and adapt proposals via in-context learning by appending evaluated (x, f(x)) pairs to the prompt history Ht.
• Problem class targeted: black-box optimization (simulators, experiments, non-analytic constraints, unavailable gradients).
• Three implemented strategies:
  • OPRO-style iterative prompting: prompt contains problem description + history; sample candidates, evaluate externally, append feedback.
  • HLMEA (Hybrid LLM–Evolutionary Algo): LLM is prompted to act like an evolutionary algorithm (select parents, propose crossovers/mutations, suggest hyperparameters); elitism implemented by keeping top K examples in the prompt.
  • HLMSA (Hybrid LLM–Simulated Annealing): multiple trajectory setup with stochastic acceptance; LLM proposes neighbor moves and can propose cooling rates; acceptance uses classical annealing criterion.
• Key framework features: modular Optimizer base class, prompt management, batched candidate generation, parallel evaluation, solution parsing, callbacks (early stop, adaptive control), model-agnostic initialization, result tracking.
• Strengths: natural-language injection of constraints, domain heuristics and safety rules (no need to hand-code operators), accessibility to domain experts without deep optimization expertise.
• Limitations highlighted: not competitive with classical solvers for simple/structured problems; inference overhead and prompt length limits; stochastic and brittle behavior depending on model; feasibility often enforced by penalization rather than hard guarantees.
• Implementation and experiments: uses instruction-tuned LLM gemma-3-27b-it for many experiments; evaluated on convex optimization, linear programming, Traveling Salesman Problem, neural net hyperparameter tuning, and nuclear fuel lattice optimization (prior work showed parity or better vs a GA in that domain).

Data & Methods

• Formalization: optimization as min/max over X of f(x) where f is a black-box evaluator. Solutions are serialized as text/structured outputs and parsed back programmatically.
• In-context learning loop:
• Construct prompt Pt = Prompt(Problem description, optional domain knowledge, Ht).
• Sample B candidate solutions from the LLM: x(i) ∼ pθ(x | Pt).
• Evaluate each candidate externally: y = f(x).
• Append (x, y) to history Ht and update best solution.
• Repeat until budget or termination condition.
• Algorithms: paper provides Algorithm 1 (OPRO loop), Algorithm 2 (HLMEA), Algorithm 3 (HLMSA) pseudocode.
• Practical implementation details:
  • Truncated history in prompts to respect context window.
  • Batched generation + parallel objective evaluation to amortize inference and expensive simulations.
  • Callback system for adaptive sampling hyperparameters, early stopping, target-based termination.
  • LLM-selected hyperparameters (e.g., suggested mutation/crossover/mutation rates or annealing schedule) can be parsed from model outputs and fed into the loop.
• Experiments & benchmarks: case studies across multiple problem types to demonstrate generality; runtimes dominated by LLM inference; comparisons suggest classical solvers win on simple problems but LLMize is useful where constraints/heuristics are naturally expressed in language. Code: GitHub repository provided by the author.

Implications for AI Economics

• Cost vs. value trade-off
  • Compute cost: LLM-driven proposals incur substantial inference (token) cost relative to classical optimization iterations. For low-dimensional or convex problems, this makes LLM approaches economically inefficient versus specialized solvers.
  • Value when formalization is expensive: In domains where formal objective/constraint modelling is time-consuming or requires scarce expertise (e.g., engineering simulators, safety-critical design, proprietary heuristics), LLMize can shorten time-to-solution and reduce engineering labor costs by letting domain experts express rules in language.
• Labor and skill shifts
  • Demand increases for prompt engineering, dataset curation of exemplar solutions, and verification specialists rather than traditional metaheuristic engineering.
  • Potential re-allocation of optimization work toward domain experts who can craft natural-language constraints rather than optimization algorithm developers.
• Market and productization
  • Opportunity for optimization-as-a-service that packages LLM-driven workflows for black-box, domain-specific problems (e.g., design optimization, experimental parameter search).
  • Hybrid offerings are attractive: combine classical solvers where applicable and fall back to LLM-based search for unconstrained or poorly formalized subproblems.
• Risk, governance, and regulatory cost
  • In safety-critical applications (nuclear fuel lattice design example), LLM-based optimizers that express and rely on human-language constraints require additional verification and auditing costs; errors in prompts or misinterpretations can be costly.
  • Economic liability and insurance considerations can increase for firms relying on language-driven optimization in regulated industries.
• Long-run scaling and competitive dynamics
  • If LLMs continue to improve in reasoning and inference becomes cheaper, the economic case for LLM-based optimization will strengthen—especially for high-value black-box problems.
  • Proprietary LLM capabilities (better reasoning, longer context windows, lower-latency inference) could become a competitive advantage for firms offering optimization services; open frameworks like LLMize lower entry barriers and may commoditize part of that market.
• Practical guidance for AI economists and practitioners
  • Use LLMize when (a) problem constraints/heuristics are hard to formalize, (b) objective evaluations are expensive (so fewer but higher-quality proposals matter), and (c) expert time to encode constraints is costly.
  • Prefer classical/analytic solvers when problem structure (convexity, gradients, linearity) is available — they are cheaper and more reliable.
  • Account for extra costs: LLM inference, prompt engineering labor, validation/audit overhead, and reproducibility concerns due to model stochasticity and prompt truncation.
  • Consider hybrid deployments: use LLMize to generate candidate regions or heuristics, then refine with classical solvers where feasible to reduce overall cost and risk.

Summary takeaway: LLMize demonstrates a practical, general-purpose approach to bring language-grounded domain knowledge directly into black-box optimization workflows. Economically, it is most promising where human expertise is expensive to formalize; its adoption will hinge on LLM inference costs, validation/regulatory overhead, and improvements in model reliability and efficiency.