A lightweight multi-LLM strategy halves compilation time and quarters API bills: LiteCoOp turns the search tree into a collaboration channel so smaller models do most work, invoking the largest model only about 23% of calls while still improving optimization quality across GPU and CPU benchmarks.

Main Finding

LITECOOP is a lightweight multi-LLM framework for LLM-guided compiler optimization that uses a shared MCTS search tree as the collaboration substrate. By making model selection an endogenous part of the tree search (each LLM proposes a transformation and the next LLM), LITECOOP enables heterogeneous LLMs to cooperatively optimize compiler transformations while strongly biasing toward smaller/cheaper models. On diverse GPU and CPU benchmarks, an 8-LLM LITECOOP configuration (heterogeneous set including GPT-5.2 as the largest model) outperforms single-large-LLM baselines while substantially reducing compilation time and API cost: - Average speedup over pre-optimized code: 30.1× (GPU) and 10.9× (CPU) across evaluated kernels. - Relative to a single large model baseline, LITECOOP(8) reduces total compilation time by 1.95× (GPU) and 1.74× (CPU). - API cost reduced by 4.47× (GPU) and 4.32× (CPU). - The largest model is invoked for only ~23% of total calls (23.1% GPU, 23.9% CPU).

Key Points

• Collaboration mechanism: A single shared MCTS tree where each node is a joint state (program, active LLM). On expansion, the active LLM proposes both a compiler transformation (or short sequence) and the next LLM to expand the child node. All proposals live in the same tree and rewards are backpropagated through it, so different models build on common prefixes and receive credit for downstream program quality.
• Cost-aware tree policy (LA-UCT): An LLM-aware UCT variant that scores child nodes by a surrogate combining downstream reward and a normalized preference for smaller models:
  • surrogate mean eµ = (1 − λ)µ + λ · ϕsmall(llm), with λ controlling preference strength (paper uses λ = 0.5).
  • ϕsmall(llm) is a normalized function of model parameter count (log scale), giving higher score to smaller models.
• Contextual prompting: Expansion prompts include the current program (and recent ancestors), transformation history, predicted performance, and per-LLM statistics (size, invocation counts, hit/error rates). LLMs are instructed to prefer the smallest model likely to make progress.
• Course-alteration (safety valve): If persistent regressions attributable to small models are detected (two regressions along a path from small-model expansions), the last regressive child is pruned and the largest LLM is invoked with a short targeted prompt to revise the proposal. This reduces propagation of bad small-model mistakes while keeping large-model interventions limited and cheaper than full-size calls.
• Empirical result trends: Increasing the number and diversity of collaborating LLMs (2 → 4 → 8) improves optimization outcomes and cost-efficiency; the best balance observed at 8 heterogeneous LLMs in these experiments.

Data & Methods

• Benchmarks: five representative neural-kernel workloads (Llama-3-8B self-attention, DeepSeek-R1 MoE layer, FLUX attention and convolution layers, Llama-4-Scout MLP), plus end-to-end Llama-3-8B compilation.
• Target hardware: NVIDIA 2080 Ti GPU and Intel Core i9 CPU.
• LLM sets:
  • 2-LLM: GPT-5.2 (largest) + gpt-5-mini (small).
  • 4-LLM: add DeepSeek-R1-Distill-Qwen-32B and Llama-3.1-8B-Instruct.
  • 8-LLM: further add DeepSeek-R1-Distill-Qwen-7B, Qwen3-8B, Qwen3-14B, Devstral-Small-2505.
• Implementation: Built on TVM MetaSchedule (Apache TVM v0.20.0); replaces default search with LITECOOP. OpenAI and Nscale APIs used for model serving.
• Cost model and rollout: Short random rollouts; terminal programs evaluated by TVM’s hardware-agnostic cost model (XGBoost-based) to avoid executing on target hardware every step.
• MCTS settings reported: λ = 0.5, exploration c = sqrt(2), branching factor B = 2. Each experiment repeated 10 times; means reported (confidence intervals and ablations in appendices).
• Metrics:
  • Speedup = latency(original unoptimized IRModule) / latency(optimized IRModule).
  • Compilation time and API-call cost measured vs single-large-LLM baseline.
  • Invocation rates: percent of total LLM calls per model (regular + course-alteration calls separated).

Implications for AI Economics

• OPEX reduction for model-serving compiler workflows: By diverting most calls to smaller, cheaper LLMs while keeping optimization quality high, LITECOOP materially reduces API cost and total compilation time. This directly lowers operational expenses for organizations that run compiler searches (e.g., model-serving platforms, ML systems teams).
• Demand-side effects on LLM markets: Techniques like LITECOOP increase value of a mixed-model ecosystem (cheaper smaller models plus occasional large-model interventions) rather than single-model dominance. Providers of smaller/medium models could capture more invocation volume even if per-call revenue is lower, while demand for very large models becomes more targeted (critical interventions) rather than continuous heavy use.
• Capacity planning and infrastructure: Reduced reliance on continuous large-model serving can smooth load, lower peak provisioning needs for high-capacity models, and reduce expenditure on large-model inference infrastructure (GPU/TPU-backed endpoints). Conversely, total call volume may increase as more smaller-model calls are used—this shifts cost from per-call price to aggregate throughput and latency tradeoffs.
• Democratization and competitive dynamics: Smaller organizations or teams with limited budgets can feasibly use multi-LLM search to approach or match performance of optimization guided by top-tier LLMs, reducing barriers to deploying hardware-optimized models and potentially broadening participation in ML system tuning.
• Limitations and caveats that affect economic interpretation:
  • Use of an XGBoost hardware-agnostic cost model for rollouts speeds experiments but may misestimate some hardware-specific tradeoffs; real-world gains and cost savings depend on how well the cost model reflects actual target-device performance.
  • Empirical results are on a set of kernels and two hardware platforms; generalization to other compilers, larger program spaces, or very different hardware needs further validation.
  • The framework still requires occasional large-model calls (course-alteration and ~23% of calls in the 8-LLM setup); cost and availability of these models remain a factor.
  • Market effects depend on LLM pricing models (per-token vs per-call vs subscription) and latency/SLA requirements—these details determine the realized economic gains.
• Overall: LITECOOP demonstrates a practical approach to cutting AI operational costs for a concrete, high-value ML-systems task by exploiting model heterogeneity and algorithmic (search) coordination rather than expensive constant use of top-tier LLMs. This pattern—optimizing when and how to use the largest models—could generalize to other production AI workflows and shift supplier/provider economics toward more diversified model ecosystems.