A routing layer cuts ensemble LLM costs and delays: RouteMoA screens models with a cheap scorer and refines choices with lightweight judges, reducing inference cost by nearly 90% and latency by over 60% compared with conventional mixture-of-agents approaches.

Main Finding

RouteMoA introduces a lightweight, query-aware router for Mixture-of-Agents (MoA) LLM ensembles that performs pre-inference screening plus posterior correction (a “mixture of judges”) to dynamically select a small subset of models per layer. Compared to classical MoA and Sparse MoA, RouteMoA preserves or improves task accuracy while dramatically reducing inference cost and latency — e.g., in a 15-model pool it cuts cost by ≈89.8% and latency by ≈63.6% while improving average accuracy.

Key Points

• Motivation: MoA yields strong results by multi-model collaboration but is expensive because all candidate models are typically invoked each round (high token and compute cost, latency, context-size limits).
• Two-stage routing design:
  • Lightweight SLM-based scorer: computes coarse performance scores per model from the user query alone (no model inference), using an embedding similarity f(x, Mj) = σ(E(x)⊤ k_j).
  • Mixture of judges: refines scores without extra full-model inference by using (1) self-assessment scores output by active models and (2) selective cross-assessment (highest-scoring model evaluates prior outputs). Scores are normalized and averaged to yield posterior-aware rankings.
• Model ranking balances performance, token-output cost, token-input cost, and latency in priority order; supports early stopping.
• Scorer design and training:
  • Small encoder (mDeBERTaV3-base, ≈86M) produces query embeddings; each LLM has a learnable embedding.
  • Training uses dataset of queries + model responses + reward-model/ground-truth scoring, with dual contrastive losses (sample-LLM and sample-sample).
• Empirical results:
  • Large pool (15 LLMs): RouteMoA increases average accuracy vs MoA (e.g., overall avg 78.6% vs 71.3%), while reducing monetary cost from ~$447.8 to ~$45.6 (~89.8% reduction) and latency from ~248s to ~90s (~63.6% reduction).
  • Small pool (5 LLMs): RouteMoA achieves small but significant accuracy gains over MoA/SMoA (avg 83.1% vs MoA 80.9%), with major cost/latency savings.
  • Out-of-distribution (AGIEval-Gaokao): RouteMoA matches or modestly improves robust generalization compared to SMoA.
• Ablations confirm benefit of self- and cross-assessment for improving routing reliability.
• Practicality: intended for large heterogeneous model pools (including specialized models) and feasible given small scorer footprint.

Data & Methods

• Model pool: experiments use both a small 5-LLM pool (Gemma-2-9B-it, Mistral/Ministral 8B, Qwen variants, Bio-Medical-Llama) and a large-scale pool of 15 state-of-the-art LLMs (4B–235B, mixture of general-purpose and specialized).
• Evaluation benchmarks: 30 datasets spanning Language Understanding, Reading & QA, Logic Reasoning, Math Reasoning, Language Generation; plus focused 5-dataset evaluation (MATH, ARC-c, MBPP, RACE-high, MMLU-bio) and AGIEval-Gaokao OOD test.
• Scorer details:
  • Encoder: mDeBERTaV3-base (≈86M params), projected to 768-d embedding space; model embeddings kj ∈ R^768 learned.
  • Score function: s_j = σ(E(x)⊤ k_j).
  • Training dataset D constructed by collecting queries, each model’s response, and scoring responses via a mixture of ground-truth match and a reward model (InternLM-2-1.8B-Reward) with mixing parameter λ (used λ = 0.5).
  • Loss: dual contrastive loss L = L_sample-LLM + α L_sample-sample (α = 0.2), with k-means clustering (k=6) used for sample-sample positives.
  • Optimization: AdamW, lr = 5e-5, weight decay 0.01, batch size 64.
• Mixture-of-judges:
  • Self-assessment: active models output confidence scores for their answers.
  • Cross-assessment: highest-scoring active model evaluates outputs from previous round(s).
  • Final sl is formed by normalizing and averaging s1 (initial scorer) and posterior assessment signals.
• Model ranking:
  • Selects top-k models by priority: performance > output token cost > input token cost > latency.
  • Uses OpenRouter pricing/latency metadata; applies early stopping if max score > sth.
• Baselines: classical MoA, Sparse MoA (SMoA), routing baselines (RouteLLM, RouterDC), plus Oracle routing upper bound.

Implications for AI Economics

• Per-query cost reduction: RouteMoA shows large reductions in API/inference spending for multi-model ensembles. For firms deploying ensemble systems, routers like RouteMoA materially lower unit costs and can make multi-model approaches economically viable at scale.
• Efficiency enables larger model pools and specialization markets: by sharply cutting the cost of using many candidate models, RouteMoA allows suppliers and integrators to exploit more specialized models (math, code, biomedical) without linearly increasing expense. This supports a marketplace where specialized, potentially cheaper models can be profitably incorporated.
• Impacts on pricing and monetization:
  • API billing models and SLAs may be restructured: buyers will value low-cost, low-latency tiers combined with router intelligence; providers could offer router-aware pricing or partner bundles.
  • Value capture for small “router” SLMs: the SLM router itself becomes a high-leverage component—its marginal cost is tiny relative to LLM calls—so router models/services can become a new monetizable layer.
• Competitive effects: more widespread routing could reduce the premium on building monolithic, universally-capable LLMs; firms might instead differentiate via specialized models plus routing infrastructure.
• Resource and environmental externalities: fewer expensive full-model invocations reduces compute energy and carbon footprint per query, improving sustainability for deployed LLM services.
• Risks and strategic considerations:
  • Gaming and misrepresentation: providers might attempt to manipulate publicized latency/cost metrics or craft outputs that score well under popular routers/scorers. Routing systems will need robust, transparent inputs (pricing/latency) and defenses against adversarial manipulation.
  • Concentration & fairness: routers trained on historical performance could systematically prefer cheaper or dominant providers, concentrating demand; careful design/regularization may be needed to preserve market competition and domain coverage.
  • Upfront training/integration costs: building and maintaining scorer datasets (responses, ground-truth, reward-model signals) requires investment; smaller firms may rely on third-party router providers.
• Policy/regulatory notes: transparency in selection criteria and auditability (why a model was/was not selected) could become important for regulated domains (healthcare, finance), suggesting router designs should support explainability and traceability.

Overall, RouteMoA demonstrates a practical, high-leverage approach to reduce inference costs and latency for multi-model LLM systems, which has direct and substantial implications for the economics of deploying specialized model ensembles and for marketplace architectures around model provision and routing services.