A new router, PROTEUS, lets operators set accuracy targets at runtime and reliably meets them, achieving near-oracle accuracy across benchmarks while cutting inference costs by as much as ~90% versus fixed-model deployments; its learned dual-variable approach yields consistent floor compliance and high target-response correlation.

Main Finding

PROTEUS is a learned LLM router that accepts accuracy targets τ at runtime and reliably enforces them. Using a τ‑conditioned policy trained with learned Lagrangian (dual) control, a single trained model serves a wide accuracy spectrum (τ ∈ [0.85, 0.95]) without retraining and achieves strong cost/accuracy tradeoffs: near‑oracle accuracy (RouterBench: 90.1% vs oracle 91.4; SPROUT: 94.0% vs oracle 98.6), 100% floor compliance (acc ≥ τ) on the benchmarks, very high τ→µ correlation (0.97–0.98), and large cost reductions (up to 89.8% vs best fixed model).

Key Points

• Operational problem: operators want to specify accuracy SLOs per request (τ) directly; existing routers require offline parameter tuning and do not reliably translate settings to realized accuracy.
• Core idea: condition the routing policy on τ and enforce the constraint during training via a learned Lagrangian dual variable λ that is injected into the policy. During training λ tracks batch-level accuracy shortfalls and teaches the network how τ maps to a continuous quality preference µ.
• Continuous quality preference µ ∈ [0,1] (sampled from a Beta policy) decouples “how much to prioritize quality” from model selection. Model selection uses a scoring function: si = pi(x) + µ · bi − (1−µ)^γ · ci where pi(x) is predicted per-model correctness, bi is a per-model quality boost, ci is normalized cost, and γ (learnable, bounded) controls cost sensitivity.
• Training uses primal‑dual RL (PPO) with session-based τ sampling and fast dual updates (ηλ ≈ 0.4) so the policy learns a stable τ→µ mapping. At inference λ is fixed and τ is the runtime input.
• Empirical performance:
  • τ–µ Pearson correlation: 0.973 (RouterBench), 0.981 (SPROUT).
  • Floor compliance (acc ≥ τ): 100% on both benchmarks.
  • Close to oracle accuracy: within 1.3% (RouterBench) and 4.6% (SPROUT).
  • Cost savings: up to 89.8% vs best fixed (always‑best) model.
  • Baseline comparison: OmniRouter (another Lagrangian baseline) met floor only 22% of the time on RouterBench; PROTEUS is far more reliable.
• Architectural choices that matter: DeBERTa‑v3‑small encoder (fast, ≈2ms), Beta output for µ (learnable concentration for exploration), separate performance head for pi(x) (prevents τ from biasing correctness predictions), and learnable non-linear cost weighting γ.

Data & Methods

• Benchmarks:
  • RouterBench: 405K labeled outcomes across 11 models (open and proprietary), tasks include reasoning (MMLU), math (GSM8K, MATH), coding (HumanEval); costs $0.0001–$0.01/query. Oracle accuracy 91.4%. Evaluated τ ∈ [0.85, 0.91].
  • SPROUT: 45K queries across 14 models (including frontier systems like GPT‑4o), instruction‑following queries with LLM evaluation; costs $0.0001–$0.05/query. Oracle accuracy 98.6%. Evaluated τ ∈ [0.85, 0.95].
• Model & training details:
  • Query encoder: DeBERTa‑v3‑small → 256‑dim embedding; two heads: performance prediction pi(x) (one sigmoid per model) and τ‑conditioned policy.
  • Policy: 2‑layer MLP mapping (embedding, τ, λ) → Beta(α,β) parameters → µ sample. µ is used with scoring function to choose model.
  • Reward: Lagrangian formulation combining quality weight, cost weight, and λ · (pm∗ − τ). Costs normalized online.
  • Optimization: PPO (policy lr 3e‑4), batch size 32, 10k steps (~4 hours/dataset on one A100), dual updates every 5 batches, dual LR ηλ=0.4.
• Metrics:
  • Standard routing metrics: accuracy, cost, oracle gap.
  • Adaptability metrics: τ–µ correlation, SLA floor compliance, ±2%/±5% tolerance bands.
  • Efficiency metrics: Routing Efficiency (accuracy gain per ms of routing latency) and Routing Performance Index (RPI) combining accuracy/oracle gap, normalized cost, and router latency.
• Ablations and baselines: compared to static strategies (Always‑Best, Always‑Cheapest), learned predictors (KNN, MLP), CARROT, RouteLLM style, OmniRouter (Lagrangian baseline). Ablations include no τ conditioning, fixed γ, and no critic.

Implications for AI Economics

• Direct monetization and tiering: PROTEUS enables platforms to offer explicit accuracy‑bounded service tiers (e.g., economy/standard/premium) that are enforceable at runtime. Operators can price tiers by reliably trading off cost vs. delivered accuracy, enabling more predictable SLAs and product differentiation.
• Large cost reductions per unit of delivered accuracy: up to ~90% savings relative to always‑using the highest‑accuracy model implies substantial reductions in marginal serving cost, improving margins and enabling lower price points or higher profit per query. This affects providers' pricing strategies and cost pass‑through to customers.
• Efficient capacity allocation and multi‑tenant economics: SLA‑aware per‑query routing lets providers allocate expensive models selectively to queries/customers that need them, increasing utilization efficiency across tenants and reducing overall compute provisioning requirements (lower peak capacity).
• Dynamic pricing and surge management: because the router accepts τ at runtime, platforms can combine dynamic pricing with explicit accuracy guarantees (e.g., discounted low‑accuracy slots). During load spikes, operators can enforce lower‑τ tiers to protect capacity, or raise prices for guaranteed high‑τ slots.
• Competitive dynamics for model vendors: improved routers increase the value of cheaper models (when they can meet a requested τ on many queries), potentially shifting demand toward more cost‑efficient models and changing vendor bargaining power. Conversely, frontier models retain value for high‑τ or hard queries.
• Contracting and risk allocation: PROTEUS makes accuracy SLOs operationally enforceable, reducing ambiguity in contracts. That can lower dispute costs but also raises the need to precisely define measurement methods, evaluation sets, and treatment for distribution shift or adversarial inputs.
• Environmental and externality impact: large compute/cost savings imply lower energy usage per delivered accuracy, reducing emissions for inference workloads — a measurable externality relevant to sustainability accounting and carbon pricing.
• Implementation and operational caveats that affect economics:
  • Training and maintenance costs: the approach requires training the τ‑conditioned policy and periodically re‑training as model pools change or distributions shift; those overheads must be amortized against serving savings.
  • Dependence on good performance predictors: the router relies on accurate pi(x) predictions; miscalibration can create SLA violations or overprovisioning. Monitoring and recalibration costs are economic factors.
  • Model pool dynamics: when models are added/removed or costs change, the router may need retraining or fine‑tuning; frequent model churn raises operational costs.
  • Risk of gaming and metric design: tenants or downstream agents could attempt to manipulate measured accuracy (e.g., adversarial queries, label ambiguity), necessitating careful metric design and auditing, which incur governance costs.
• Policy & regulatory considerations: PROTEUS can help enforce accuracy guarantees in regulated domains (finance, healthcare) but also necessitates transparent reporting of guarantees and failure modes; regulators may require auditable records of routing decisions, which adds compliance costs.

Limitations & open questions (economic relevance) - Benchmarks supply per‑query correctness labels for many models; in live systems accurate labels are harder to obtain, affecting calibration and long‑run enforcement. - Robustness to distribution shift, adversarial inputs, and model‑pool churn needs further study — economic value depends on reliability under real‑world dynamics. - The abstract assumes cost per query is known and stable; in spot or metered settings (e.g., variable cloud pricing) economics become more complex. - Welfare and fairness: differential routing by tier could disproportionately affect outcomes for lower‑paying users in social or public‑good contexts; pricing/ethical tradeoffs matter.

Bottom line PROTEUS provides a practical, learned mechanism to translate operator‑specified accuracy SLOs into per‑query routing that meets those SLOs while dramatically reducing cost. That capability unlocks more precise product tiering, better capacity utilization, and large cost savings — but realizing those economic benefits requires investing in monitoring, calibration, and governance to manage risks from distribution shift, metric gaming, and model‑pool changes.