Brick, a multimodal router that scores models on six capability dimensions and estimates per-query difficulty, slightly outperforms the best single model (76.98% vs 75.02% accuracy) while letting operators trade quality for cost. At a neutral setting it preserves 74.11% accuracy at 4.71x lower cost and can cut costs up to 22.15x (with accuracy loss) while halving median latency.

Main Finding

Brick is a cost-aware, capability-space router for the Mixture-of-Models (MoM) paradigm that dispatches each query to the cheapest model likely to answer it correctly. On a 3‑model heterogeneous pool (qwen3.5-9b, deepseek-v4-flash, kimi2.6) and a 5,504-query benchmark (Dataset A), Brick traces a Pareto frontier of accuracy vs cost that dominates several public routing baselines. At its max-quality operating point Brick achieves higher accuracy (76.98%) than the best single model (always‑kimi, 75.02%) while lowering per-call cost by ~28% and halving median end-to-end latency.

Key Points

• Problem framing: routing whole pretrained models (Mixture-of-Models, MoM) — not MoE — so routing decisions are per model per query. The router’s goal is "cheapest model that will solve this query."
• Superficial routing (domain labels, token counts, regexes) fails because within‑domain difficulty varies widely.
• Capability-space idea: represent both queries and models as 6‑dimensional capability vectors; route by geometric covering (distance) plus a cost penalty. This captures multi‑dimensional strengths/weaknesses (e.g., code vs planning).
• Cost-awareness and preference knob: Brick adds a cost penalty term and exposes a continuous user-facing knob r ∈ [−1, 1] to trade accuracy vs spend at deploy time (no code change).
• Model pool and pricing used in experiments:
  • Models: qwen3.5-9b (cheap), deepseek-v4-flash (mid), kimi2.6 (expensive/frontier).
  • Per-call empirical mean costs on Dataset A: qwen $0.001386, ds4 $0.002895, kimi $0.030703.
  • Design-time cost scalars used in routing: cqwen=0.10, cds4=0.40, ckimi=0.60.
• Dataset A (brick2-dataset-a-eval): N=5,504 queries across six capability dimensions and ten protocols; mixed grading (unit tests, programmatic checks, and LLM judges). Three-model oracle accuracy = 83.25% (922 queries unsolvable by any model).
• Empirical results vs baselines (same evaluation protocol, zero-shot external routers):
  • Brick max-quality: 76.98% accuracy at $0.022083 per call. Beats always-kimi (75.02% at $0.030703) on both accuracy and cost.
  • Brick neutral: 74.11% at $0.006513 — 4.71× cheaper than always-kimi for only −0.91 pp accuracy.
  • Brick low: 71.62% at $0.003557 — outperforms FrugalGPT (69.42%) by +2.20 pp at comparable cost.
  • Brick min-cost: cost reduced 22.15× vs always-kimi, at 11.85 pp accuracy loss.
  • Median end-to-end latency reduced from 51.2 s (baseline behavior) to 22.8 s under Brick routing.
  • Brick recovers ~24% of the headroom between the best single model and the three-model oracle; remaining gap indicates information the router does not yet extract.
• Baselines examined: always‑model baselines (qwen, ds4, kimi), RouteLLM (pairwise/tournament single-step), FrugalGPT (cascade with DistilBERT verifier), Cascade Routing (probabilistic utility-scored cascade). Cascade approaches incur structural overheads for agentic (multi-call) workloads because rejected stages re-run on every agent step.

Data & Methods

• Dataset A composition: 14 public sources + curated subsets, 10 scoring protocols (coding unit-tests, math (SymPy/regex), IFEval/IFBench, MCQ extraction, tool-call AST/state checks, LLM judges for open-ended/factual), mixed deterministic and single-LLM-judge regimes; one protocol used a 3-judge panel for variance characterization.
• Evaluation metric: selected-answer accuracy — routed query is correct iff the dispatched model passes the dataset grader for that query. No router-specific re-labeling; unsolvable queries count as incorrect for all routers.
• Model pool: three whole models (listed above). Costs reported as empirical per-call averages under observed token distributions and list prices; routing math uses calibrated, dimensionless cost scalars fixed at calibration time.
• Brick internals (method sketch):
  • Calibrate a 6‑dim capability vector per model and a mapping from query to capability-vector (multi‑modal inputs).
  • Compute a geometric distance Dm between the query vector and each model vector.
  • Add a cost-penalty term β · cm (cm is design-time cost scalar for model m) to form a combined routing score.
  • Dispatch to argmin_m (Dm + cost penalty scaled by operator preference r).
  • Preference knob r slides global behavior toward max-quality (favor accuracy) or min-cost (favor cheaper models).
• Baselines: external routers used public checkpoints in zero-shot; cascades account for cumulative billing of rejected stages when reporting per-query cost. Agentic cost models consider N-step trajectories where cascade overhead multiplies across steps.

Implications for AI Economics

• Direct cloud-bill lever: Even modest per-request savings compound at production scale (millions of calls). Brick demonstrates concrete per-call cost reductions (e.g., ~28% lower cost at higher accuracy than the pricier single-model baseline), implying substantial operational savings.
• Open-weight vs frontier models: MoM + Brick enables financially efficient blends of cheap open-weight models for the long tail of easy queries and expensive frontier models for hard ones, reducing demand for always-on frontier API calls and shifting spending toward selective use.
• Deploy-time tuning of cost-quality trade-offs: The r knob turns a once‑static accuracy/cost trade-off into an operational control. Firms can tune r to meet budget/SLAs dynamically without code changes, supporting flexible pricing/SLO strategies.
• Agentic systems: For multi-step agentic workloads, cascades are structurally costly because rejected stages repeat on each step. Pure single-step routers like Brick avoid per-step cascade waste, lowering both bill and latency for agentic applications — important for products that call models frequently within a session.
• Market and product effects: If routers like Brick become standard, cheaper models can capture the majority of routine traffic, reducing marginal demand for high-priced frontier APIs. This could pressure pricing and change vendor competition dynamics (value shifts to selective high-quality responses rather than bulk availability).
• Operational costs and limitations:
  • Brick requires a labeled routing-evaluation set and calibration (collecting per-query correctness per model), which is an upfront evaluation cost. Its performance depends on the quality and representativeness of that calibration data.
  • The current prototype recovers a fraction of the oracle headroom; further gains need richer query features or better capability extraction.
  • Cost scalars (cm) are locked at calibration time; dynamic price changes would require recalibration or careful design.
• Policy/strategy: operators can use MoM + capability routing to optimize cloud spend, latency, and model-use policies (data locality, regulatory constraints), while maintaining a measurable accuracy budget; this architectural choice can materially affect unit economics of LLM products.

If you want, I can: - Extract the exact operating points and dispatch shares per-profile (qwen/ds4/kimi) reported in the paper. - Produce a short checklist for deploying Brick in a production stack (data collection, calibration, monitoring, re-calibration triggers).