A lightweight streaming router uses a decoder-KV GLiClass to pick the best LLM per task, squeezing modest quality gains from cache-aware, zero-shot model scoring; on a 1,000-task subset it edges out the best fixed model (top-1 0.707 vs 0.696) and supports cost- and policy-aware endpoint selection.

Main Finding

SCX Router is a lightweight, streaming, zero-shot model-selection router that uses a causal decoder with a persistent text-only KV cache and a shallow bidirectional scorer to produce non-generative, per-label suitability scores. It enables per-request routing across heterogeneous inference endpoints (varying in cost, latency, context length, tools, domain expertise, safety/residency constraints) and—when coupled with a deterministic policy layer that applies eligibility and cost/latency/cache trade-offs—produces small but consistent end-to-end gains over fixed-model baselines in LiveBench-style evaluations (selected 1,000-task subset: top-1 = 0.707 vs 0.696 for the strongest fixed model). The model, code, and data tooling are released (Apache 2.0).

Key Points

• Architecture
  • A GLiClass-derived decoder-KV design built on a Qwen3 causal backbone (~0.6B parameters).
  • Decoder processes only new context tokens into a persistent KV cache; candidate labels are appended transiently, scored, and discarded (label KV not written back).
  • A shallow DeBERTa-v2-style bidirectional scorer over transient label suffix produces per-label logits via a shared MLP (non-generative classification).
  • Labels are text inputs (dynamic), enabling zero-shot scoring of arbitrary candidate endpoints and auxiliary labels (task type, difficulty, reasoning mode, output length, custom labels).
• Separation of concerns
  • Learned predictor (suitability scores and auxiliary signals) is kept separate from a deterministic decision/policy layer that enforces hard constraints (tools, modality, region/privacy/safety) and trades predicted performance against incremental token cost, latency, and cache reuse.
• Streaming & operational design
  • Persistent text-only KV cache amortizes request-side encoding across turns; triggers (chunk/delimiter/window) control when classification runs.
  • Implementation supports batched sessions with left-padding, TTL/LRU cache policies, optional CPU offload, and physical truncation with increasing rotary positions.
• Task ontology & synthetic data
  • Ontology: 23 families, 115 task types, 345 routable subtypes, 1,173 synthetic example specifications, and 30 domains (orthogonal axis).
  • Generates structured task packages (context, artifacts, checks) rather than bare prompts, to approximate real-world workflows.
  • Synthetic corpus: 150,000 verifier-scored tasks (deterministic/executable checks) and 15,000 open-ended tasks judged by gpt5.6-sol.
• Training & supervision
  • Mixed training: a broad 524k-record GLiClass mixture followed by a focused 65k-record routing mixture (22k routing labels, 17k difficulty, 15k task-type, plus other auxiliary labels).
  • Benchmark-derived supervision: existing benchmark prompts executed across candidate models; positive target set defined as models tied for best observed outcome (with tolerance ε). Missing evaluations are treated as masked (masked cross-entropy loss).
• Empirical results
  • Outperforms mean-candidate across six LiveBench subsets.
  • On a selected 1,000-task subset, aggregate top-1 = 0.707 vs 0.696 for the strongest fixed model (benchmark-dependent gains).
• Release
  • Checkpoint and code released under permissive Apache 2.0 (links provided in paper).

Data & Methods

• Model details
  • Backbone: Qwen3-0.6B causal decoder (28 layers, hidden size 1024, 16 query heads, 8 KV heads), trained up to 4,096 tokens.
  • Scorer: two-layer DeBERTa-v2 encoder (no separate embeddings), projections, shared MLP head (2h → 1024 → 512 → 1).
  • Special tokens: «LABEL», «SEP», «EXAMPLE».
• Sequence & scoring flow
  • Persistent context formatted as c≤t = format(prompt, examples, h<t, xt).
  • Label suffix q(L) = «SEP» ℓ1 «LABEL» … ℓK «LABEL» «SEP».
  • Persistent cache updated with new context tokens: (Hx_t, Kt, Vt) = Dθ(Δc_t; Kt−1, Vt−1).
  • Label tokens evaluated conditionally: Hℓ_t = Dθ(q(L); Kt, Vt) — label-stage keys/values not written back.
  • Bidirectional scorer Bϕ processes Hℓ_t → joint representations → per-label MLP gives logits at,k → sigmoid/softmax as appropriate.
• Training supervision and loss
  • Target set Yi per task: models with si,m ≥ max_j si,j − ε (ties allowed).
  • Masked binary cross-entropy for multi-label targets to handle missing observations: Lmasked = − (1 / Σ oi,m) Σ oi,m [yi,m log pi,m + (1 − yi,m) log(1 − pi,m)].
  • Thresholding default τ = 0.5 for multi-label emission; softmax/argmax for single-label tasks.
• Data sources
  • Benchmark-derived: prompts from many benchmarks (classification, QA, reasoning, code, summarization, translation, instruction-following, multi-turn) executed on candidate models; ~104–524k records in mixtures (broad + focused splits).
  • Synthetic: ontology-driven generation producing structured packages; 150k verifier-scored tasks and 15k judge-scored open-ended tasks.
• Evaluation protocol
  • Multiple LiveBench-like subsets and an 11-endpoint expanded collection with observation masks to avoid treating missing outcomes as negatives.
  • Router compared to mean-candidate and fixed-model baselines; also reports auxiliary signal accuracy (task type, difficulty, reasoning mode, expected output length).

Implications for AI Economics

• Per-task cost-quality optimization
  • SCX Router operationalizes per-request model selection, allowing platforms to push requests to cheaper endpoints when predicted quality is sufficient and reserve expensive endpoints for tasks that need them—directly improving cost-efficiency of ML inference supply.
  • By integrating predicted performance with incremental token cost, latency, and cache-reuse considerations, routers can materially alter marginal economics of serving multi-turn workflows (lowering average cost per user session).
• Market dynamics & productization
  • Dynamic, semantic label scoring (zero-shot) enables plugging in new endpoints quickly—supporting broader marketplaces of models and APIs without retraining the router—though empirical calibration still requires outcome data.
  • Router-as-a-service becomes feasible: value accrues to systems that can reliably improve end-to-end quality-per-dollar and respect operational constraints (privacy, residency, tool access).
• SLAs, pricing, and competition
  • Fine-grained routing supports differentiated SLAs (latency- or cost-sensitive routing), more accurate price discrimination, and richer billing models (e.g., premium routing guarantees).
  • Vendors will compete not only on peak model quality but on marginal quality/cost trade-offs that routers exploit, potentially reshaping endpoint pricing strategies.
• Operational externalities & design choices
  • Cache reuse reduces marginal encoding cost within sessions—this changes the effective switch-cost of moving between endpoints (routers must expose downstream cache state and account for replay costs).
  • Separation of learned predictor and deterministic policy supports governance (safety, sovereignty, legal constraints) and monetization decisions independently from the model’s semantics.
• Risks and limitations (economic relevance)
  • Calibration dependency: semantic zero-shot scoring allows immediate adoption of new endpoints but does not guarantee calibrated performance; empirical outcome data is necessary to avoid suboptimal economic choices.
  • Unequal coverage/missing data problems: marketplaces with uneven endpoint coverage complicate reliable routing and performance estimation; misrouting can impose hidden costs (retries, downstream replays).
  • Potential for gaming and incentive misalignment: providers might optimize for the classifier’s semantic labels rather than true downstream utility, or pricing schemes could be exploited without proportional quality improvements.
  • Complexity & maintenance costs: maintaining the router, ontologies, verifiers, and outcome collection pipelines adds operational overhead that must be weighed against expected cost-savings.
• Strategic opportunity
  • Firms operating multi-endpoint deployments (multi-cloud/multi-vendor inference) can extract short-term returns by deploying a lightweight streaming router like SCX Router to improve utilization and reduce average inference spend, while preserving safety and regulatory constraints via the policy layer.

References / artifacts (from paper) - Checkpoint: https://huggingface.co/scx-admin/scx-router-v0.1 - Code: https://github.com/Knowledgator/GLiClass-Model-Router

If you want, I can: - Extract quantitative tables (data counts, model specs) into a CSV for further analysis. - Produce a short decision-tree for when an organization should adopt a router like SCX from an economic ROI perspective.