A lightweight two-map ‘glance-and-verdict’ lets a frozen LLM pick the right tool without crowding its prompt, improving skill-selection accuracy substantially across benchmarks; on Qwen3-32B Gavel beats heavyweight external retrievers and even lets a 32B model out-select much larger frontier models in skill-use tests.

Main Finding

Gavel (Glance And Verdict from a frozen LLM) is a lightweight skill-routing algorithm that extracts routing signals directly from a frozen agent LLM’s forward passes using only two trained linear projections (7.9M parameters). It ranks an entire skill library with a single cheap “glance” (token-level late-interaction readout into per-skill key banks built at installation) and then issues a small number of full forward-pass “verdicts” over the glance shortlist. With Qwen3-32B as the frozen backbone, Gavel outperforms progressive-disclosure and retrieve-and-rerank baselines (which add 1.2B–16B external parameters) across multiple benchmarks, including a new mid-rollout benchmark (SkillTraj), with gains up to 13.4 points on written-task routing and up to 21.9 points when needs arise mid-rollout.

Key Points

• Architecture and stages
  • Glance: grafts two linear maps (Wq, Ws) onto a chosen mid-layer (ℓ*) of the frozen LLM to produce query vectors for task tokens and key vectors for skill tokens. Each installed skill is rendered once, run through the frozen model, and its per-token keys are cached (a “bank”). Token-level votes use max-similarity; per-token top-k voting prevents dilution by many weak tokens.
  • Verdict: for a small shortlist from the glance, the frozen LLM resumes full attention with the skill as a prefix and the task appended. Two native signals are read from one resumed forward pass: (i) mean task log-likelihood L(s|x) and (ii) the model’s yes/no log-odds V(s|x) to a fixed question u (“does this skill provide what the task needs?”).
  • Ruling: scores are fused as S(s|x) = g(s|x) + α L(s|x) + γ V(s|x), treated as a product-of-experts over three estimates of log p(s|x).
• Installation-time indexing and compression
  • Each skill costs exactly one forward pass at installation to build its key bank — no retraining of the backbone and no per-skill training runs.
  • Per-skill banks are compressed via an ε-cover (farthest-first traversal). Proposition: compressing keys to an ε-cover lowers max similarity scores by at most ε (i.e., bounded distortion). Compression shrank banks ~8.5× in experiments at <1.6-point cost on benchmarks.
• Training and compute budget
  • Only Wq and Ws are trained (7.9M params). Training used 51,104 SkillRet queries (synthetic) at temperature τ=40. Gradients are stopped at hidden states; backbone remains frozen.
  • Hyperparameters reported: ε = 0.83 (cover radius), α = 1.0, γ = 0.025, pruning margin Δ = 0.133; shortlist size averaged ≈9 candidates; per-token top-k set to min(10, max(3, round(0.1 N))).
• Empirical results
  • Backbones: main instantiation uses Qwen3-32B (read-out layer ℓ* ≈ block 45 of 64).
  • Benchmarks: SkillRet (4,997 queries, 6,660 skills), SRA-Bench (861 sampled tasks, 26,262 skills), Eval-Core (75 queries vs ≈78K documents), and SkillTraj (new, 372 simulated agent trajectories capturing mid-rollout contexts).
  • Gains: Gavel’s full pipeline beats strongest retrieve-and-rerank pipelines by 3.8 pts (SkillRet), 13.4 pts (SRA-Bench), and 1.3–2.7 pts (Eval-Core). On SkillTraj scenarios, improvements range up to 21.9 points.
  • Glance alone is already a strong retrieval stage; the verdict adds the crucial re-examination that recovers subtler semantic judgments.
• Practical properties
  • No skill text enters the live agent context until a skill is selected (addresses “context crowding” of progressive disclosure).
  • No standalone retrieval model required; selection capability scales with improvements to the backbone (positive transfer).
  • Works zero-shot across libraries after training Wq/Ws once.

Data & Methods

• Data used for training and evaluation
  • Training: 51,104 queries from SkillRet (synthetic queries written by Qwen3.5 over 9,084 public skills).
  • Evaluation benchmarks:
  • SkillRet test v1: 4,997 Claude-written queries vs 6,660 held-out skills.
  • SRA-Bench: 861 sampled tasks (from 6 reasoning/coding sources) vs 26,262 skills.
  • Eval-Core: 75 real task.md queries vs 78K documents (two pools).
  • SkillTraj: newly introduced 372 simulated agent trajectories to assess routing when need arises mid-rollout (four noisy multi-turn scenarios).
  • Skill-Use harness: minimal bash-agent experiments comparing Qwen3-32B to larger open frontier models (Claude Code, Codex) on triggering correct skills.
• Method details
  • Glance readout
  • Choose ℓ* at model’s “compression valley” (empirically ~70% through layers) where hidden-state matrix entropy bottoms out.
  • At installation, render each skill r(s), forward-pass to get h_{ℓ*}(r(s)); Ws maps token states to unit-norm keys; farthest-first ε-cover keeps representative keys only.
  • At routing: Wq maps live task token states h_{ℓ*}(x)_i to unit queries q_i; per-token score m_i(s) = max_j ⟨q_i, d^s_j⟩; contribute only to token top-k candidates; aggregate into g(s|x) (weighted token sum normalized to training scale).
  • Contrastive training (InfoNCE-like) trains Wq and Ws with multi-positive contrastive loss; later Wq fine-tuned against compressed banks while Ws & keys frozen.
  • Verdict readout
  • For shortlisted skills, reuse rendered prefix r(s), append task x, run causal forward pass: compute mean per-token log-likelihood L(s|x) and append fixed question u to compute yes/no log-odds V(s|x).
  • Fusion and selection
  • Combine g, L, V linearly (coefficients α, γ) and pick argmax among shortlist. Parameters calibrated on SkillRet validation; pipeline is zero-shot thereafter.
• Theoretical result
  • Proposition: compressing a unit-norm key bank to an ε-cover guarantees the max inner-product against any unit query decreases by at most ε (upper bound unchanged). This controls distortion from compression.

Implications for AI Economics

• Reduced operational cost for routing infrastructure
  • Eliminates the need for large external retrieval/reranking stacks (1.2B–16B extra parameters in baselines), lowering serving costs, system complexity, and engineering maintenance. Providers can avoid provisioning additional heavyweight models just for routing.
  • Per-skill marginal cost becomes one installation forward pass (compute) and storage of a small compressed key bank, instead of retraining or serving large rerankers. This materially lowers the marginal cost of adding skills and supports very large and dynamic skill libraries.
• Better scaling and product value
  • Since routing quality improves as the backbone improves, vendors who control the agent LLM capture increasing value: improving the core model directly improves routing without separate retriever upgrades. This creates an economy of scale and potentially stronger vendor lock-in for backbone providers.
  • Removing context crowding allows larger skill menus without degrading core task performance, increasing the practical utility of vast public skill repositories and enabling richer third-party marketplaces for skills.
• Market for auxiliary models and services
  • Demand for specialized embedding/ran­ker models may decrease when agent backbones can be used directly for retrieval via cheap glance+verdict methods; this could shift investment from standalone retrievers to tooling that supports installation-time rendering, storage, and secured key caches.
  • Startups offering large reranker-as-a-service may face competitive pressure; conversely, new services may arise around secure storage, compression, and privacy-preserving key representations.
• Latency, deployment, and UX tradeoffs
  • Gavel’s two-stage design gives a predictable tradeoff: cheap global scan (one sweep of small linear head over cached banks) plus a small number of resumed forward passes per routing event. This reduces average latency and compute versus performing full attention over many skills or preloading metadata into the context.
  • For real-time/edge deployments, the per-skill installation pass cost may be amortized easily; but initial onboarding of very large libraries still requires batch compute.
• IP, privacy, and governance considerations
  • Skills are represented by compressed key banks derived from rendered prompts rather than uploading raw SKILL.md into the live context. This reduces exposure in the live context, but the cached keys still encode semantic content—raising questions about intellectual property (IP) leakage, reverse-engineerability, and privacy. Economic contracts and technical mitigations (encryption, access controls, differential privacy) may be required for marketplaces.
• Incentives for open vs closed ecosystems
  • Because routing benefits from improvements to the backbone, major backbone providers may gain disproportionate downstream value (control over skill-routing performance and combined capabilities), affecting competition dynamics in skill marketplaces. Smaller integrators may prefer open backbones with high-quality readouts to avoid paying for external retrievers.
• Labor and tooling impacts
  • Lower marginal costs to add and maintain skills could increase demand for skill authorship and curation (marketplaces, OSS contributions), shifting labor toward producing high-quality SKILL.md artifacts and well-structured skills.
• Limitations and further economic unknowns
  • Gavel relies on the frozen backbone having discoverable routing signals—performance depends on backbone architecture and pretraining/fine-tuning history. If firms vertically integrate backbone and skills, competitive advantages may consolidate.
  • The paper does not quantify absolute compute/latency dollar savings; adopters should model per-skill installation compute, storage costs for compressed banks, and inference costs of shortlist verdict passes against their current reranking stack to estimate ROI.

Overall, Gavel shows a practical pathway to lower-cost, backbone-aligned skill routing that scales to large dynamic skill libraries and shifts the economics away from heavy external retrievers toward leveraging the core agent model.