Specialist ‘judgelets’ trained from scratch underperform a shared judge: splitting adapter training across rubric families cuts accuracy and safe coverage unless adapters are initialized from a shared trained judge; separately, simple learned risk-routing lets small-to-large cascades beat a single large evaluator in accuracy at lower normalized compute.

Main Finding

Specializing small evaluators by splitting judgment into many independently trained adapters (judgelets) hurts performance and audited safe-coverage when training data per rubric family is limited; by contrast, specialization that only gates or defers examples (learned deferral / risklets) can reduce compute and improve overall accuracy. Practical rule: share judgment training across rubrics until there is enough data to justify per-rubric adapters, and put domain-specific adaptation inside an auditable release/deferral boundary.

Key Points

• Dataset and scope
  • 99,952 rubric-conditioned direct-assessment examples (996 criteria) from a public Feedback Collection; held-criterion protocol reserves 206 criteria / 20,696 examples for criterion-shift tests.
  • RewardBench 2 experiments use 1,763 non-tie tasks (8,977 candidate completions) for cascade correctness-routing.
• Architectures compared
  • Monolithic evaluator: single LoRA adapter on a frozen 0.6B Qwen3 base (rank-8 and rank-64 variants).
  • Judgelet bank: K criterion-family LoRA adapters (K ∈ {1,4,8}) on same shared base; only the router-selected adapter is active per example. Judgelet size: rank-8 LoRA, 5.05M trainable params each (≈0.84% of instantiated evaluator).
  • Deferral/routing: learned logistic “risklet” probes (small logistic heads) to decide whether to accept a stage’s ranking or forward to a larger model; cascades tested across 0.6B–1.7B–4B–8B reward-model family.
  • Router: TF–IDF (word + bigram) features + K-means clustering to assign rubrics to families (inspectable, deterministic).
• Main numerical results (direct-assessment)
  • Providing the correct rubric to a monolithic evaluator improves locked-test accuracy by +2.11 percentage points vs response-only control (paired bootstrap 95% interval +1.48 to +2.75).
  • Replacing the rubric with an unrelated rubric reduces locked-test accuracy by −2.66 points (paired 95% interval −3.26 to −2.09).
  • Fragmenting training across eight independently trained rank-8 judgelets (same total corpus split across families) causes a large performance drop: −10.05 points relative to the monolith and dramatically reduces audited coverage (at a 5% risk target) from 24.44% to 5.43% (reported in abstract).
  • Matching stored capacity with a single rank-64 adapter does not reproduce the judgelet loss — the loss is statistical, not purely a capacity accounting effect.
  • Initializing family adapters from a shared, trained monolithic judge (warm-start) largely recovers performance: test accuracy reaches 76.85%, which is +19.94 points above scratch training at the same learning rate (95% interval 18.88–21.02).
• Risk calibration and coverage
  • Global confidence thresholds overstate safe coverage (e.g., global threshold accepted 37.47% but exceeded target risk). Per-family thresholds and Bonferroni-adjusted family policies give tighter audited bounds (Bonferroni: 24.44% coverage with a 3.54% one-sided 95% Wilson upper bound on risk).
  • Rubric corruption (wrong rubric text) can both lower accuracy and invalidate calibration — a policy audited with correct rubric text is not safe under corrupted or missing rubric text.
• Cascade / deferral results (RewardBench 2)
  • Learned risklets routing a 0.6B–4B–8B cascade achieved mean accuracy 89.40% vs 84.75% for 8B-only, at 0.415 normalized parameter compute (i.e., substantial compute savings and higher accuracy).
  • Across 20 deterministic repartitions, every run passed an exact one-sided 95% Clopper–Pearson risk audit for the learned-risklet cascade.
  • Margin-based baselines remained near ~84.8% accuracy while using at least 0.94 normalized compute (so they offered little compute savings without accuracy gain).
  • Gains arise from complementary errors: early exits rescued 5.32% of tasks and harmed 0.66% (net +4.66 points). An oracle over all sizes gives ≈91.80% accuracy; a restricted oracle for the deployed subset gives 90.97%.
  • Signal for routing in RewardBench largely comes from multivariate score geometry (margins, gaps, ranges) rather than skill labels (removing the one-hot skill changed accuracy only by 0.03 points).
• Additional operational findings
  • The response-only baseline is a strong shortcut; much of the rubric benefit disappears when rubric text is truncated (1,024-token context vs 512-token truncation: monolith drops from 76.38% to 73.82%, making it indistinguishable from response-only).
  • Capacity-matched monolithic adapters (rank-64) outperform fragmented judgelet banks, showing fragmentation imposes a statistical learning cost beyond raw parameter count.
  • Optimization controls (matched updates, learning-rate variants) indicate the loss from judgelets is not explained solely by optimizer schedule or steps — initialization and shared pretraining matter.

Data & Methods

• Data partitions (direct-assessment)
  • Total: 99,952 rows; criterion holdout: 206 criteria / 20,696 examples.
  • Remaining: 47,638 training; 9,441 calibration; 9,597 development; 12,580 locked test.
  • External test: Feedback-Bench (1,000 examples; some criteria excluded from adapter training).
• Criterion holdout and robustness
  • A deterministic hash reserved the held criteria (ensures complete criterion shift — no examples governed by held criteria used in training/calibration).
  • Rubric-intervention test: replace rubric + five score descriptors with an unrelated rubric from another cluster to test rubric sensitivity.
• Judge construction and training
  • Base model: Skywork-Reward-V2 Qwen3 0.6B used as frozen base.
  • LoRA adapters trained on attention and FF projections; rank-8 judgelets (5.05M params); rank-64 monolith used for capacity-matching.
  • Primary judgelet bank training: experiments with matched aggregate optimizer budget and with per-family cycling to equalize updates; warm-start variants initialized from a trained monolith and then fine-tuned per family.
• Router & risk functions
  • Router: TF–IDF vectorizer on rubric text + K-means clustering to assign rubric families.
  • Confidence functions / policies: global threshold, per-family thresholds, and Bonferroni-adjusted per-family thresholds; thresholds selected on calibration data and audited on locked test with one-sided 95% Wilson / exact Clopper–Pearson bounds.
• Deferral/risklet fitting
  • For cascades, small logistic probes (risklets) predict whether the stage’s top candidate is correct using: top scores, gap, range, mean, variance, candidate count, winning position, and skill-family one-hot; fitted on disjoint partitions for fitting / calibration / test.
  • Cascades tested across all compute-ordered subsets (0.6B, 1.7B, 4B, 8B); compute cost reported as normalized parameter compute to compare architectures.

Implications for AI Economics

• Statistical vs stored capacity tradeoff
  • Storing many small adapters (judgelets) is attractive because only one adapter needs activation at inference, saving runtime cost. However, fragmenting training data across many adapters imposes a statistical cost: per-adapter sample sizes shrink, hurting accuracy and safe audited coverage. Economically, the marginal value of storing more specialized adapters depends on dataset size per adapter; storing specialization is cheap, but it creates downstream risk and audit costs unless enough data supports per-adapter training.
• Deployment design: share vs split
  • For organizations with many rubrics but limited labeled data per rubric, it is economically efficient to share judgment learning (monolithic or warm-start shared base) and only implement domain-specific behavior at the audited release boundary (routers and risklets). This minimizes retraining costs and preserves audited coverage while allowing stored specialization to grow later.
• Compute–accuracy frontier and cost savings
  • Learned deferral (risklets) can both reduce compute and improve accuracy compared with always running a large judge: e.g., the 0.6B–4B–8B risklet cascade achieved higher accuracy at ~41.5% of the normalized compute of 8B-only. This implies tangible operational cost savings for evaluation-heavy pipelines (lower inference cost per evaluated item) while preserving or improving quality.
• Auditing and regulatory considerations
  • Release boundaries must be auditable and robust to rubric changes: calibration audited on correct rubric text can be invalidated if rubric text is corrupted or omitted. Firms should include robustness checks (rubric integrity, truncation effects) in audits. Bonferroni or per-family adjustments can tighten guarantees but reduce automatic coverage.
• Practical recommendations for product managers / economists
  • If you have limited per-rubric data: centralize training (shared base / monolithic adapter), use a router + risk-calibrated deferral cascade for operational scaling, and store specialized adapters but avoid relying on them until adequately warmed from shared training.
  • If you have large per-rubric datasets: per-rubric adapters become more viable; still audit per-family thresholds and monitor for rubric corruption or truncation.
  • Evaluate compute savings versus development/audit costs: cascades reduce average inference compute but add calibration and auditing workloads; quantify audit cost and operational risk when deciding architecture.
• Research/economic value
  • The paper quantifies how statistical sample sizes and architecture choices shift the evaluation accuracy–compute frontier; this provides a concrete basis for cost-benefit analyses of stored adapters vs centralized models and for pricing evaluation QA in agentic systems.

If you want, I can extract the key numeric results into a one-page table (parameters, accuracy deltas, coverage/risk figures, normalized compute) for use in a cost model or slide.