A cost-aware router halves expensive LLM reasoning spend while preserving accuracy: deployed on a production image-annotation workflow, DRR matches the top confidence-based system’s accuracy (≈82.8%) while cutting incremental reasoning-token use by about two-thirds and routing roughly 21–22% of items to human review.

Main Finding

The Differential Reasoning Router (DRR) is a cost-aware, three-way routing policy (Direct model, Reasoning model, Human review) for cold-start LLM annotation in e-commerce. By learning separate success probabilities for a low-cost Direct model and a higher-cost Reasoning model—both at the sample and rule levels—and predicting “double-failure” (ambiguity), DRR routes examples so that reasoning is used only where it has positive marginal value and human review is used for likely unsolvable or ambiguous cases. In a production lead-image eligibility task, DRR matches or slightly exceeds a strong confidence-based baseline in accuracy (82.8% vs 82.0%) while materially reducing reasoning-token usage (66.2% savings vs 55.2% for the baseline) and surfacing rule-level disagreement for targeted labeling/rule refinement.

Key Points

• Differential supervision: DRR trains heads that predict whether each automated annotator (Md = Direct, Mr = Reasoning) is correct versus human ground truth, at both system (conjunctive rules) and per-rule levels. It also trains an ambiguity head that predicts when both models fail.
• Marginal Value of Reasoning (MVOR): DRR estimates MVOR(q) = p_r(q) − p_d(q) (expected correctness gain from reasoning). Routing to the reasoning model requires MVOR > λ · ΔC(q), where ΔC is incremental reasoning cost and λ is a learned budget multiplier.
• Human gate: DRR uses the ambiguity score and a confidence threshold to escalate likely double-failure or broadly uncertain cases to human review (these become informative labels for cold-start learning).
• Budgeted objective and optimization: Training minimizes supervised + routing loss subject to an expected reasoning budget Btarget via a Lagrangian multiplier λ, optimized with alternating primal (model) and dual (λ) updates.
• Cold-start design: Conservative human-escalation thresholds generate a concentrated, informative label stream (ambiguous and disagreement cases) useful for prompt engineering, fine-tuning, calibration, and rule refinement.
• Practical architecture: Precomputed multimodal features (OpenCLIP ViT-L/14), small MLP routing head for low-latency inference, and model-agnostic routing that only needs candidate model outputs, realized costs, and ground-truth agreement labels.

Data & Methods

• Task: Production lead-image eligibility (multimodal: image + metadata) evaluated against k = 11 conjunctive business rules. Single rule failure makes an image invalid; rules mix objective and subjective checks.
• Dataset / cold-start protocol: 9,358 labeled examples (6,550 train / 1,403 val / 1,405 test). Labels come from mandatory pre-launch review (limited but available) representing a realistic cold-start.
• Models:
  • Md: Gemini 2.5 Flash Lite with “thinking” disabled (cheap).
  • Mr: Same model with dynamic thinking enabled (more expensive reasoning tokens).
  • Router feature extractor: OpenCLIP ViT-L/14 embeddings; features cached and fed to a shared MLP. Router trains system-level, rule-level, and ambiguity heads.
• Training objective: J(θ,λ) = Lsup(θ) + Lroute(θ) + λ( Croute − Btarget ), where Lsup is BCE for all heads, Lroute is expected error under relaxed sigmoid routing, and λ is updated in log-space as a budget price.
• Inference policy:
  • Human escalation if p_amb(q) > τ_amb or max(p_d, p_r) < τ_conf (thresholds tuned on validation under a human-referral budget).
  • Otherwise select Mr iff MVOR(q) > λ* ΔC(q); else select Md.
• Baselines compared: Direct-only, Reasoning-only, Random Md/Mr, MVOR-only (no human gate), MaxConf (confidence-based selection + human escalation), and Oracle (best-of-two automated selector).
• Main quantitative results (test N=1,405):
  • DRR: Accuracy 82.8% [80.8, 84.8], Human review 21.7%, Reasoning-token savings 66.2% (vs always using Mr).
  • MaxConf: Accuracy 82.0% [80.0, 84.0], Human review 20.0%, Reasoning-token savings 55.2%.
  • Oracle (best-of-two without humans): 79.1% accuracy, highlighting the need to model human review as an action.
  • Direct-only / Reasoning-only: ~68.0% / 69.3% accuracy.
• Diagnostic finding: DRR partitions test items into three zones—Direct, Reasoning, Ambiguity. The Ambiguity zone (~21.7% of examples) has much higher unsolvable incidence (≈50.8% rule-level joint failure), justifying human escalation for that slice.

Implications for AI Economics

• Cost-effective compute allocation: DRR operationalizes a marginal-value approach (MVOR) that conditions reasoning spend on expected benefit per sample, enabling substantial reasoning-token savings without sacrificing accuracy. This directly reduces per-annotation compute cost and can defer or reduce paid inference budgets for expensive LLM modes.
• Better human-in-the-loop ROI: Treating human review as both a safety and label-acquisition action concentrates expensive human effort on the most informative/ambiguous cases. That improves label efficiency for downstream calibration, prompt engineering, or fine-tuning, accelerating improvement per human-dollar spent.
• Budgeted deployment and pricing decisions: DRR’s Lagrangian budget multiplier λ provides a principled mechanism to trade off cost and accuracy and can be used for cost-SLA planning (e.g., set Btarget to meet monthly compute budget). It also supports negotiating costs with LLM providers by quantifying token-savings attributable to selective reasoning.
• Operational risk reduction in cold-starts: Cold-start deployments face scarce, evolving labels and ambiguous business rules. DRR reduces rollout risk by routing uncertain/ambiguous cases to humans, producing targeted ground truth that reduces long-run verification costs and time-to-market for new attributes or rule changes.
• Evaluation beyond accuracy: The results exemplify the need for cost-adjusted performance metrics (accuracy conditional on compute + human budget). Procurement, forecasting, and ROI calculations should incorporate reasoning-token savings and human-review rates, not just raw model accuracy.
• Strategic labelling and model investment: By identifying per-rule disagreement patterns, DRR helps prioritize where human labeling or supervised fine-tuning yields the highest marginal value—informing decisions like whether to invest in more expensive models, dataset collection, or rule clarifications.
• Potential limits and generalizability: Savings and routing effectiveness depend on having distinct Direct and Reasoning model behaviors, measurable incremental costs, and an initial seed of ground truth. Economic benefits scale with task frequency and model invocation costs; tasks with very low incremental reasoning cost or where both models almost always agree may yield smaller gains.

Caveats / Further considerations - Results come from a single production e-commerce image-rule task; effectiveness may vary across tasks (NLP-only, higher-dimensional rule sets, different cost structures). - DRR requires initial human-reviewed labels to train reliability heads (though the design targets realistic pre-launch label availability). The economics of label acquisition to bootstrap DRR should be weighed against expected long-run savings. - Implementation requires accounting for both token costs and human labor costs; current experiments only report reasoning-token savings (human labor excluded from token accounting). - As base models evolve, router retraining or recalibration is necessary; however, the router is model-agnostic and can be updated by regenerating candidate outputs.

Overall, DRR provides a practical, budget-aware method to lower annotation costs and structure human review in early deployment, offering a quantifiable and operationally useful trade-off between compute (reasoning tokens), human effort, and annotation quality.