LLMs can tell when their single-pass answer is likely wrong, supporting a cheap stay-or-escalate gate; but they cannot reliably choose which expensive multi-agent collaboration workflow will succeed, so cost-aware routing remains an open deployment problem.

Main Finding

Multi-agent collaboration protocols (iterative self-correction, planner–executor–reviewer (PER), broadcast deliberation) increase solve rates versus a single direct LLM pass, but the marginal value of each protocol is task-dependent and hard to predict. Same-model confidence signals reliably predict when the Baseline is likely to fail (useful for a binary stay-or-escalate gate), but they are weak at selecting which expensive protocol will pay off. As a result, conservative routers under-escalate (miss recoverable solves) and aggressive/frozen-LLM routers over-escalate (pay large token costs), leaving cost-aware protocol routing an open problem.

Key Points

• Benchmark & protocols
  • Main benchmark: 4,181 competition-level math problems (Omni-MATH 2), each run once under four fixed protocols with the same solver:
    • BASELINE (direct): avg 18.2K tokens, solve 56.3%
    • SINGLE (iterative self-correction): 47.6K tokens, solve 78.5%
    • PER (planner–executor–reviewer): 401.9K tokens, solve 84.9%
    • BROADCAST (multi-agent deliberation): 622.1K tokens, solve 88.9%
  • Retrospective fixed-order oracle (choose first successful protocol in cost order) achieves 92.4% solve on held-out split and serves as an upper-bound diagnostic.
• Failure-risk vs protocol-value distinction
  • Post-answer probe (same model, after producing Baseline answer) predicting Baseline correctness:
    • AUROC = 0.8847 (n = 4,151 parseable cases), AUPRC = 0.8950 — strong signal for failure risk.
  • Same failure-risk score applied to protocol-specific targets:
    • Any collaboration helps: AUPRC = 0.7683
    • PER-first success: AUPRC = 0.1674
    • Broadcast-only success: AUPRC = 0.1041 — protocol-specific precision weak.
• Practical gate
  • Pre-answer self-confidence gate (keep Baseline if confidence ≥70, else escalate to Single) reaches 78.0% solve at 45.0K tokens. This dominates some frozen-router points on the cost–solve frontier but remains 14.4 percentage points below the retrospective oracle on held-out data.
• Router behavior & trade-offs
  • Heuristic Tier-majority policy: 65.0% solve at 28.9K tokens.
  • Trained lightweight logistic routers (text+metadata) offered modest gains but did not approach oracle.
  • Frozen-LLM routers (e.g., gpt-oss-120b shown as router) raise solves (73.8% → up to ~80–83% for larger routers) at dramatically higher token costs and large excess relative to oracle.
  • Errors are directional:
    • Tier-majority: under-escalates 27.4%, over-escalates 12.5%
    • gpt-oss-120b router: under 18.0%, over 33.3%
    • Larger frozen routers: under 6–11%, over 63–71%
• Robustness & task dependence
  • Paired checks across 4 benchmarks (OmniMath, JEEBench, SciBench, LAB-Bench) and two solver families (gpt-oss-120b, Gemma-4-31B-it) — oracle improves coverage over Baseline by 23.2–58.3 percentage points across settings.
  • Protocol profiles vary by dataset and solver; Broadcast is often the strongest but does not uniformly dominate PER.
• Limitations noted by authors
  • Cost measured in token counts only (not latency, price, energy).
  • One deterministic run per problem–protocol (no repeated-sampling expected-utility estimate).
  • Oracle is retrospective and non-deployable; findings show task dependence rather than universal rules.

Data & Methods

• Matched-execution design: every problem evaluated deterministically under all four protocols using the same solver stack per comparison, enabling exact post hoc comparisons and a fixed-order oracle.
• Main dataset: Omni-MATH 2 subset (4,181 problems); additional datasets: JEEBench, SciBench, LAB-Bench (with two prompt variants) yielding 10 paired settings across two solvers.
• Routers evaluated:
  • Fixed policies (Baseline, Tier-majority).
  • Learned lightweight routers: five-class logistic regression over metadata ± TF–IDF text features (stratified splits for train/dev/test).
  • Frozen-LLM routers: label-only prompting that outputs one of the five actions.
  • Pre-answer and post-answer confidence probes from the solver model.
• Metrics:
  • Solve rate (coverage), average tokens consumed, excess tokens (per-problem positive overpayment vs the retrospective oracle), under- and over-escalation rates, AUROC/AUPRC for failure/benefit prediction.
• Key numerical results cited above (solve rates, token costs, probe AUROC/AUPRC, oracle gaps) are from the held-out splits and broader matched runs; confidence intervals given for many comparisons via bootstrap.

Implications for AI Economics

• Deployment trade-offs and pricing
  • Stronger protocols materially increase token consumption (order-of-magnitude jumps), so deploying them universally is economically inefficient. Platforms and providers should consider per-instance routing policies or differentiated pricing to reflect marginal computation costs.
  • Given strong gains in solve rates but large token overheads, monetization can follow a tiered-service model: charge more for "deliberation" protocols or expose protocol choices as premium options (pay-for-escalation).
• Decision-theoretic routing and expected value of computation
  • The useful post-answer failure signal supports simple decision rules (binary gates) that improve cost-efficiency. But the inability to predict protocol-specific marginal gains implies the need for richer per-instance expected-value estimation (expected benefit of running PER vs Broadcast minus their incremental costs).
  • Economic optimization requires modeling utility of a correct answer (task-dependent) vs token cost; this naturally calls for cost-aware objective functions during router training (e.g., maximize expected net utility rather than solve rate).
• Market and product design consequences
  • Over-escalation by aggressive routers (especially frozen LLMs) implies high operational costs for cloud providers / app operators — incentives exist to tune routers towards economically optimal escalation thresholds.
  • Service-level agreements (SLAs) and capacity planning must account for skewed, protocol-dependent token demands — unpredictable bursts when many instances escalate to PER/Broadcast can affect provisioning and pricing.
• Research & investment priorities from an economics perspective
  • Investment in protocol-value prediction (better signals, access to intermediate reasoning traces, causal analyses of when protocols help) can yield large cost savings by preventing unnecessary escalations.
  • Training routers with explicit cost-sensitive losses, reinforcement learning under realistic cost models, or multi-armed bandit approaches with costs could close part of the oracle gap.
  • Developing lightweight proxies for protocol value (e.g., richer metadata, partial traces, cheap verification tests) could be high ROI: reducing token waste while recovering much of the oracle coverage.
• Risk-management and welfare considerations
  • For critical applications where correctness has high value, paying for expensive protocols may be justified; for low-stakes settings, conservative gating limits costs but accepts higher failure risk. Economists and product managers should quantify the societal/firm-level value of correctness to guide these choices.

Overall, the paper shows that while LLM self-confidence can be monetically useful for a first-stage decision to escalate, per-problem selection among costly collaboration protocols remains unresolved — a clear opportunity for economically-motivated methods that optimize expected net benefit (value of correct output minus computation cost).