Combining many LLMs and multiple sampled outputs dramatically boosts effective performance: across 21 models and 16 benchmarks, multi-model multi-generation selection reduces error rates by up to 82% versus typical single-model single-run evaluation and reaches state-of-the-art accuracy at 85% lower cost.
The Capability Frontier: Benchmarks Miss 82% of Model Performance
Allowing selection across multiple LLMs and multiple sampled generations substantially raises achievable accuracy relative to single-model single-run benchmarks—correcting for single-model evaluation cuts error rates by 54% and further correcting for single runs yields an 82% improvement, matching SOTA at 85% lower cost.
Existing benchmarks typically report accuracy for a single model on a single run. This systematically understates real-world LLM capabilities, particularly under heterogeneous data distributions: (i) different models get different questions correct according to their specializations, and (ii) given a budget, multiple generations can be sampled and selectively retained. To quantify this gap, we introduce the Capability Frontier: a Pareto frontier over a set of models that characterizes the best achievable performance at each cost level under optimal selection across models and generations (i.e., via an oracle). Our construction corrects for two opposing biases: underestimation from single-model evaluation and overestimation from taking maxima over noisy samples. We study 21 LLMs across 16 widely used benchmarks spanning coding, reasoning, medicine, factuality, instruction following, and agentic tasks, comparing Capability Frontier performance at matched cost to each benchmark's top-performing model. Correcting for single-model evaluation yields a 54% error rate reduction; additionally correcting for single runs yields an 82% improvement, with SOTA accuracy matched at 85% cost reduction. Complementing these empirical results, we use controlled probabilistic simulations to show that higher query topic entropy produces a near-monotonic increase in the performance gap between oracle routing and the best single model. Our findings suggest collective LLM capabilities are substantially underestimated, with implications for evaluation and deployment in data-heterogeneous, multi-domain settings.
Summary
Main Finding
Standard single-model, single-run benchmarks substantially understate what a heterogeneous ensemble of LLMs can achieve. The paper defines the "Capability Frontier" (oracle routing across models and generations) and — after correcting for finite-sample selection bias — finds large, actionable gains: at matched cost, routing across 21 models on 16 benchmarks yields a 54% average error-rate reduction versus the SOTA single model; allowing posthoc multi-generation selection raises that to an 82% error reduction. Equivalently, SOTA accuracy can be matched at ~85% lower token cost on the Capability Frontier.
Key Points
- Capability Frontier: the Pareto frontier over model-selection policies (quality vs. cost) obtained by optimally selecting model (and generation) per query under an oracle.
- Large empirical gains: across 16 benchmarks (coding, reasoning, medical, factuality, instruction-following, agentic), Otrue improves average accuracy from 82.4% → 88.8% (≈54% error reduction) and reduces average token cost by ≈85% at matched quality.
- Posthoc selection (using a perfect verifier to choose among multiple generations) is especially powerful: k=1 gives 66% error reduction vs SOTA; k=10 gives 82% reduction.
- Naive oracle (max over sample means) is positively biased: quality overestimated by ≈1.2% on average; cost overestimated by ≈37.5% on average. Bias arises because taking maxima over noisy estimates preferentially selects positive outliers.
- Bias scaling: analytic Gaussian case and simulations show bias decays ∝ G^(-λ) with λ ≈ 0.5 in many settings (empirically λ∈[0.25,0.75]). Reliable debiasing by extrapolation needs large G (≈50) — unrealistic in many cost-constrained settings.
- Debiasing methods:
- Extrapolation: fit Obiased(G) = α + β G^-λ (or a smooth variant) and take α as Otrue.
- Probabilistic graphical model (PGM): latent variables for prompt difficulty, topic, and model aptitude; infer per-prompt success probabilities and compute Otrue from inferred π̂nl.
- Simulations: oracle uplift grows monotonically with workload/topic entropy — heterogeneous workloads amplify the value of routing and model complementarity.
- Practical caveats: experiments used G=10 generations per prompt, 21 LLMs, binary correctness metrics; posthoc gains assume a perfect, free judge; PGM structural assumptions and conditional independence may under/over-estimate in settings with shared data/strong correlations.
Data & Methods
- Benchmarks: 16 datasets with verifiable answers (execution-based code verification, exact-match QA), covering coding (HumanEval-X variants, MBPP, LeetCode, BigCodeBench, LiveCodeBench), reasoning (LiveBench-Reasoning, GPQA Diamond), instruction-following (LiveBench-IFEval), medical (MedCalcBench), factuality (TruthfulQA), and agentic (Terminal-Bench 2.0, agentic LiveBench-Coding).
- Models: 21 LLMs from major providers (OpenAI GPT-5 series, Anthropic Claude 4.5, Google Gemini 2.5 variants, Meta Llama 4 variants, Mistral family, Qwen family, Moonshot Kimi K2, DeepSeek R1, GLM-4.6), evaluated with provider default sampling settings.
- Observations: for each prompt-model pair, G = 10 independent generations; metrics collected: binary correctness (quality) and API token cost (USD as of 2026-01-01).
- Oracle estimators:
- Obiased: per-prompt argmax over sample means (standard, biased).
- Otrue (debaised): estimated via (a) extrapolation of Obiased(G) to large G, and (b) PGM inferring latent difficulty/topic/aptitude and predicting per-model success probabilities π̂nl; Otrue = avg_n max_l π̂nl.
- Posthoc routing: modeled both combinatorially (naive all-models/all-generations) and via PGM closed-form with perfect free judge; results reported for k attempts per model.
- Synthetic simulations: PGM used to generate controlled workloads (L=10 models, T=30 topics, N=1,000 datapoints, large G for ground truth) with varying Dirichlet concentration for topic entropy; measured oracle uplift vs. single best model.
Implications for AI Economics
- Measurement and valuation
- Benchmarks that report single-model single-run performance underprice system-level capability. Procurement, capability comparisons, and R&D prioritization based on such benchmarks will understate what can be achieved via orchestration.
- Debiased, ensemble-aware evaluation (Capability Frontier) should be incorporated into model valuation, benchmarks, and SLAs to reflect deployable performance.
- Market structure and product strategy
- Model specialization and complementarity create value for model diversity: marketplaces and buyers will benefit from access to heterogeneous model pools plus routing/orchestration layers. Firms offering cheap, specialized models plus orchestration could outcompete monolithic SOTA providers on cost-effectiveness.
- Pricing strategies: providers may face pressure to unbundle capabilities (cheap specialized models + orchestration fees) rather than selling "one-size-fits-all" premium models.
- Deployment costs and efficiency
- Large potential cost savings (≈85% token-cost reduction at matched quality) imply substantial efficiency gains for AI-driven services. Enterprises can lower marginal cost of inference by combining models and routing intelligently.
- Investing in inexpensive routing/judging infrastructure (including partial verifiers) can yield outsized returns relative to model improvements.
- Incentives and competition
- Because oracle gains grow with workload heterogeneity, firms operating across diverse task mixes (e.g., multi-domain customer support, medical triage) will have higher incentives to adopt orchestration. This can shift competitive advantage toward platforms that integrate model pools and routing.
- Conversely, if orchestration becomes standard, single-model SOTA premiums might compress, changing R&D incentives (less pressure to build universally superior single-models, more to build complementary specialists and cheap models).
- Policy, safety, and regulation
- Regulators and auditors should account for ensemble/orchestrated capability when assessing risks, impact, or compliance; single-model benchmarks can understate deployed system capability (and hence risk).
- Posthoc selection (verifiers) can dramatically increase combined capabilities. If verifiers are imperfect or costly, economic trade-offs and safety checks must be modeled explicitly.
- Research and procurement guidance
- For economic decision-making, measure and report (a) capability frontiers, (b) debiased oracle estimates, (c) model reliability, and (d) workload topic entropy — these dimensions better predict deployable performance and cost-effectiveness than point estimates from single-model benchmarks.
- Procurement contracts could be structured around ensembles + routing performance (cost-quality frontier) rather than single-model averages.
Overall, the paper argues that the unit of economic analysis for LLM capability should often be the ensemble plus orchestration policy (the Capability Frontier), not an isolated model-run metric. Accounting for debiased ensemble performance changes cost-benefit calculations, market incentives, and regulatory assessment for LLM deployment.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper provides systematic, multi-benchmark empirical evidence across 21 models and 16 tasks and supports claims with controlled simulations; however, the central 'oracle' selection assumption is optimistic and some results depend on cost-model and benchmark choices, limiting how strongly one can generalize the measured gaps to real-world deployments.
Methods Rigormedium — The authors construct a principled Pareto Capability Frontier, correct for two opposing biases, and validate with probabilistic simulations — methodological choices are transparent and appropriate — but key assumptions (perfect routing/oracle, cost matching specifics, finite model set) and potential sensitivity to benchmark selection reduce overall methodological robustness.
SampleEvaluation of 21 large language models across 16 widely used benchmarks spanning coding, reasoning, medicine, factuality, instruction following, and agentic tasks; analyses compare single-model single-run performance to multi-model multi-generation oracle selection at matched cost and include controlled probabilistic simulations to explore effects of topic entropy.
Themesproductivity adoption
GeneralizabilityOracle routing (perfect selection across models/generations) is not achievable in practice; operational routing/selection will perform worse., Limited to 21 models and 16 benchmarks — may not capture all model specializations or real-world task heterogeneity., Benchmarks are static and may not reflect temporal drift, production data distributions, or multi-turn interactive settings., Cost model and generation-budget assumptions may not match diverse deployment constraints (latency, API limits, privacy, integration costs)., Excludes organizational/infrastructure frictions (latency, model access, combinatorial policy, legal constraints) that affect deployability.
Claims (6)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Correcting for single-model evaluation yields a 54% error rate reduction. Other | positive | error rate |
Reading fidelity
high
Study strength
medium
|
n=21
54% error rate reduction
|
| Additionally correcting for single runs (i.e., sampling multiple generations and selective retention) yields an 82% improvement. Other | positive | performance (improvement over baseline single-run single-model evaluation) |
Reading fidelity
high
Study strength
medium
|
n=21
82% improvement
|
| Using the Capability Frontier, state-of-the-art (SOTA) accuracy is matched at an 85% cost reduction. Other | positive | accuracy at matched cost (cost reduction required to match SOTA accuracy) |
Reading fidelity
high
Study strength
medium
|
n=21
85% cost reduction
|
| We study 21 LLMs across 16 widely used benchmarks spanning coding, reasoning, medicine, factuality, instruction following, and agentic tasks. Other | null_result | study/sample scope (number of models and benchmarks) |
Reading fidelity
high
Study strength
high
|
n=21
|
| Higher query topic entropy produces a near-monotonic increase in the performance gap between oracle routing (Capability Frontier) and the best single model (shown via controlled probabilistic simulations). Other | positive | performance gap between oracle routing and best single model (as a function of query topic entropy) |
Reading fidelity
high
Study strength
medium
|
near-monotonic increase (qualitative)
|
| Existing benchmarks typically report accuracy for a single model on a single run, which systematically understates real-world LLM capabilities—particularly under heterogeneous data distributions—because (i) different models get different questions correct according to their specializations, and (ii) given a budget, multiple generations can be sampled and selectively retained. Other | negative | reported benchmark accuracy vs achievable collective capability |
Reading fidelity
high
Study strength
medium
|
n=21
|