9 cumulative citations
View corpus contextSampling many LLM answers doesn't buy truth: even at 25× inference cost, aggregating outputs fails to improve accuracy in unverifiable tasks and can amplify shared errors; self-reported confidence offers little help.
Consensus is Not Verification: Why Crowd Wisdom Strategies Fail for LLM Truthfulness
Polling many model outputs does not improve truthfulness in domains without reliable verifiers—errors are strongly correlated across models, so aggregation often reinforces shared misconceptions and confidence signals do not reliably separate correct from incorrect answers.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Pass@k and other methods of scaling inference compute can improve language model performance in domains with external verifiers, including mathematics and code, where incorrect candidates can be filtered reliably. This raises a natural question: can we similarly scale compute to elicit gains in truthfulness for domains without convenient verification? We show that across five benchmarks and models, surprisingly, it cannot. Even at 25x the inference cost of naive sampling, polling-style aggregation yields no consistent accuracy gains over single-sample baselines and often amplifies shared misconceptions. We find that under uncertainty, models are better at predicting what other models will say within model ensembles than at identifying what is true, revealing a separation between social prediction and truth verification. Across models and benchmarks, aggregation fails to provide a robust truth signal because language model errors are strongly correlated. The source of correlation goes beyond any individual benchmark: we show that even when conditioned on out of distribution random strings and asked to produce pseudo-random outputs, different models produce correlated outputs. Confidence-based weighting provides no benefit because self-reported confidence fails to reliably distinguish correct from incorrect answers. These results delineate a boundary for inference-time scaling: in verified domains, additional samples provide more candidates for a verifier to filter; in unverified domains, additional samples merely reinforce shared misconceptions.
Summary
Main Finding
Polling-style aggregation (repeated sampling, majority voting, confidence-weighting, and the Surprisingly Popular algorithm) does not reliably improve LLM truthfulness in domains where no external verifier exists. Even at large inference costs (e.g., 25× sampling), aggregation increases consensus but not accuracy, and often amplifies shared mistakes because model errors are strongly correlated across samples and across model families.
Key Points
- Aggregation failure is systematic:
- Across four verifier-absent benchmarks (Com2Sense, Humanity’s Last Exam (HLE) subset, BoolQ, and a new Predict-the-Future forecasting set) and five open-source instruction-tuned models, no aggregation rule consistently outperforms single-sample baselines.
- Forecasting questions (outcomes postdating model knowledge) remain at chance under all aggregation methods.
- Aggregation signals reflect consensus, not correctness:
- Self-reported confidence and predicted vote-share correlate with expected agreement, not with ground-truth correctness.
- The Surprisingly Popular (SP) signal is unstable: on HLE the inverse-SP signal correlates strongly with truth (i.e., SP is anti-correlated), while on other tasks SP is near chance.
- Errors are strongly correlated:
- When models err, they frequently converge on the same incorrect choice (high concentration of wrong answers).
- Error correlation persists across temperatures and across model families; ensembling different architectures and sizes does not reliably restore independence.
- Correlation is structural, not just shared facts:
- Negative-control experiment: models given uniformly random 32-character strings and forced to choose among {A,B,C,D} show above-chance inter-model correlations (up to ~0.35), indicating shared inductive biases/priors rather than shared factual knowledge.
- Models are better at predicting collective opinion than at predicting correctness: vote-share prediction is strong, confidence-to-accuracy correlation is weak.
Data & Methods
- Tasks and response formats:
- Binary-choice tasks (YES/NO or multiple choice), with three elicited signals per sample: binary answer, self-reported confidence (0–100), and a prediction of vote share for a canonical option.
- Benchmarks: Com2Sense (commonsense), BoolQ (binary factual QA), HLE subset (expert-level binary questions), Predict-the-Future (forecasting with outcomes resolved after model cutoffs).
- Note: paper focuses on binary tasks as the most favorable regime for polling methods.
- Models:
- Five instruction-tuned open-source models across sizes/families: Gemma-3-4B; GPT-oss-20B & GPT-oss-120B; Qwen-32B; Qwen3-235B.
- Intra-model crowds: repeated samples from one model. Inter-model crowds: pooled responses across all five models.
- Sampling protocol:
- 25 independent samples per question per temperature T ∈ {0.7, 1.0} (moderate-to-high temperatures to ensure diversity).
- For ensembles, 125 votes per question (5 models × 25 samples).
- Total reported scale: ~375,000 model samples across experiments.
- Aggregation methods evaluated:
- Majority vote, Highest Confidence, Confidence-Weighted Vote, Prediction-Weighted Vote, Surprisingly Popular (SP), plus an “inverse-SP” diagnostic.
- Evaluation metrics:
- Accuracy with 95% bootstrap confidence intervals (resampling questions).
- Correlation and dependence diagnostics: majority stability under temperature change, inter-rater agreement (vote entropy, Fleiss’ κ), concentration of wrong answers, Pearson/Cohen correlations between model pairs and truth.
- Additional controls:
- Negative-control random-string experiment to remove any ground-truth signal.
- Verifiable-domain comparison (MATH and AIME) showing aggregation succeeds only when an external verifier can filter candidate outputs.
Implications for AI Economics
- Limits of inference-time compute as an investment:
- Allocating extra compute at inference (sampling / ensemble polling) yields diminishing or no returns for truthfulness in unverifiable domains. Economic models that treat inference-time compute as a general-purpose substitute for scaling truth should be adjusted: returns are task-dependent and hinge on verifier availability.
- Product design and business risk:
- Aggregation-based features that raise apparent consensus (e.g., “most models say…”, confidence-boosted answers) can increase user confidence without improving truth, creating reputational and liability risks for firms deploying LLMs in high-stakes or factual contexts.
- Firms should avoid relying on internal-aggregation-only mechanisms to claim improved factuality; instead invest in external grounding, retrieval, or verifiers where possible.
- Market for verification and provenance:
- There is economic opportunity and growing value for external verification services, grounded knowledge sources, and audit tools. Buyers will prefer LLM products with verifiable pipelines and provenance, creating a market premium for systems that integrate verifiers.
- Cost-benefit and pricing strategies:
- Charging premium prices for “ensemble” or heavy-sampling modes is only justified in verifiable tasks (code, math, programs with checkers). For unverifiable tasks, additional compute may only raise confidence and not accuracy—pricing should reflect this.
- Research and investment priorities:
- Prioritize methods that break error correlation rather than just increasing sample counts: architectural diversity, training diversity, grounding to external data/sensors, and better calibration mechanisms.
- Fund work on reliable confidence signals and verification modules (including human-in-the-loop adjudication) rather than purely inference-time sampling.
- Regulatory and policy considerations:
- Regulatory frameworks or procurement standards that require verifiability, audits, or independent checks of model outputs will reduce harm from false consensus effects and align incentives toward verifiable systems.
- Labor and substitution effects:
- Use of LLMs as decision aids in tasks lacking external verification (e.g., forecasting, judgment calls) should account for correlated machine biases; economic models of labor substitution must discount the naive productivity gains claimed from aggregation-based “quality” improvements.
- Limitations and economic caution:
- The paper studies binary-choice tasks and open-source instruction-tuned models; outcomes may differ for open-ended generation, proprietary models, or models with explicit grounding. Still, the core message holds: without breaking correlated errors or adding verifiers, more sampling is unlikely to reliably increase truthfulness.
Overall takeaway for AI economists and decision-makers: additional inference-time compute via polling/ensemble sampling is not a general-purpose substitute for verification. Investments should focus on verification, ground-truth acquisition, architectural/training diversity to reduce correlated errors, and reliable calibration—especially when the economic value or societal risk hinges on factual accuracy.
Assessment
Paper Typecorrelational
Evidence Strengthmedium — The paper runs systematic experiments across multiple benchmarks, model families, and interventions (large-sample polling, confidence-weighting, out-of-distribution random-string controls), which provides consistent empirical support for the central claim; however, the evidence is limited to the tested benchmarks and models, is not causal in an economic sense, and may not generalize to all architectures, prompts, or future models.
Methods Rigorhigh — The study appears to employ careful, multi-pronged tests (varying sample budgets up to 25x, ensemble polling, confidence-weighting, and OOD random-string conditioning) that expose correlated-error mechanisms and rule out some obvious alternative explanations, indicating strong experimental design and robustness checks; lacking are details about exact model families, prompt variants, and statistical uncertainty reported here, which would further strengthen rigor.
SampleExperiments span five benchmarks including domains with external verifiers (e.g., mathematics, code) and unverified tasks; multiple language models and model ensemble configurations were evaluated with polling-style aggregation up to 25× the inference cost of naive sampling; tests included confidence-weighted aggregation and out-of-distribution conditioning on random strings to probe intrinsic output correlations.
Themeshuman_ai_collab adoption
GeneralizabilityResults are conditional on the particular benchmarks and model families tested; different architectures or substantially newer models may behave differently., Prompt engineering, chain-of-thought or other decoding strategies not exhaustively covered could change outcomes., Human-in-the-loop verification or external verifiers (including tool-augmented models) can alter whether aggregation helps., Tasks with reliable automated verifiers (e.g., formal math, unit-testable code) remain an exception and are not contradicted by these findings., Findings on OOD pseudo-random conditioning may not fully capture all realistic real-world question distributions.
Claims (8)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Pass@k and other methods of scaling inference compute can improve language model performance in domains with external verifiers, including mathematics and code, where incorrect candidates can be filtered reliably. Output Quality | positive | language model performance (accuracy) in domains with external verifiers |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Scaling inference compute cannot improve language model performance in domains without convenient verification. Output Quality | negative | accuracy / truthfulness in unverified domains |
Reading fidelity
high
Study strength
medium
|
no consistent accuracy gains
|
| Even at 25x the inference cost of naive sampling, polling-style aggregation yields no consistent accuracy gains over single-sample baselines and often amplifies shared misconceptions. Output Quality | negative | accuracy; prevalence of shared misconceptions in aggregated outputs |
Reading fidelity
high
Study strength
medium
|
25x the inference cost (no consistent accuracy gains); often amplifies shared misconceptions
|
| Under uncertainty, models are better at predicting what other models will say within model ensembles than at identifying what is true, revealing a separation between social prediction and truth verification. Output Quality | mixed | ability to predict other models' outputs vs ability to identify true answers |
Reading fidelity
high
Study strength
medium
|
models better at predicting other models than truth (qualitative statement)
|
| Across models and benchmarks, aggregation fails to provide a robust truth signal because language model errors are strongly correlated. Error Rate | negative | correlation of errors across models; effectiveness of aggregation for truth signal |
Reading fidelity
high
Study strength
medium
|
strongly correlated errors (qualitative)
|
| Even when conditioned on out-of-distribution random strings and asked to produce pseudo-random outputs, different models produce correlated outputs. Error Rate | negative | correlation of pseudo-random outputs across models |
Reading fidelity
high
Study strength
medium
|
correlated outputs even under OOD pseudo-random conditioning (qualitative)
|
| Confidence-based weighting provides no benefit because self-reported confidence fails to reliably distinguish correct from incorrect answers. Output Quality | negative | effect of confidence-weighted aggregation on accuracy; calibration of self-reported confidence |
Reading fidelity
high
Study strength
medium
|
no benefit from confidence-based weighting; self-reported confidence fails to distinguish correctness
|
| In verified domains, additional samples provide more candidates for a verifier to filter; in unverified domains, additional samples merely reinforce shared misconceptions. Task Allocation | mixed | mechanism by which additional samples affect final correctness (filtering vs reinforcement of misconceptions) |
Reading fidelity
high
Study strength
speculative
|
not reported
|