Large language models tend to accept externally supplied answers they already favor, meaning identical tool outputs can improve weaker models but degrade stronger ones; crucially, models often represent verification results without recruiting them into decisions, and candidate integration occurs late in the network via a causal promotion-and-transport pathway.

Main Finding

Large language models (LLMs) integrate external evidence by reweighting their pre-existing answer distribution according to a simple, receiver-dependent law: p(y | e) ∝ q(y)^(1+a) · exp(ϕe(y)), where q is the model’s prior answer distribution, a (per model–domain) is a prior-dependence scalar, and ϕe is a local evidence tilt concentrated on the proposed candidate. This law predicts — and the authors validate empirically and mechanistically — that (i) candidates the receiver already favors are more persuasive, (ii) receivers more readily accept errors that resemble their own characteristic mistakes (correlated errors amplify harm), and (iii) identical evidence can help weaker models but harm stronger ones. Crucially, verification (the model’s ability to judge a candidate in isolation) only affects final use through a learned recruitment gain; models frequently verbalize correct rejections yet still adopt the rejected candidate.

Key Points

• Formal arbitration shift: The change in pairwise odds after evidence equals the pre-existing margin plus an external arbitration shift. There exists an evidence potential Φe with Φe(y) = a·log q(y) + ϕe(y), ϕe(y) = 0 for y ≠ v (locality).
• Rational receiver is recovered as a = 0 and ϕe given by source likelihoods; scalar source trust is the special case when source errors are uniform.
• Prior weight (1 + a) is receiver- and domain-specific. Empirically measured between ~0.20 and ~0.65 on the arithmetic instrument (i.e., always below the Bayesian benchmark of 1).
• Evidence tilt ϕe decomposes into:
  • ρ · V (recruitment gain times model’s checking log-odds V),
  • source-cue premia γ,
  • receiver competence effects and interactions. Here ρ is the causal gain with which internal verification is recruited into use.
• Generation vs checking vs use are distinct: G = generate-correct without candidate, C = correctly judge candidate, U+ = adopt correct candidate, U− = adopt incorrect candidate. These can dissociate strongly (high C but low recruitment ρ).
• Empirical phenomena:
  • Candidates already supported by the model are disproportionately persuasive.
  • Receiver-congruent (aligned) errors are more damaging than random errors at the same rate (receiver-relative reliability frontier).
  • Identical evidence can be beneficial for weaker receivers and harmful for stronger ones (crossing effect).
  • Sequential evidence shows sub-additivity and order effects as predicted by the distributional law.
• Verification often does not control use:
  • Models may correctly reject invalid candidates in isolation but still adopt them in final answers (93–100% adoption for propositional constraints; up to 99.4% in held-out scientific reasoning).
  • Recruitment ρ is therefore a distinct, learnable policy parameter.
• Mechanistic findings:
  • Candidate integration follows stages: admission, promotion, transport, late integration into answer state.
  • Candidate promotion magnitude ≫ truth-suppression magnitude (promotion ≈ 9× suppression).
  • The causal handoff from evidence to answer occurs late (~79–81% of network depth in Llama and Qwen).
  • Representations that encode/verbalize verification can be decodable but often have little causal impact on the answer; J-lens decomposition shows dissociable internal subspaces for verbalized verification versus causal arbitration.
• Model- and domain-specificity:
  • Same-scale checkpoints differ in behavior at fixed domain.
  • Model–domain interactions explain much of between-cell variation.
  • Parameter count alone does not predict prior-dependence or recruitment.

Data & Methods

• Scope: Over 10 million trials, 12 LLMs drawn from four families (examples include Llama, Qwen, Gemma), across 8 domains; four domains are scientific discovery tasks (quantum mechanics, physics, genetics, molecular biology), plus arithmetic and other reasoning tasks.
• Experimental design:
  • For each item, measured before-candidate answer distribution q (calibration samples at temperature 0.7).
  • Separated measurements for generation (G), checking competence (C), and use/adoption (U+ / U−) on the same items.
  • Cross-fitting and matched designs to disentangle prior support q from recruited verification.
  • Varied source cues (role, claimed provenance, wording, delivery format) and certificate/check types (propositional constraints, local scans, extended checks).
  • Sequential evidence experiments to test composition laws (order and sub-additivity).
• Estimation:
  • Fitted the prior-dependence scalar a (so 1 + a is the prior weight), recruitment gain ρ, source premia γ, and other interaction terms via logistic/logit regressions derived from the theoretical form logit p = (1 + a)·Ami + ϕe + item constants, where Ami is the model’s pre-evidence log-probability.
• Mechanistic causality:
  • Used residual-stream patching, attention knockouts, activation steering, and J-lens / logit-lens style probes to trace candidate states and test causal effects.
  • Identified stages (admission → promotion → transport → integration), measured depths of handoff, and compared causal vs decodable verification directions.
• Calibration & decoding details:
  • Calibration at temperature 0.7; checking and use/adoption trials decoded greedily so adoption = exactly decoded answer.
• Key quantitative results:
  • Prior weight 1 + a measured between ~0.20 and ~0.65 on arithmetic instrument.
  • Adoption of rejected candidates: 93–100% (propositional constraints); up to 99.4% in held-out scientific reasoning.
  • Candidate promotion ≈ 9× truth suppression.
  • Causal handoff depth ≈ 79% (Llama) and 81% (Qwen).

Implications for AI Economics

• Reliability and value of information are receiver-relative:
  • Source/tool reliability cannot be evaluated independently of the receiving model. Market participants (tool providers, deployers, auditors) must consider the receiver’s prior landscape and recruitment policy, not just the source’s aggregate accuracy.
  • Certification and auditing should test model + tool combinations (receiver–source pairs), because identical tool outputs can have opposite welfare effects depending on receiver competence and prior.
• Contracts, procurement, and liability:
  • Contracts for tool integration, SLAs, and liability assignment should account for model-specific prior-dependence and recruitment behavior. A tool that reproduces receiver-congruent errors can be disproportionately harmful, increasing downside risk beyond naïve error rates.
  • Insurance and risk models for AI-assisted decision systems must consider correlated-error amplification: higher expected loss than predicted by independent-error assumptions.
• Competitive dynamics and product strategy:
  • Identical evidence can help weaker models and harm stronger models. This creates non-monotone effects for product improvements and tool bundling: upgrading a receiver may reduce the marginal value of some tools or even make them harmful. Vendors and integrators should empirically evaluate whether an augmentation helps or hurts each target model.
• Pricing and markets for model–tool combos:
  • Value-of-information pricing should be receiver-aware: buyers value a tool partly for how it shifts their own model’s distribution (1 + a, recruitment ρ), not solely for standalone accuracy. Market mechanisms (e.g., auctions, bundles) should allow specifying compatibility tests.
• Incentives for provenance and interface design:
  • Source cues (claimed provenance, role, format) systematically affect integration independently of factual support. Interface design, provenance authentication, and standardized provenance signals matter economically — they change adoption behavior and thus downstream outcomes (productivity, error rates).
  • There is economic value to provenance authentication channels that alter γ premia in predictable ways.
• Policy and regulation:
  • Regulations that require tool outputs to be “verified” by the receiving model (or that accept the model’s verbalized verification as evidence) are insufficient unless they also require testing of recruitment behavior. Models often verbalize correct checks without recruiting them into decisions.
  • Standards for safety and deployment should mandate empirical stress tests that measure generation, checking, and recruitment separately; certification should include causal probing (or equivalent behavioral tests) to detect decoupling.
• System design for multi-agent and scientific workflows:
  • In pipelines where multiple models/agents/tools exchange outputs (scientific discovery agents, instrument-control agents), the receiver-dependent amplification of errors implies that pipeline-level reliability must be assessed at the level of how each agent integrates upstream evidence.
  • Designers should avoid relying on downstream verbal verification as a seal of correctness; instead, build architectures that actively enforce recruitment (higher ρ) where correctness matters — e.g., require recomputation, enforce checks cheaper than generation, or design handoff points earlier where interventions can be effective.
• Investment and R&D priorities:
  • Improving scalar accuracy of tools is necessary but insufficient; investments in methods to increase recruitment (making verifications causally control use), reduce pathological prior-dependence, and decouple causal arbitration from mere verbalization may yield high welfare returns.
  • Mechanistic interpretability that locates and modifies late-integration pathways is a promising route to align use with verification.

Summary takeaway: Evaluating and regulating augmented LLM systems requires treating evidence integration as a receiver-specific, distributional operation. Economic decisions (procurement, certification, liability, and market design) should account for how a receiving model’s prior, recruitment policy, and interaction with source cues determine whether external evidence helps or harms.