Bayesian rationality is hard to reject for black‑box AI: unless models make jointly contradictory zero‑probability claims, almost any pattern of probabilistic reports can be rationalized as Bayes‑consistent; additionally, more informative internal reasoning and optimal aggregation are observable only through coherent exclusions, leaving coarse reports fundamentally underdetermined.

Main Finding

The paper develops a black-box, Bayesian model of how agents (e.g., LLMs) form and report posterior beliefs after prompt-dependent internal computation, and derives sharp behavioral tests and limits on what can be rejected or inferred from repeated prompts. The central conclusions are: - Repeated stochastic reports are Bayes-rationalizable exactly when they do not produce jointly contradictory "hard exclusions": there must exist at least one complete world-state that every observed report (across all prompts) leaves possible. Apart from these zero-probability exclusions, reported frequencies and confidence/variability impose no constraints. - Comparing two agents for who is “more informative” (Blackwell order) reduces observationally to comparisons of hard exclusions: the more informative agent’s reports must (in a coupling) rule out every answer that its counterpart rules out, and possibly more. - Aggregating coarse reports (reports about answers/partitions rather than full-state posteriors) is highly under-identified: beyond enforcing exclusions, many aggregation weights are arbitrary. Arithmetic averaging can violate Bayesian feasibility; affine log-odds pooling (with prior correction when full-state posteriors are reported) is compatible and arises optimally in certain Gaussian models.

Key Points

• Model primitives:
  • Finite state space Ω, full-support prior μ0 (unknown to the observer).
  • Each prompt p induces an arbitrary finite experiment Ep = (Sp, πp); the agent observes a signal s ∼ πp(·|ω) and applies Bayes’ rule.
  • Agent is assumed obedient: it (truthfully) reports requested function of its posterior (typically the marginal posterior over answer classes Yp).
  • Repetitions at fixed true state ω produce i.i.d. draws from πp(·|ω), yielding an empirical report distribution τp.
• Theorem 1 (Bayes-plausibility): A family {τp} is consistent with the model iff there exists at least one complete state ω that lies in the support of every reported posterior across all prompts — equivalently, no set of observed reports jointly assigns zero posterior mass to every state. Everything else (positive probabilities, means, variances, frequencies) can be rationalized by appropriate prior/experiments.
  • Consequence: variability or confident bimodal reports (e.g., switching between 0.1 and 0.9) do not falsify Bayesian updating unless a zero occurs that cannot be reconciled across reports.
  • Extends to adaptive prompting: treat each history+next prompt as a distinct prompt; same support-condition applies.
• Theorem 2 (Plausible intelligence / Blackwell order): (Under a mild coarseness assumption |Ωy| ≥ 2 for each answer class)
  • τ2 plausibly Blackwell-dominates τ1 iff (i) there exists a state compatible with every report, and (ii) there is a coupling of reported posteriors so that every report by agent 2 rules out every answer that the paired report of agent 1 rules out (i.e., hard exclusions are nested).
  • Intuitively: observable implications of one agent being more informative are limited to a coherent nesting of zero-probability exclusions; confidence or variance differences alone do not identify an intelligence order.
• Theorem 3 (Aggregation limits): With coarse reports and at least two distinct reports and at least two underlying states per answer class:
  • Any Bayesian aggregate must exclude any answer that any constituent report excludes, but may distribute probability arbitrarily among answers that are not excluded by any report.
  • Arithmetic averaging of reported marginals can violate this feasibility condition if some reports exclude answers that others do not.
  • Affine log-odds pooling (i.e., summing log-odds with possible affine weights and correcting for the prior) respects the constraint; when agents report full-state posteriors, the Bayesian aggregate equals a prior-corrected sum of log odds (requires knowing the prior but not the experiment).
• Example illustrating rationalization: two agents report {0.4,0.6} or {0.1,0.9} with equal frequencies — both distributions are Bayes-plausible despite apparent disagreement (unless zeros force inconsistency).

Data & Methods

• Formal framework: classical Blackwell statistical experiments adapted to model an agent’s internal inference as prompt-dependent experiments Ep = (Sp, πp). The observer only sees the reported posteriors (marginals over requested answer partitions).
• Assumptions:
  • Finite Ω, full-support prior μ0 (unknown to observer), correctness of signal model (agent uses the true πp(·|ω) in Bayes’ rule).
  • Obedience: the agent reports the requested posterior function of its updated full-state posterior.
  • Repeated prompts are i.i.d. conditional on the fixed true state.
  • Prompts may induce arbitrarily different experiments (no structure assumed across prompts).
  • Some results additionally assume each answer class contains at least two complete states (|Ωy| ≥ 2).
• Methods: theoretical characterization proofs (constructive rationalizations where necessary) using support/compatibility arguments and Blackwell/garbling concepts; results formalized in three theorems (proofs in appendices).
• No empirical dataset—this is a purely theoretical/axiomatic contribution targeted at what can be inferred from observable report distributions.

Implications for AI Economics

• On detectability of irrationality: It is fundamentally hard to reject that an LLM or similar system is performing Bayes-consistent inference from black-box repeated reports, unless you observe jointly contradictory zero-probability reports across prompts. Metrics based only on report frequencies, confidence, or variability are insufficient to falsify Bayesian updating.
• For model/economist comparisons of "intelligence": Empirical comparisons based on reported confidence or variance do not establish that one agent has access to strictly more informative internal signals. Only consistent nesting of zero-claims (hard exclusions) across matched reports supports a Blackwell dominance ordering.
• For combining or aggregating AI outputs (ensembles, expert aggregation, policymaking):
  • Aggregation of coarse marginal reports is under-identified: without full-state posteriors (or the prior and more structure) any Bayesian aggregate is only pinned down on which answers must be excluded; weights among remaining answers are arbitrary.
  • Simple averaging of reported probabilities can be Bayesian-infeasible if reports differ by exclusions; practitioners should avoid blind arithmetic averaging when some reports (or agents) put exact zero(s) on outcomes.
  • Affine log-odds pooling with prior correction is a principled aggregation compatible with the constraints and is optimal in some parametric settings (Gaussian intelligence model in the paper). When full-state posteriors are elicited, a prior-corrected sum of log odds recovers the Bayesian aggregate (but it requires knowledge of the prior).
• For experimental design and evaluation of LLMs in economic tasks:
  • To meaningfully test Bayes-consistency, elicit full posteriors over sufficiently rich state partitions or design prompts that force contradictory hard exclusions if the model is non-Bayesian.
  • Avoid solely eliciting coarse marginals when the goal is to validate aggregation or to compare internal informativeness.
  • Adaptive prompting and repeated queries do not introduce martingale restrictions detectable at a fixed state; focus instead on cross-prompt support inconsistencies.
• Limitations and cautions:
  • The model assumes obedience and correct specification of the agent’s likelihoods; real systems may misreport, be strategic, or have misspecified internal models—these behaviors are outside the paper’s rationalization.
  • The prior is unobserved; many aggregation procedures require the prior to recover a unique Bayesian aggregate.
  • Theorems apply to finite state spaces and rely on some coarseness assumptions for certain results; practical state spaces (e.g., language-model belief about reality) are huge, so implementing full-state elicitation is challenging.

Overall, the paper formalizes strong identification limits when interpreting LLM/agent reports from outside: apart from detecting impossible joint zeros (hard exclusions), observers have little leverage to refute Bayesian updating or to rank informativeness or uniquely aggregate coarse reports without extra structure or richer elicitation.