Language models often give logically inconsistent probability forecasts—GPT-OSS-120B allowed an average guaranteed arbitrage profit of about 0.21% per unit stake while weaker models permitted much larger exploits (up to ~20%). Rich logical links between events and irrelevant prompt details substantially increase incoherence.

Main Finding

Language models commonly produce probabilistic forecasts that are internally incoherent: their stated probabilities admit Dutch books (arbitrage) of economically meaningful size. Coherence can be measured without observing realized outcomes by applying de Finetti’s Dutch‑book characterization and solving a linear program that finds the maximal worst‑case arbitrage profit against the model’s quoted probabilities.

Key Points

• Coherence test and metric

  • The paper uses de Finetti’s theorem: a vector of event probabilities is coherent iff no Dutch book exists.
  • They operationalize incoherence as the maximal worst‑case profit per unit gross stake an arbitrageur could guarantee:
  • ℓ∞(p) = max_{||b||1 ≤ 1} min_{atoms ω} Σ_i b_i (1_{Ei contains ω} − p_i).
  • Dual interpretation: ℓ∞(p) equals the sup‑norm distance from p to the set of coherent forecasts (i.e., projection distance to some distribution over atoms).
  • This test is label‑free (does not require realized outcomes), exhaustive for a fixed event set, and computed by linear programming.

• Empirical design

  • Domain: stock returns (binary/binned events tied to returns normalized by a 60‑day std), chosen because binary option–style events are natural and the logical structure of event sets is known.
  • Sample: 100 stock‑days selected from CRSP with matched Refinitiv headlines; horizons up to 4 days and nested event panels built from 4 return bins (edges at −1, 0, 1 in units of past‑std).
  • Elicitations: 365,100 total elicited probabilities across 15 models and many prompt/elicitation variants. Baseline: single‑event queries per session, 14 nontrivial events at horizon 1.
  • Models: 15 open/closed‑weight models; GPT‑OSS‑120B used as the workhorse (5 passes averaged per stock‑day); other models usually run with one pass.

• Main empirical findings

  • Coherence is routinely violated. For GPT‑OSS‑120B (baseline, horizon 1), mean arbitrage profit = 0.002067 per unit gross stake (95% CI [0.001471, 0.002737]). Rounding cannot explain this magnitude.
  • Across models incoherence varies widely: arbitrage profit ranged ≈ 0.0021 to 0.1994 (factor ~100), while accuracy (Brier score) varied little. Models that are more coherent also tend to be more accurate (Spearman ≈ 0.91), but coherence differences are much larger in relative terms and more statistically separable.
  • Logical complexity matters: adding joint events that introduce logical dependencies (across assets or across days) substantially increases incoherence. E.g., including joint events for two stocks raises mean arbitrage profit relative to marginals.
  • Elicitation/context sensitivity:
  • Providing simple summary statistics (bin frequencies) reduced incoherence.
  • Adding procedural instructions, naming the company, or showing a news headline often increased incoherence—sometimes by an order of magnitude for irrelevant contextual detail.
  • Grouping related questions (e.g., asking event + complement together) tends to reduce incoherence relative to asking unrelated groups or isolated single queries.
  • Averaging multiple repeated elicitations (“take K runs and average”) produced only small reductions in incoherence.
  • Results are robust to bootstrap inference clustered on date.

Data & Methods

• Events and atoms
  • Events defined from one‑day (and multi‑day) return bins relative to prior 60‑day standard deviation; the algebra generated by the queried events partitions outcomes into atoms; the incidence matrix M maps atoms to events.
• Incoherence computation
  • For a given vector p of model probabilities on n events and incidence matrix M (m atoms), solve a linear program:
  • primal: maximize worst‑case payoff per unit gross stake by choosing stake vector b with L1 norm ≤ 1;
  • dual: compute min_{π ∈ Δ_{m−1}} ||Mπ − p||_∞ (distance to coherent set).
  • Practically: compute ℓ∞(p) for each elicitation pass; average over five passes for the main model; single pass for many other models for cost control.
• Sample and elicitation arms
  • 100 stock‑days (drawn from Aug 15–Dec 23, 2025), 98 distinct stocks, matched headlines.
  • Baseline: 14 nontrivial horizon‑1 events per stock; extensions include multi‑asset/multi‑day joint events, grouped queries, information variants (statistics, instructions, identity, headline), red‑herring/irrelevant context arms, and averaging arms.
• Inference
  • 95% percentile bootstrap confidence intervals, clustered on date, 20,000 replicates.

Implications for AI Economics

• Evaluation advantage: coherence is a label‑free, structural diagnostic of model probabilistic forecasts. This makes it valuable for settings where outcomes are unavailable, slow to arrive, or manipulable.
• Decision‑making and market use
  • If practitioners treat model‑quoted probabilities as prices or as inputs into decisions (trading, insurance, automated allocation), incoherence can be exploited or lead to systematically inconsistent decisions.
  • In prediction‑market style uses or programmatic trading, incoherent probabilities imply model outputs could be arbitraged if used as tradable prices; even absent tradability, incoherence signals unreliable internal world models.
• Prompting and deployment
  • Prompt design materially affects coherence. Designers should be cautious about adding irrelevant or evocative context (headlines, identity) that can degrade internal consistency.
  • Grouping logically related questions (asking complements or joint events together) helps reduce incoherence and should be used in elicitation protocols when coherent probability judgments matter.
  • Simple averaging of multiple responses is insufficient to remove incoherence.
• Training and mitigation strategies
  • Potential approaches to improve coherence:
  • Train or fine‑tune models with explicit coherence regularizers (e.g., penalize Dutch‑book exploitable patterns, add adversarial Dutch‑book loss).
  • Multi‑task training that exposes models to many logically related events so the algebraic constraints are learned.
  • Post‑hoc projection: map model p to the nearest coherent probability vector (minimize sup‑norm or other norm to the coherent set) before use.
  • Constrained decoding or consistency‑enforcing modules that ensure event algebra constraints (complements sum to 1, unions/intersections consistent).
  • These strategies trade off calibration, sharpness, and potentially accuracy; the paper discusses such tradeoffs and the fact that coherence ≠ accuracy.
• Policy and risk
  • For high‑stakes decision systems, incoherent probabilistic outputs are a reliability failure mode; regulators and auditors could require coherence checks as part of model validation.
  • Economic models of automation that incorporate model forecasts should account for the possibility that forecasts violate probability axioms, which can affect incentives, contracts, and market equilibria if models are trusted as probabilistic agents.

Limitations and caveats - Coherence depends on the set of queried events; a model can be coherent on one set and incoherent on another. The metric increases weakly as events are added. - Coherence is distinct from accuracy: a dogmatic but wrong forecaster can be coherent. The test diagnoses internal logical consistency, not epistemic correctness. - Domain choice (stock returns) is convenient and economically relevant but results may vary by domain and event construction. - Some models’ knowledge cutoffs overlap the sample period; while incoherence measurement is unaffected by lookahead, accuracy comparisons may be biased.

Concise takeaway Language models’ probability outputs are often internally inconsistent in economically meaningful ways; these inconsistencies are measurable without outcomes via Dutch‑book LPs, vary strongly with modeling and elicitation choices, and suggest concrete training, prompting, and validation interventions for safer and more reliable probabilistic forecasting.