Leading language models are systematically overconfident when forecasting real-world events: in a 300-question Kalshi benchmark most models underperform base-rate predictions and only one achieves a positive Brier Skill Score, while enhanced reasoning often raises confidence without improving calibration.

Main Finding

KalshiBench — a temporally-filtered benchmark of 300 real-world prediction-market (Kalshi) questions resolving after model cutoffs — reveals that five frontier LLMs (Claude Opus 4.5, GPT-5.2-XHigh, DeepSeek‑V3.2, Qwen3‑235B, Kimi‑K2) are systematically overconfident. Accuracy ranges modestly (64–69%) but calibration error varies widely (ECE 0.12–0.395). Only Claude Opus 4.5 marginally outperforms a base‑rate baseline (positive Brier Skill Score); most models give high-confidence (>90%) predictions that are wrong 15–32% of the time, i.e., far worse than well-calibrated behavior.

Key Points

• Benchmark design: KalshiBench uses CFTC-regulated prediction-market questions with verifiable post-training outcomes and strict temporal filtering to avoid memorization.
• Core empirical results:
  • Accuracy: 64.3%–69.3%.
  • Mean reported confidence: ~74%–82%, exceeding actual accuracy.
  • Expected Calibration Error (ECE): 0.120 (best, Claude) to 0.395 (worst, GPT‑5.2‑XHigh).
  • Brier score / Brier Skill Score: only Claude has a positive BSS (0.057); others perform worse than predicting the base rate.
  • High‑confidence failures: in the 90–100% confidence bin, average reported confidence ≈94–96% while actual accuracy ranges 30–70%, producing calibration gaps up to ~62 percentage points.
• Reasoning paradox: models with “extended reasoning” or more verbose chain-of-thought (GPT‑5.2‑XHigh, Qwen3, Kimi) show worse calibration despite similar accuracy and higher token costs.
• Calibration is decoupled from accuracy and cost: similar accuracies can correspond to 3× differences in ECE; more compute/verbosity does not guarantee better uncertainty quantification.
• Domain effects: performance and calibration vary by category (better on entertainment/sports, worse on crypto/science), suggesting dataset exposure and domain uncertainty matter.
• Prompting: models were explicitly instructed to be calibrated, yet still miscalibrated — indicating prompts alone are insufficient.

Data & Methods

• Data source: Kalshi prediction-market contracts (CFTC-regulated), full cleaned set = 1,531 questions (Sep 2021–Nov 2025). Evaluation sample: 300 questions temporally filtered to resolve after the latest model cutoff (effective cutoff Oct 1, 2025), seed=42.
• Coverage: 13 categories in the sampled set (sports, politics, entertainment, crypto, etc.), ~40% yes rate in sample.
• Temporal filtering: exclude any question resolving on or before the latest knowledge cutoff to avoid memorized outcomes.
• Models evaluated: Claude Opus 4.5 (Anthropic, Apr 2025), GPT‑5.2‑XHigh (OpenAI, Oct 2025, extended reasoning), DeepSeek‑V3.2 (open-weight, Oct 2025), Qwen3‑235B‑Thinking (Alibaba, Jun 2025), Kimi‑K2 (Moonshot, Jun 2025).
• Prompting & protocol: structured prompt requiring (, reasoning), (yes/no), and (0–100). Temperatures: 0.7 normally, 1.0 for GPT‑5.2‑XHigh per provider recommendations.
• Metrics:
  • Classification: accuracy, F1.
  • Probabilistic: Brier score, Brier Skill Score (baseline = base rate), Expected Calibration Error (ECE), Maximum Calibration Error (MCE), Overconfidence Rate (OCR at thresholds).
  • Reliability diagrams and bin-level analysis (10 bins).
• Quality control: deduplication (≤2 questions per series), use of explicit resolution criteria from contracts.
• Limitations called out by authors: binary outcomes only, moderate sample size (300), prompt sensitivity not exhaustively explored, self-reported confidence may differ from internal probabilities.

Implications for AI Economics

• Forecasting & market use:
  • Do not treat high model confidence as high reliability. LLM-produced probabilities can systematically overstate certainty and would mislead traders, risk models, or automated strategies if used naively for pricing or position sizing.
  • LLMs in market-making or automated trading: overconfident probability estimates could create persistent mispricings or ill-timed trades. Systems must avoid equating model confidence with objective probability without calibration adjustments.
• Policy, regulation, and audit:
  • When LLM outputs inform regulated economic decisions (financial advice, risk disclosures), firms and regulators should require quantified calibration checks (Brier, ECE) and domain‑specific validation before deployment.
  • Because calibration failures persist even when instructed to be calibrated, auditing should include out‑of‑sample, temporally fresh event sets (like KalshiBench) to detect hindsight leakage or overfitting to historical patterns.
• Decision-making and risk management:
  • Use calibrated surrogates or conservative transforms (probability shrinkage toward base rate) before integrating LLM forecasts into expected-value calculations or risk models.
  • Require abstention or deferral thresholds: if a model’s calibrated probability carries high expected loss when wrong, route forecasts to human experts or ensembles.
• Market design and aggregation:
  • Aggregation/ensemble strategies remain important — combining model outputs with actual market prices (which embed real money) or calibrated human forecasts can reduce overconfidence impact.
  • Proper-scoring incentivization: using proper scoring rules (Brier, log score) and monetary incentives for human raters/forecasters can help align behavior; analogous incentives could be simulated during model fine-tuning to encourage calibrated outputs.
• Research & product recommendations:
  • For economic forecasting tasks, prioritize calibration diagnostics (Brier, BSS, ECE) alongside accuracy during model selection and monitoring.
  • Employ post-hoc calibration (temperature / Platt / isotonic), domain-specific recalibration, or meta‑models that predict model error to adjust probabilities.
  • Consider training objectives or RLHF variants that explicitly penalize miscalibration (optimize calibrated scores, not just token likelihood).
  • In high-stakes deployments, prefer hybrid workflows: use LLMs for scenario generation, feature synthesis, or structured reasoning but rely on calibrated probability providers (markets, ensembles, or human experts) for final probabilities.
• Practical mitigations before deployment:
  • Validate on temporally out-of-sample datasets (KalshiBench-like) relevant to target markets.
  • Apply conservative probability shrinkage where model overconfidence is known.
  • Require human sign-off or automated checks for predictions above high‑confidence thresholds.
  • Monitor live Brier score and ECE, and retrain/recalibrate when these drift.

Overall, this work highlights that uncertainty quantification is a distinct capability needing targeted evaluation and intervention. For AI-enabled economic forecasting or market applications, calibration should be treated as a first‑class requirement, not an afterthought.