How you reward an LLM matters: using different proper scoring rules produces forecasters that reach similar overall accuracy via different error structures—Brier training yields the best Brier score and discrimination, while log training gives the best log score and calibration, and each reward reshapes the model’s bias, information, and noise profile.

Main Finding

Although all five reward functions compared are strictly proper scoring rules (so they incentivize truthful probability reports in expectation), using them as terminal rewards under Group Relative Policy Optimization (Dr. GRPO) produces measurably different LLM forecasters in finite training. Reward choice affected calibration, probability-scale use, discrimination, and—importantly—the estimated composition of forecasting error (bias vs information vs noise). In short: proper scoring rules are not interchangeable as finite-sample training objectives; they shape not just how well LLMs forecast but how their errors are structured.

Key Points

• Rewards compared: logarithmic (log), Brier (negative Brier score used as reward), spherical, Beta(2,8), Beta(8,8). All are strictly proper but differ in curvature, tail sensitivity, and where they emphasize probability errors.
• Aggregate performance (held-out pooled evaluation, 965 questions):
  • Best Brier score: Brier-trained model (Brier = 0.1648) vs base = 0.1861.
  • Best AUC-ROC (discrimination): Brier-trained (0.7511).
  • Best log score and lowest expected calibration error (ECE): log-trained model (log score = −0.5132, ECE = 0.0434).
  • All reward-trained variants improved over the untrained base on Brier and ECE; differences between reward conditions on aggregate metrics were present but modest.
• Error composition (bias–information–noise, BIN decomposition):
  • Different reward functions achieved similar aggregate improvements via different mixes:
  • Beta(2,8): largest estimated bias contribution (systematic probability shift).
  • Brier: largest estimated information contribution (better use of outcome-relevant information) with little noise.
  • Log: comparatively large positive noise contribution (more nonsystematic variability).
  • Spherical and Beta(8,8): showed negative noise contributions offsetting some gains.
• Probability-use and calibration:
  • Reward-trained models put substantially more mass on lower predicted probabilities versus the base model.
  • Log-trained model exhibited the best calibration curve (lowest ECE).
• Training and practical caveats:
  • Training setup held model initialization, data, optimizer, steps (200), rollout budget (8 rollouts per question), LoRA rank (32), and other hyperparameters fixed; only reward function varied.
  • Each condition used a single random seed — observed differences may partly reflect training stochasticity.
  • Realized token-generation varied widely across conditions (avg tokens per condition ranged ~362 to ~2692), so token-level compute differed despite identical step budgets.
  • Probabilities clipped to [0.001, 0.999]; invalid outputs penalized on rule-specific scales.

Data & Methods

• Task: temporal-masked binary forecasting of real-world events (Will E occur by time s?), using only evidence available before cutoff t (future-as-label setup).
• Dataset: 8,041 resolved binary forecasting examples from news (July 2024–Jan 2026); train = 7,076, held-out evaluation = 965; positive-rate ~27–28.5%.
• Model and training:
  • Base: GPT-OSS-120B; post-training via LoRA adapters (rank 32) using Dr. GRPO (group-relative policy optimization variant).
  • For each training example, K = 8 rollouts sampled; advantage for each rollout = its reward minus average reward across the group (no SD-scaling, no KL penalty).
  • Terminal rewards computed from final parsed probability and resolved outcome.
  • Training steps: 200; batch size 32; learning rate 2e-5; warmup ratio 0.1.
• Evaluation:
  • Sample five responses per held-out question (temperature 1.0); primary metrics pooled across 5 rollouts: Brier score (lower better), log score (higher better), expected calibration error (ECE, lower better), AUC-ROC (higher better).
  • Bias–information–noise decomposition (Satopää et al., 2021) fit to pooled forecasts to decompose Brier differences into bias, information, and noise components.
  • Numerical stability: probabilities clipped to [0.001, 0.999]; invalid outputs penalized to be worse than worst valid forecast for each rule.

Implications for AI Economics

• Reward design as an economic instrument: Choice of scoring rule functions like a policy lever that shapes forecasters’ systematic tendencies (bias), informativeness, and volatility (noise). In mechanism-design terms, scoring-rule selection affects the forecast distribution and therefore downstream economic outcomes (trading, betting markets, managerial decisions).
• Market and mechanism implications:
  • If a market designer wants better discrimination/ranking ability (e.g., rank-based allocation or selection), training under Brier-like objectives may be preferable (Brier-trained model had highest AUC-ROC).
  • If calibration is paramount for risk-sensitive decisions (e.g., expected-utility maximization, actuarial pricing), log scoring produced the best calibration (lowest ECE) in this study.
  • If one cares about reducing nonsystematic noise (e.g., for ensemble aggregation or policy-making where stable signals matter), reward choice that boosts information rather than noise (Brier here) matters.
• Evaluation and experiment design:
  • Reporting only aggregate metrics (Brier, log score) can miss economically important differences in how forecasts fail. BIN-style decompositions reveal differences that matter for aggregation, information markets, and welfare calculations.
  • Comparisons of forecasting systems or market participants should account for their training reward functions; two systems with equal Brier scores can have very different bias/noise profiles and hence different economic value.
• Practical guidance for researchers and practitioners:
  • Choose the scoring rule with the downstream decision loss in mind (e.g., calibration vs ranking vs tail-sensitivity).
  • When training forecasters via RL-style procedures, expect finite-sample, algorithmic interactions to produce different operational forecasters even when rules are proper.
  • Run multiple seeds and report BIN-style decompositions and calibration curves in economic evaluations; control for token-generation and compute differences.
• Broader mechanism-design caution:
  • Even when a scoring rule is proper (truthful in expectation), finite-sample training dynamics and group-relative updates can create incentives and learning signals that produce systematically different forecasting behaviors. Designers of prediction markets, forecasting contests, or corporate scoring incentives should therefore not assume interchangeability of proper scoring rules.

Suggestions for follow-up (research agenda): - Replicate with multiple random seeds and longer training to separate stochastic effects from reward-induced effects. - Investigate continuous/outcome-sensitive scoring rules, asymmetric decision losses, and multi-class forecasts. - Study downstream economic impacts (expected utility, market-maker P&L, allocation decisions) of differently trained forecasters in simulated or real decision environments.