Backtests overstate LLM forecasting skill: prompt 'fake date' tests show lookahead bias in every modern model examined. The finding implies in-sample accuracy of LLM-based macro forecasts cannot be naively extrapolated to real out-of-sample performance.

Main Finding

The paper proposes simple prompt-sensitivity checks (the "fake date tests") that detect two biases—lookahead bias and context bias—in LLM-based retrospective macroeconomic forecasting. Empirically, none of the modern LLMs tested (under the fairly strict prompts used) passed either Fake Date Test I or II, implying that in‑sample accuracy obtained by backtesting these LLMs can be misleading.

Key Points

• Problem: Many papers evaluate LLM forecasting by asking a fixed (pretrained) LLM to backcast/forecast with a prompt that tells the model to ignore post-cutoff information. Because LLMs were trained on large historical corpora, those prompts may not prevent the model from implicitly using future information (lookahead) or richer context present in the training set (context bias).
• Two biases defined:
  • Lookahead bias: the model uses information from after the stated cutoff date (data leakage from the future).
  • Context bias: the model has richer/up-to-date context around in-sample periods (e.g., policy narratives, event descriptions) that will not be available for true out-of-sample forecasting.
• Test family idea: if prompts truly restrict an LLM to the stated information set, model forecasts should be insensitive to certain changes in dates in the prompt. Significant sensitivity signals bias.
• Fake Date Test I (lookahead probe): hold the cutoff and economic-statistics prompt constant but compare forecast distributions for (a) a “real” historical tcurrent (with tforecast = tcurrent + h) and (b) a fake tcurrent/fake tforecast shifted beyond the model release. If the model correctly ignores post-cutoff data, the two distributions should match. Mismatch ⇒ lookahead bias (or other prompt-following failure).
• Fake Date Test II (context probe): move the cutoff to coincide with the forecast date (so the cutoff becomes very recent relative to tcurrent). If Test I is passed, mismatch now indicates reliance on up-to-cutoff context (context bias); if Test I failed, a mismatch still indicates at least one of the biases.
• Result: For the authors’ strict prompts and supplied statistics, none of the tested modern LLMs passed either test. Failing either test casts serious doubt on in-sample accuracy being informative about out-of-sample performance.
• Important caveat: Passing the tests does not guarantee absence of bias; failing the tests strongly suggests problematic in-sample evaluation.

Data & Methods

• Formal setup:
  • Model MT is trained on information set IT. Prompt x combines: current date (tcurrent), forecast date(s) (tforecast), cutoff (tcutoff), variables to forecast (vforecast), economic statistics/context (scurrent), and linking text (ltext).
  • Model output y = MT(x, ε), where ε captures stochasticity (sampling temperature, implementation randomness).
  • For each prompt variant k, authors draw N forecast samples to estimate forecast distributions.
• Fake Date Test I:
  • Generate N samples for a historical prompt with tcurrent = tretro, tforecast = tretro + h, and tcutoff = tretro − d (cutoff placed sufficiently in the past to remove contemporary context).
  • Generate N samples for a fake prompt where only tcurrent and tforecast are shifted far into the future (beyond the model’s release), but the cutoff and supplied statistics remain the same.
  • Compare the two empirical distributions (aggregation s(·) and comparison metric h(·)). Under assumptions (A1) measurability w.r.t. scurrent ∪ I_tcutoff and (A2) time-invariance of distribution given the same cutoff/info, the distributions should coincide.
• Fake Date Test II:
  • Similar sampling but change the cutoff so it coincides with the forecast date; compare to the baseline used in Test I to reveal context dependence.
• Implementation details & examples:
  • Prompts explicitly instruct the model: “Do not use any knowledge after [tcutoff] except the statistics provided.”
  • Example prompt and model response shown (Kimi-K2 Instruct illustrated).
  • Tests use strict prompts and multiple draws to estimate full forecast distributions rather than single point outputs.
• Evaluation:
  • The paper frames comparisons as distribution-equality checks (practically implemented via aggregated statistics and tests). Authors note various practical issues: model non-determinism, ambiguous/unknown model cutoffs, retrieval-augmentation or tool use, memorization of time-specific narratives.
• Empirical scope:
  • Authors applied tests to several modern LLMs (unnamed in the excerpt except an example output). Under their settings, none passed. The paper compares its approach to related tests in the literature (e.g., event-based lookahead tests, recall/memorization probes) and argues the fake date tests are more general for forecasting contexts.

Implications for AI Economics

• Do not trust in-sample/backtest accuracy delivered by pretrained LLMs without sanity checks. In-sample performance can be inflated by lookahead/contextual leakage.
• Required researcher practices:
  • Run Fake Date Test I and II (or similar prompt-sensitivity distribution checks) as a routine pre-publication sanity check when evaluating LLMs on retrospective forecasting tasks.
  • Prefer genuine out-of-sample evaluation (periods after the model release) whenever possible.
  • If in-sample testing is unavoidable, ensure stronger controls: deterministic model settings (or many draws with statistical tests), explicit checks for retrieval/tooling, and disclosure of model version and known cutoff.
  • Consider dynamic retraining or building dated model variants (year-limited models) for honest pseudo‑real‑time backtests—acknowledging higher cost.
  • For expectation-elicitation studies with many heterogeneous agents, adapt the tests to aggregated or stratified designs to control token costs.
• Model-design and policy suggestions:
  • Developers should make model cutoffs and any retrieval/online-augmentation transparent to facilitate proper economic evaluation.
  • Forecasting studies should report robustness to date/fake-date perturbations and distributional comparisons, not just pointwise in-sample errors.
• Research directions:
  • Refine statistical implementations of the tests (which distributional tests, how many draws N, temperature settings).
  • Explore prompt designs or training procedures that reduce lookahead/context bias.
  • Study how retrieval-augmented inference or tool-use (with correctly implemented cutoffs) affects these biases in practice.

Summary conclusion: The fake date tests are simple, low-cost diagnostics that expose when LLM-based retrospective forecasting is likely contaminated by future data or by richer historical context in the training set. Their routine use would substantially strengthen the credibility of LLM forecasting research in macroeconomics.