Language-model agents can enrich forecasting by retrieving evidence and orchestrating tools, yet robust gains over specialized time‑series models remain unproven; measurement, contamination, and missing cost reporting prevent decisive conclusions.

Main Finding

LLM-based forecasting agents—ranging from prompt-only LLM forecasters to tool-augmented, multi-step agentic systems—offer new capabilities (retrieval, reasoning over unstructured evidence, tool use, iterative refinement) but do not yet consistently outperform specialized time-series or domain models on numerical extrapolation. Their value is context-dependent: language components help when predictive signals live in unstructured, timestamped evidence (news, filings, policy, outbreak reports) that are not fully captured by historical numerical series, but they add substantial inference cost, introduce new evaluation challenges (contamination, leakage), and sometimes do not improve or even harm predictive accuracy (ablation evidence).

Key Points

• Taxonomy and scope
  • Places LLM forecasters on a continuum of agency: prompt-only → retrieval-augmented → tool-using agents → hybrid controllers coordinating specialist models.
  • Operational definition: an LLM counts only if it contributes to a verifiable, scored prediction (point, distribution, event probability).
• Unified formalism
  • Information set I_t, forecasting query q, admissibility requirement (forecast must be measurable w.r.t. I_t).
  • Agent tuple (π_θ, A, T, M, g): policy, action space, tools, memory, readout; reason–act loop until readout produces a forecast.
• Empirical findings and limits
  • No clear evidence that LLMs extrapolate numerical sequences better than specialized forecasting models.
  • LLMs can approach human forecaster accuracy on some event forecasting tasks when retrieval and reasoning are used.
  • Ablations show cases where removing the LLM leaves or improves accuracy; inference-costs often unreported or large.
• Evaluation challenges
  • Feature leakage (lookahead covariates) and parametric leakage (pretraining contamination) are pervasive risks.
  • Proper scoring rules matter (Brier, log score, CRPS, pinball); calibration is a central concern.
  • Live or contamination-resistant benchmarks are necessary to credibly demonstrate temporal reasoning gains.
  • Aggregation, cost reporting, and handling feedback/reflexivity between forecasts and outcomes are underdeveloped.
• Applications and domain constraints
  • Finance, weather, health, energy, operations: each domain has different baselines, latency/cost constraints, and asymmetric failure costs.
  • In many domains (weather, health), specialist baselines remain strong; LLMs add interpretability and flexible evidence use but not clear accuracy wins.
• Research agenda proposed
  • Calibration under distribution shift, contamination-resistant live evaluation, explicit cost+accuracy reporting, methods for handling deployed-forecast feedback loops, and more robust ablation standards.

Data & Methods

• Corpus and review approach
  • Living survey constructed from scholarly databases + backward citation of core forecasting literature; screened for empirical contributions or relevant benchmarks.
  • Emphasizes reproducibility concerns and disclosure gaps (e.g., missing inference-cost reporting).
• Formal methods summarized
  • Forecasting tasks: continuous targets (predictive distributions; CRPS/pinball/MAE/RMSE), event targets (probabilities; Brier/log).
  • Admissibility: forecasts must use only I_t; leakage modes defined (feature, parametric).
  • Agent formalism: state s_k, policy π_θ, action set A (retrieve, call tools, write memory, delegate, emit), transition u, readout g(s_K).
• Architectures reviewed
  • Prompt-only models and serialisation methods (e.g., LLMTime, TOKON approaches).
  • Retrieval-augmented forecasters (FLAIRR-TS, TimeRAF), open-web research agents, program-synthesis tools (SEA-TS).
  • Hybrid systems pairing LLMs with time-series foundation models (Chronos, Moirai, Time-MoE) or statistical modules (Kalman, LSTM hybrids).
  • Multi-agent deliberation and controller architectures that orchestrate specialist forecasters.
• Training & adaptation strategies
  • Efficient tuning / prompt tuning / post-training on temporal data (GPT4TS, Time-LLM, TEMPO).
  • Reinforcement learning and reward-based tuning for forecasting objectives (Time-R1, TimeMaster).
  • In-context learning and retrieval-based conditioning vs. weight updates.
• Evaluation datasets and benchmarks (representative)
  • Series benchmarks: M4, M5, Monash, GIFT-Eval, TFRBench.
  • Event benchmarks & live tests: ForecastBench, FutureX, MIRAI; COVID Hub for epidemics.
  • Domain-specific datasets in finance, weather (GraphCast, Pangu), health, energy, operations.
• Diagnostics and controls emphasized
  • Leakage testing, hindcast replay, contamination controls, calibration assessment (reliability diagrams, conformal methods), reporting of inference cost and latency.

Implications for AI Economics

• Cost–benefit and adoption decisions
  • Economic adoption of LLM-based forecasters requires explicit joint reporting of accuracy and inference/operational cost; high compute and integration costs may outweigh modest or uncertain forecast gains.
  • Firms should demand live, contamination-resistant evaluation before deploying LLM agents for high-stakes forecasting (trading, policy response).
• Market effects and reflexivity
  • Agentic forecasters that act or inform many agents can introduce feedback loops: forecasts affect behaviors, which alter the very outcomes predicted (reflexivity). Models and regulators must account for endogeneity and perverse equilibria.
  • Transparency about sources, timing, and tooling matters for market fairness and to limit privileged "tool advantage" (tool privilege).
• Regulatory and competitive concerns
  • Pretraining contamination and undisclosed data sources create information asymmetries; regulators may require provenance, audit trails, and live-evaluation standards for deployed forecasting agents in sensitive domains.
  • Reporting standards (cost, datasets, leakage controls) should be adopted industry-wide to enable fair comparison and auditability.
• Research & investment priorities
  • Invest in contamination-resistant live benchmarks and infrastructure for ongoing calibration monitoring under distribution shift.
  • Develop lightweight hybrid architectures that use LLMs where unstructured evidence adds value but leave heavy numeric extrapolation to specialized models—this can optimize ROI.
  • Study market-level effects of widespread deployment (e.g., liquidity, volatility, common-mode errors) and design mechanisms (diversification, ensemble governance) to mitigate systemic risks.
• Decision-making and incentives
  • Organizations should treat language-based forecasts as complementary signal channels (news, filings, policy interpretation) rather than wholesale replacements of quantitative models unless robust, domain-specific evidence supports the switch.
  • Incentives for forecasters should emphasize calibrated probabilistic forecasts and conditional policies that account for forecast-driven behavioral changes.

Overall, the survey highlights both promise and substantial measurement/deployment challenges. For AI economics, the central takeaways are that LLM agents can add value when they ingest timely unstructured signals, but economic deployment demands rigorous evaluation (live, contamination-resistant), explicit cost-accounting, calibration under shift, and governance to manage reflexivity and systemic effects.