Web-enabled LLMs can nowcast the economy in real time: in a six-month live test across 16 U.S. indicators the best LLM (GPT‑5) matched the Bloomberg economist consensus on a market-weighted accuracy metric, while several models were competitive with Fed nowcasts on GDP and unemployment; however, accuracy varies widely across series and results depend on live web access and a limited evaluation window.
Can LLMs Take the Pulse of the Economy? A Real-Time Evaluation of LLM Nowcasts on Macroeconomic Indicators
In a six-month live, contamination-resistant benchmark across sixteen U.S. macro indicators, web-search-enabled LLM agents produced hourly nowcasts whose aggregate market-weighted accuracy matched the Bloomberg professional consensus (top model GPT-5 tied ECOS), though performance varied substantially by indicator.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
No provider observation is available for this paper.
Missing data, not a zero citation count.
Nowcasting headline macroeconomic indicators, i.e., estimating an indicator's value for the current reference period before its official release, is critical for monetary policy and financial markets, and central banks devote dedicated teams of expert economists to producing such estimates. Large language model (LLM) agents are a promising candidate for this task, combining broad world knowledge with real-time web search and supporting queries at higher frequency than institutional nowcasts. Evaluating their nowcasting capability is, however, challenging: headline indicators such as GDP and CPI are widely reported and likely memorized during pretraining, so any evaluation on historical releases is vulnerable to data contamination. To address this, we introduce LiveMacroEval, a live, contamination-resistant benchmark in which LLM agents produce hourly nowcasts for sixteen major U.S. macroeconomic indicators over a pre-release window closing at each official release. Nowcast quality is assessed through a LiveMacro Score against announcement-window equity returns and a LiveBetting Score from simulated Polymarket-style trading, with Federal Reserve regional-bank nowcasts, the Bloomberg ECOS professional consensus, and an auto-ARIMA baseline as comparators. Over six months with four state-of-the-art LLM agents configured with web search, aggregate nowcast accuracy is broadly comparable to the institutional and professional benchmarks, with performance varying widely across individual indicators. This highlights LLM agents' potential as real-time estimators of macroeconomic conditions.
Summary
Main Finding
LiveMacroEval, a live, contamination-resistant benchmark, shows that web-search-enabled LLM agents can nowcast major U.S. macroeconomic indicators in real time at an aggregate level comparable to professional economist benchmarks (Bloomberg ECOS and several Federal Reserve nowcasts). Performance varies substantially across individual indicators and by evaluation metric, but the leading LLM (GPT-5) ties the Bloomberg consensus on the aggregate market-relevant LiveMacro Score, and several agents are competitive with Fed nowcasts on model-driven trading (LiveBetting) for GDP and unemployment.
Key Points
- Problem and contribution
- Nowcasting (estimating a headline macro indicator before official release) is critical for policy and markets; historical evaluation risks pretraining data contamination.
- The paper introduces LiveMacroEval: a live pre-release evaluation protocol (contamination-resistant) that collects hourly LLM nowcasts for 16 headline U.S. indicators.
- Indicators covered
- 16 indicators across 4 thematic blocks: Supply & Production (e.g., Real GDP, Industrial Production, Durable Goods, ISM Manufacturing PMI); Demand & Inflation (CPI, PCE, PPI, Real PCE, Retail Sales, ISM Services PMI); Labor Market (Nonfarm Payrolls, Unemployment Rate); Housing (Building Permits, Housing Starts, New & Existing Home Sales).
- Models and baselines
- Four web-search-enabled LLM agents evaluated: GPT-5, Claude-sonnet-4.5, Qwen3-235B, Qwen3-80B.
- Benchmarks: Bloomberg ECOS consensus, five Federal Reserve regional nowcasts (Atlanta, New York, St. Louis, Cleveland, Chicago), and a univariate auto-ARIMA time-series baseline.
- Novel evaluation metrics
- LiveMacro Score: maps model-implied standardized surprises into announcement-window E-mini S&P 500 returns using pre-estimated causal weights (βi) and reports a bounded performance measure (analogous to out-of-sample R2 vs consensus). Rewards skill on market-impactful indicators.
- LiveBetting Score: simulates hourly Polymarket-style bets on numeric buckets corresponding to the model’s latest nowcast; reports cumulative betting return versus the crowd.
- Results (summary)
- Aggregate: GPT-5 ties Bloomberg ECOS on LiveMacro Score over a six-month live run.
- Heterogeneity: performance differs substantially across indicators and themes; LLMs perform better on some series (e.g., GDP, unemployment in betting metric) and worse on others.
- Live revisions by LLMs align with meaningful incoming information events in the pre-release window.
- Robustness and contamination control
- Strictly live pre-release windows prevent retrieval of the released value, avoiding pretraining contamination.
- Sanity checks show models snap to the released value only after the release (confirming the release was not available during evaluation).
- Appendix tests find no evidence LLMs copied Bloomberg consensus or Fed nowcasts in this live setup.
- Resources
- Code and data publicly available: https://github.com/LiveMacroEval and project site.
Data & Methods
- Live evaluation window
- Six-month live deployment covering release reference periods Nov 2025–Apr/May 2026 (paper reports six months).
- For each indicator and reference period: an hourly nowcasting window opens in the final week of the reference month and closes at the official release; models free to issue web searches during the window but cannot access post-release data.
- Prompting & tooling
- Each agent receives a strict prompt defining the target variable (units, transformation, whether to use headline/core, growth measure) and can call web search iteratively; agents decide what/when to search.
- Comparative nowcasts
- Economist nowcasts: Bloomberg ECOS median consensus; multiple Fed regional nowcasts (GDPNow, NY Fed Staff Nowcast, St. Louis Fed nowcast, Cleveland Fed inflation nowcast, Chicago Fed CHURN).
- Time-series baseline: auto-ARIMA fit at the start of each prediction window.
- LiveMacro Score details
- For indicator i: standardize surprise Si = (X_released − X_consensus)/σi; use pre-estimated βi linking Si to high-frequency announcement-window E-mini S&P 500 log-return r (window [−5, +30] minutes).
- For each event, compute model-implied return r̂i = βi·Ŝi; aggregate across events to compute a bounded score: LiveMacro = (SS_cons − SS_model)/(SS_cons + SS_model), where SS compares squared errors vs consensus baseline.
- βi estimated on historical data and frozen before scoring live outcomes.
- LiveBetting Score details
- Polymarket-style numeric-bucket markets used; each hour pre-release the evaluator places a $1 bet on the bucket containing the agent’s latest nowcast (buying 1/p shares at market price p). Return accrues when the release resolves the winning bucket; average net return reported across hourly bets.
- Additional checks
- Post-release continuation demonstrates models retrieve official numbers after release (sanity).
- Tests in appendices check for copying and re-estimation sensitivity (e.g., excluding COVID window when estimating βi).
Implications for AI Economics
- Practical promise
- LLM agents with web search can serve as fast, high-frequency nowcasting tools that are competitive with professional consensus on aggregate market-relevant metrics. They could augment institutional nowcasting (more frequent updates, broader unstructured-signal synthesis).
- Use in policy and markets
- Market-weighted evaluation (LiveMacro) aligns model performance with economic relevance; using these metrics helps prioritize models that forecast market-moving surprises rather than minimizing raw forecast error across heterogeneous series.
- The LiveBetting setup connects model accuracy to actionable trading outcomes and can reveal where models beat crowd expectations.
- Methodological guidance
- Live, pre-release evaluation is essential to avoid pretraining contamination for widely reported macro targets; this protocol should be a standard for future LLM assessments on real-world economic tasks.
- Combining LLMs with econometric baselines and professional nowcasts provides informative benchmarking and highlights complementary strengths.
- Cautions and next steps
- Heterogeneous per-indicator performance suggests hybrid systems (LLM + targeted econometric models) or indicator-specific calibration will likely outperform an off-the-shelf LLM-only pipeline.
- Dependence on web-search quality and proprietary retrieval pipelines matters — robustness to search failures and transparency of sources should be addressed.
- Evaluation length (six months) and model/sample scope limit long-term generalization claims; further multi-year and multi-market evaluations would strengthen conclusions.
- Ethical and governance considerations: if deployed in markets or policy contexts, ensure auditability, source-traceability, and monitoring for drift or manipulation of input signals.
Overall, LiveMacroEval demonstrates that LLM agents are a viable, high-frequency complement to traditional nowcasting methods when evaluated in a contamination-free live setting, but careful, indicator-specific validation and integration with established methods will be necessary for operational adoption.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The study uses a contamination-resistant, live evaluation with real-time web-enabled LLMs and multiple institutional and econometric baselines, which strengthens credibility. However, the evaluation covers a relatively short six-month window, four LLM configurations, and 16 U.S. indicators only, so results may be sample-specific and sensitive to market conditions and model/web configurations.
Methods Rigorhigh — The authors implement a careful, pre-registered-seeming live protocol that prevents pretraining contamination, use multiple benchmarks (Bloomberg ECOS, several Fed nowcasts, auto-ARIMA), pre-estimate and freeze event-study betas, and run robustness checks (e.g., copying tests and COVID-window exclusions). Limitations include reliance on the event-study identifying assumption, potential sensitivity to the chosen announcement window and β stability, and a modest temporal sample.
SampleSix-month live evaluation (target periods spanning late 2025 into 2026) of hourly pre-release nowcasts for sixteen U.S. headline macroeconomic indicators (GDP, CPI, PCE, retail sales, nonfarm payrolls, unemployment rate, industrial production, ISM PMIs, housing series, etc.). Four web-search-enabled LLM agents were evaluated (GPT-5, Claude-sonnet-4.5, Qwen3-235B, Qwen3-80B). Comparators: Bloomberg ECOS professional consensus, five Federal Reserve regional nowcasts (Atlanta, New York, St. Louis, Cleveland, Chicago) where available, and a univariate auto-ARIMA baseline. Evaluation metrics: LiveMacro Score (market-weighted fit to E-mini S&P 500 announcement-window returns using pre-estimated βi) and LiveBetting Score (simulated Polymarket-style hourly betting returns).
Themesadoption innovation
IdentificationLive, pre-release nowcasting with pre-estimated event-study betas: the authors map each model's predicted standardized surprise to an implied equity return using βi coefficients estimated on historical high-frequency announcement-window regressions (r = βi S_i). The key identifying assumption is the standard high-frequency event-study assumption that announcement-window equity returns respond only to the macro surprise (isolating causal effect of the surprise on returns); βi are frozen before live scoring to avoid look-ahead.
GeneralizabilityLimited to U.S. headline macro indicators; findings may not transfer to other countries with different market microstructure or data-release practices., Six-month evaluation window is short and may not capture different economic regimes or rare events., Only four LLM agents and specific web-search/tooling configurations were tested; different models, toolchains, or access limits could materially change results., Market-based metrics rely on stability of historical βi and on the assumption that announcement-window equity returns isolate macro surprises; these may vary with liquidity or concurrent news., Polymarket-style simulated betting may not reflect real-world market depth, transaction costs, or institutional constraints.
Claims (9)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| LiveMacroEval evaluates LLM agents on sixteen major U.S. macroeconomic indicators spanning supply and production, demand and inflation, the labor market, and housing. Decision Quality | positive | Accuracy of real-time forecasts for major U.S. macroeconomic indicators |
Reading fidelity
high
Study strength
high
|
n=16
|
| The study conducted a six-month live evaluation of four web-search-enabled LLM agents: GPT-5, Claude-sonnet-4.5, Qwen3-235B, and Qwen3-80B. Decision Quality | positive | LLM macroeconomic nowcasting performance |
Reading fidelity
high
Study strength
medium
|
n=4
six months
|
| Aggregate LLM nowcast accuracy was broadly comparable to institutional and professional benchmarks over the six-month evaluation. Decision Quality | null_result | Aggregate macroeconomic nowcast accuracy |
Reading fidelity
high
Study strength
medium
|
n=4
broadly comparable
|
| GPT-5 tied the Bloomberg ECOS professional consensus on the aggregate LiveMacro Score. Decision Quality | null_result | Aggregate LiveMacro Score |
Reading fidelity
high
Study strength
medium
|
n=16
tied
|
| LLM nowcasting performance varied substantially across individual macroeconomic indicators and thematic blocks. Decision Quality | mixed | Variation in nowcast performance across macroeconomic indicators |
Reading fidelity
high
Study strength
medium
|
n=16
varying widely
|
| Several LLM agents were competitive with the corresponding Federal Reserve institutional nowcast for GDP and the unemployment rate on the LiveBetting Score. Decision Quality | positive | Simulated prediction-market trading return |
Reading fidelity
high
Study strength
medium
|
n=4
competitive
|
| LLM nowcast revisions aligned with economically meaningful information events occurring during the evaluation window. Decision Quality | positive | Responsiveness of nowcast revisions to incoming economic information |
Reading fidelity
high
Study strength
low
|
n=4
aligned
|
| The benchmark’s pre-release evaluation procedure is designed to prevent the target official value from entering the LLM’s training corpus or retrieved web content before prediction. Ai Safety And Ethics | positive | Resistance to pretraining and retrieval-data contamination |
Reading fidelity
high
Study strength
high
|
n=16
pretraining-contamination-free by construction
|
| When allowed to continue forecasting for five days after an official release, the LLM predictions clustered around the released value, indicating that the agents could retrieve official numbers once they existed. Decision Quality | positive | Post-release retrieval of official macroeconomic values |
Reading fidelity
high
Study strength
low
|
n=16
five days after the release
|