A specialized live benchmark shows state-of-the-art agentic LLMs handle open-domain search but lack the accuracy and domain grounding required for high-value vertical prediction tasks such as market forecasting, supply-chain demand, epidemic tracking and disaster monitoring; the gap raises caution for immediate industrial deployment.

Main Finding

FutureX-Pro extends the original FutureX live, contamination-free benchmark into high-value verticals (Finance, Retail, Public Health, Natural Disaster) plus a retrieval-oriented FutureX-Search. Across domain-specific, entry-level prediction tasks, state-of-the-art agentic LLMs (e.g., GPT-5 variants, Grok-4) show competence in broad reasoning and calibrated uncertainty but fall short of the numerical precision, reliability, and domain grounding required for capital-intensive or safety-critical industrial deployment.

Key Points

• Scope and motivation
  • Focuses on four verticals with high economic or societal impact: Finance (market indicators), Retail (demand forecasting), Public Health (epidemiological metrics), Natural Disaster (hazard impacts). Adds FutureX-Search to turn resolved events into pure retrieval tasks.
  • Uses a contamination-impossible live-evaluation pipeline to ensure verifiable, out-of-sample future events.
• Finance (FutureX-Finance)
  • Dataset: 150 equities (100 US — NASDAQ/S&P constituents; 50 China A-shares).
  • Tasks: Spot prediction (single-day exact value), Window extremum (max absolute value across N trading days), Directional momentum (largest positive change in window).
  • Metric: High-sensitivity linear penalty S = max(0, 1 − 20 × |ŷ − y| / |y|). Errors >5% score zero (strict tolerance).
  • Result highlights: Performance tiers emerged; GPT-5-High and Grok-4 lead (Type 1 scores ≈46.37 and 41.25 respectively). No model averaged above 50 — indicating substantial difficulty converting information into <5% numerical forecasts.
• Retail (FutureX-Retail)
  • Dataset: 240 cross-border e-commerce products (4 major categories, 24 subcategories) sourced from Temu.
  • Task grid: 2 (input conditions: snapshot-only vs. sparse time-series) × 3 (output granularity: point, top‑K probabilistic, full distribution).
  • Scoring: Hybrid — linear-decay relative-error metric for exact ground truth (ε = 0.05) and binary range checks for coarse badge-like counts (>999). Probabilistic tasks scored as expected value under predicted distribution.
  • Results highlights:
    • "Probabilistic advantage": models often score better when asked for distributions (Top‑3 or full distribution) than single-point estimates.
    • Historical context substantially improves accuracy.
    • In-context examples strongly influence output granularity and reported confidence (models mimic example counts/probabilities).
    • Model personalities: Grok-4 tends to output long-tail distributions; GPT-series tends to be more conservative/calibrated.
• Public Health (FutureX-PublicHealth)
  • Data strictly from authoritative national bulletins (US CDC, China CDC).
  • Scope: 70 weekly event templates → 382 variables with high temporal frequency (weekly).
  • Goal: Evaluate agents on structured epidemiological report interpretation and near-term metric tracking (high temporal resolution).
• FutureX-Search
  • Transforms predictive tasks with known outcomes into retrieval/search challenges to benchmark pure information retrieval and grounding separately from forecasting ability.
• Models evaluated
  • Broad evaluation across ~18 leading agentic LLMs (GPT-5.1/GPT-5 variants, Grok-4, Claude-Opus, Kimi-K2 variants, Qwen3-Max, DeepSeek, GLM, etc.). SOTA models show relative stability but absolute precision remains limited for high-stakes thresholds.

Data & Methods

• Contamination-free live evaluation
  • All tasks are constructed to be verifiably about future events so model training data contamination is effectively prevented.
  • Live updates and a weekly competition pipeline (Hugging Face dataset integration).
• Domain datasets and design
  • Finance: 150 tracked companies, sector-balanced across US/China; tasks mimic junior quantitative analyst workflows.
  • Retail: 240 products, HTML snapshots (T−7) and sparse time-series input (T−14) used to simulate proprietary/ephemeral sales data scenarios.
  • Public Health: Official weekly bulletins only; fine-grained demographic and geographic stratifications.
  • Natural Disaster and FutureX-Search are defined at the benchmark level (Natural Disaster aims at meteorological/hazard forecasts; FutureX-Search uses historical resolved events), though full construction details are in the complete report and prior FutureX methodology.
• Task formats and evaluation
  • Finance: strict linear penalty with 5% tolerance cutoff; distinct task types to probe point vs. temporal/extrema reasoning.
  • Retail: deterministic, top‑K probabilistic, and full-distribution forecasts; scoring accounts for exact vs. coarse labels and rewards proper probability mass assignment.
  • Public Health: high-frequency weekly variables; evaluation focuses on correct recent-period identification and numerical tracking.
• Experimental protocol
  • Agents were given search tools in many evaluations to simulate agentic behavior (retrieval + reasoning).
  • Analyses included ablations on historical context, in-context learning sensitivity (few-shot example design), and cross-model behavior comparisons.

Implications for AI Economics

• Economic value vs. required precision
  • High-value verticals impose narrow precision requirements (e.g., <5% for market predictions) where current SOTA models are not yet reliable; errors translate into real economic losses if used directly for trading, inventory optimization, or health responses.
  • The observed "probabilistic advantage" suggests commercial value in supplying calibrated distributions (risk-aware decision support) rather than brittle point predictions—this is especially useful for inventory managers, risk desks, and public-health triage.
• Automation opportunity and limits
  • Agents show promise for junior-analyst augmentation (information synthesis, preliminary screening, uncertainty quantification) but not for fully autonomous decision-making in capital- or safety-critical workflows without additional safeguards, domain-specific models, or human oversight.
• Importance of domain grounding and authoritative data
  • Performance gains from historical/contextual inputs and from access to authoritative structured sources imply that firms should prioritize integrating vetted domain data pipelines, specialized tools (time-series modules, epidemiological models), and domain-specific fine-tuning to reach production-grade reliability.
• Market and policy considerations
  • Firms deploying agentic LLMs in finance, health, or disaster response will need robust model evaluation, explainability, and regulatory-compliant validation processes; live contamination-free benchmarks like FutureX-Pro can serve as an independent validation tool.
  • Policymakers should treat generalist-agent outputs as decision-support (not authoritative) and consider standards for model accuracy, auditability, and human-in-the-loop requirements in high-stakes domains.
• Research directions that matter economically
  • Improving numerical precision, temporal-extrema reasoning, and robust calibration under sparse/proprietary data regimes.
  • Hybrid architectures combining LLM reasoning with domain-specific model components (statistical time-series, epidemiological simulators, meteorological ensembles).
  • Better in-context and tool-use protocols to reduce sensitivity to example design and to stabilize outputs for operational adoption.
• Value of benchmarks
  • FutureX-Pro demonstrates that live, uncontaminated, domain-focused benchmarks are essential for measuring economically relevant progress and for de-risking adoption decisions by firms and regulators.

If you want, I can: - Produce a one-page slide-ready summary of this benchmark for stakeholders (investors, product managers, regulators). - Extract a short list of recommended operational guards and engineering practices for deploying agentic LLMs in each vertical.