0 cumulative citations
View corpus contextA new finance time-series benchmark of 116,897 series finds that strong performance on generic forecasting metrics does not reliably translate into decision-relevant gains for financial tasks such as directionality, cross-sectional ranking, or backtested portfolio returns; domain-aware, economically motivated metrics are necessary to evaluate practical usefulness.
FinVerse: Financial Time-Series Benchmark
FinVerse is a large finance-focused time-series benchmark that assigns decision-oriented, domain-aware metrics to individual series and shows that models performing well on generic error metrics often fail to deliver useful financial forecasts in directional, ranking, and portfolio terms.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
As time-series foundation models have emerged, the need for benchmarks that can evaluate their forecasting ability in meaningful ways has become increasingly important. Existing time-series forecasting benchmarks provide useful standardized comparisons, but they often evaluate heterogeneous series with uniform error-based metrics. Strong performance under such metrics does not necessarily imply that a model's forecasts will support the best real-world decisions across domains. For example, in stock forecasting, correctly predicting whether a price will rise or fall can be more directly relevant to realized returns than minimizing point-wise forecast error alone. To this end, we introduce FinVerse, a finance-domain time-series forecasting benchmark that takes a first step toward more realistic evaluation. The released FinVerse data artifact contains 116,897 financial time series with 171.1M observations, of which 60,232 series with 17.4M observations are selected as evaluated targets based on their economic relevance to financial decisions. Unlike generic forecasting benchmarks that primarily emphasize uniform point-forecast or probabilistic accuracy, FinVerse defines 11 metric families comprising 78 evaluation metrics and assigns the most appropriate evaluation metrics to each individual time series based on its underlying economic meaning. Our analysis of 43 public time-series forecasting foundation models shows that strong performance under generic forecasting criteria does not necessarily translate into useful financial forecasts. This finding highlights the need for domain-aware benchmarks that evaluate models under objectives closer to real-world decision making.
Summary
Main Finding
FinVerse is a large, finance-focused time-series forecasting benchmark that replaces one-size-fits-all error metrics with decision-oriented, domain-aware evaluation. By assigning appropriate evaluation metrics (11 metric families, 78 metrics) to each financial series according to its economic meaning, FinVerse shows that models that excel on generic forecasting benchmarks often do not produce forecasts that are useful for financial decision making (e.g., direction, ranking, portfolio returns).
Key Points
-
Scope and scale
- Full artifact: 116,897 financial time series, 171.1M observations across 3 scopes (inter-country, country, individual), 13 categories and 101 detail categories.
- Evaluation set: 60,232 target series with 17.4M observations selected manually for economic relevance.
- Training/pretraining pool (pre-2020-01-01): 66,134 series, 100.3M observations.
- Frequencies supported: daily, business-daily, weekly, monthly, quarterly, annual.
-
Evaluation philosophy
- Each series is evaluated using metrics chosen to reflect decision contexts implied by the series’ economic meaning rather than uniform error minimization.
- Three high-level evaluation views: point-wise forecast quality, cross-sectional ranking quality, and realized portfolio backtesting.
-
Metrics and examples
- Point-wise / directional: Hit Ratio (HR) plus first- and second-order derivative hit ratios (∆HR, ∆2HR) for direction and acceleration/deceleration.
- Scale-aware error: three MASE variants tailored to finance
- MASErel (relative-change baseline; evaluates returns),
- MASEabs (raw-value baseline over comparison window),
- MASEori (origin persistence baseline over horizon).
- Cross-sectional ranking: Information Coefficient (IC) — Spearman rank correlation of predicted vs realized changes.
- Portfolio/backtest: equal-weight long-only strategy selecting top 10% predicted assets; evaluate annualized return, volatility, Sharpe ratio, max drawdown averaged across all rebalancing offsets.
- In total: 11 metric families, 78 concrete metrics; metrics are assigned per detail-category–frequency combination (tables in appendix).
-
Empirical finding
- Evaluation of 43 public time-series foundation models (representative recent models include TimesFM, Chronos, Moirai, Timer, TimePFN, MOMENT classes) reveals that strong performance on generic error-based benchmarks does not guarantee useful financial forecasts under decision-aligned metrics (directional accuracy, ranking, portfolio outcomes).
-
Practical artifacts
- Fixed evaluation start date: 2020-01-01.
- Frequency-specific prediction lengths and minimum lookbacks: daily 365/365, business-daily 260/260, weekly 52/52, monthly 12/12, quarterly 4/4, annual prediction length 1 with minimum lookback 4.
- Public leaderboard and data artifact: hosted (link in paper — Hugging Face space).
Data & Methods
-
Dataset construction
- Hierarchical taxonomy: Scope → Category → Detail Category; metrics are assigned per detail-category–frequency unit.
- Manual selection of evaluation targets based on economic relevance, leaving a larger historical pool for pretraining/fine-tuning.
-
Forecast task setup
- Forecast origins O defined starting from evaluation start date.
- Comparison windows p and horizons h are chosen to match semantics of each series (e.g., week-over-week, month-over-month, year-over-year).
- Forecast horizons and lookback minima set by frequency (see table above).
-
Metric definitions (compact)
- Hit Ratio (HR): fraction of times predicted direction matches realized direction for a chosen (p,h). Extensions ∆HR and ∆2HR capture first and second derivative directional correctness.
- MASE variants: finance-adapted scale-normalized absolute error measures using persistence/no-change baselines appropriate to the target (relative, absolute, origin).
- Information Coefficient (IC): Spearman rank correlation across an asset set So at each origin.
- Portfolio backtest: At each rebalance origin, rank assets by predicted horizon signal, pick top 10% (equal-weight long-only), hold for h steps, average results across all offsets within the h-step window to avoid start-date bias. Summarize with annualized return, annualized volatility, Sharpe, max drawdown.
-
Baselines & model evaluation
- Benchmarked 43 public time-series foundation models; architectures skew toward decoder-only Transformers in recent literature.
- Comparison demonstrates divergence between generic error metrics and finance-centered objectives.
-
Reproducibility
- Metric assignments, mapping tables, and dataset splits are provided in the release; a leaderboard for submitted models is available.
Implications for AI Economics
-
Benchmarks should align with decision utility
- For economics/finance applications, forecast evaluation must reflect downstream decisions (trading, allocation, policy) — direction, ranking, and portfolio outcomes can matter more than pointwise RMSE or generic probabilistic scores.
-
Model development and training
- Training objectives should be multi-objective or decision-aware (e.g., incorporate ranking losses, portfolio-level objectives, return-scaled errors) to produce forecasts useful in economic contexts.
- Pretraining on broad but domain-relevant financial series (as FinVerse provides) can improve transfer for finance tasks, but evaluation-aligned fine-tuning is critical.
-
Model selection and deployment
- Practitioners should pick models based on finance-aligned metrics (e.g., IC, hit ratios, backtest returns) rather than only generic forecasting benchmarks to reduce deployment risk.
- Use of decision-aligned benchmarks can reduce false confidence when models perform well on statistical metrics but poorly on economic outcomes.
-
Research directions
- Design of loss functions and architectures that optimize ranking and portfolio utility directly.
- Robustness to non-stationarity and regime shifts in finance; calibration and risk-aware prediction.
- Extensions: include transaction costs, short/long strategies, factor-neutral constraints, and richer cost-aware backtests.
- Generalizing TimeVerse approach to other domains (medicine, energy) where evaluation must align with domain decisions.
-
Limitations to note
- FinVerse focuses on finance (U.S.-centered for country/individual scopes); manual selection of evaluation targets and metric assignments introduces subjective choices.
- Current backtesting setup is equal-weight long-only and (as described) does not by default incorporate transaction costs, slippage, or leverage constraints — these matter for real-world deployment and should be layered on by users when needed.
Overall, FinVerse provides a structured, large-scale resource to push time-series foundation models toward producing economically useful forecasts by making evaluation objectives explicit and aligned with real financial decision-making.
Assessment
Paper Typedescriptive
Evidence Strengthn/a — This is a benchmark / data paper rather than a causal or explanatory study; it reports descriptive empirical comparisons of forecasting models rather than attempting causal identification.
Methods Rigormedium — Large-scale, carefully structured dataset with many frequencies, clearly defined decision-oriented metrics (78 metrics across 11 families), explicit backtesting protocol (offset averaging), and evaluation of 43 public models indicate substantial rigor; however, important design choices are manual or subjective (selection of evaluation targets and metric assignments), potential survivorship/selection biases are not fully discussed, and backtests omit transaction costs, market impact, and other realistic frictions.
SampleFinVerse data artifact: 116,897 financial time series (171.1M observations) across daily, business-daily, weekly, monthly, quarterly, and annual frequencies. Fixed evaluation set: 60,232 series (17.4M observations) chosen manually for economic relevance. Training/pretraining set: 66,134 series (100.3M observations) using history before 2020-01-01. Coverage spans inter-country (FX, commodities, crypto, indices, liquidity, risk), country-level (fixed income, macro, market indicators) and individual-instrument series (equities, ETFs, fundamentals, news sentiment), with country/individual scope centered on the U.S.
Themesinnovation adoption
GeneralizabilityCountry- and individual-level data are U.S.-centric; limited global representativeness for instrument-level conclusions., Evaluation target selection was manual and may introduce selection bias or subjective metric assignment., Potential survivorship bias in equities/ETFs/fundamentals if delisted or legacy series were excluded or not adjusted., Portfolio backtests omit transaction costs, slippage, borrow/shorting constraints and other real-market frictions., Equal-weight long-only top-10% strategy is only one possible mapping from forecasts to economic outcomes and may not reflect institutional strategies., Evaluation start date (2020-01-01) concentrates on recent regimes (including pandemic-era volatility), limiting inference to other historical regimes.
Claims (8)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| FinVerse contains 116,897 financial time series with 171.1 million observations, spanning three scopes and 13 categories. Other | positive | Benchmark data coverage |
Reading fidelity
high
Study strength
high
|
n=116897
|
| FinVerse selects 60,232 economically meaningful target series, containing 17.4 million observations, for its fixed evaluation set. Decision Quality | positive | Coverage of financially decision-relevant evaluation targets |
Reading fidelity
high
Study strength
high
|
n=60232
|
| FinVerse assigns domain-appropriate evaluation metrics to individual financial time series rather than applying a uniform set of error-based metrics to all series. Decision Quality | positive | Decision-oriented forecast evaluation |
Reading fidelity
high
Study strength
high
|
n=60232
11 metric families comprising 78 evaluation metrics
|
| Strong performance on generic time-series forecasting criteria does not necessarily translate into useful financial forecasts. Decision Quality | negative | Usefulness of financial forecasts under domain-specific objectives |
Reading fidelity
high
Study strength
medium
|
n=43
|
| Generic benchmarks centered on metrics such as MASE and CRPS cannot fully capture domain-specific financial forecasting requirements. Decision Quality | negative | Alignment between benchmark metrics and financial decision objectives |
Reading fidelity
high
Study strength
medium
|
not reported
|
| For financial asset forecasting, predicting the direction or relative ranking of future returns may be more relevant to decisions than minimizing point-wise price error. Decision Quality | positive | Financial decision usefulness of forecasts |
Reading fidelity
high
Study strength
low
|
not reported
|
| FinVerse evaluates financial forecasts through three complementary views: point-wise forecast quality, cross-sectional ranking quality, and realized portfolio performance. Decision Quality | positive | Breadth of decision-oriented forecast evaluation |
Reading fidelity
high
Study strength
high
|
3 evaluation aspects
|
| The benchmark's portfolio evaluation converts forecasts into an equal-weight, long-only strategy that selects the top 10% of assets by predicted signal. Decision Quality | positive | Realized portfolio performance |
Reading fidelity
high
Study strength
high
|
top 10% fraction
|