An LLM-guided validation pipeline makes factor discovery more interpretable and resilient: FaVOR enforces hypothesis-level checks before backtesting and delivers stronger out-of-sample performance in 2025—achieving cumulative excess returns of ~22.25% (IR 1.53) on the CSI 500 and ~11.23% (IR 1.13) on the S&P 500 after costs—while emphasizing structural, hypothesis-aligned factors over return-optimized black-box formulas.
FaVOR: LLM-Based Agentic Framework for Factor Mining via Empirical Validation
FaVOR is an LLM-agent framework that decomposes investment hypotheses into observable conditions, validates candidate factors via distributional evidence, and integrates validated factors into trading signals that outperform several baselines in 2025 backtests on CSI 500 and S&P 500.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
No provider observation is available for this paper.
Missing data, not a zero citation count.
Traditional finance relies on experts to hand-craft factors through a principled process grounded in economic rationale. Recent LLM-based multi-agent systems have automated this process, scaling factor mining far beyond manual effort. However, these automated approaches optimize directly for returns and rarely check whether a generated factor still expresses the economic hypothesis that motivated it. We identify this inconsistency between mathematical form and economic meaning as a structural failure mode of return-oriented automation. The resulting factors blur the line between real signals and spurious correlations and break down across regime shifts. We propose FaVOR (Factor Validation through Observable Reasoning), an agentic framework that restructures factor mining around hypothesis-level evidence rather than return outcomes. In place of the standard hypothesis-to-formula leap, FaVOR enforces a three-stage consistency loop tying mathematical form to economic rationale throughout. (1) Decomposition splits a broad economic hypothesis into independent observable conditions. (2) Validation checks whether each factor reflects its intended condition. (3) Integration merges them into a composite whose structure remains interpretable. On the CSI 500 and S&P 500 in 2025, FaVOR outperforms existing baselines while remaining effective across regimes. FaVOR shows that hypothesis-grounded factor discovery produces signals that are interpretable by construction, regime-robust, and economically faithful. The code is available at https://github.com/damilab/FaVOR.
Summary
Main Finding
FaVOR (Factor Validation through Observable Reasoning) is an LLM-based, agentic pipeline for automated factor discovery that enforces hypothesis-level empirical checks before backtesting. By decomposing economic hypotheses into observable conditions, validating candidate factors against distributional evidence (OHLCV statistics) aligned to those conditions, and integrating only validated components into composite signals, FaVOR produces interpretable, regime-robust factors that outperform several ML, RL, and LLM baselines on 2025 out-of-sample tests for CSI 500 and S&P 500.
Key Points
- Problem: Existing LLM-driven factor mining often optimizes formulas for returns without verifying that the formula actually measures the intended economic mechanism; this yields spurious, non‑robust signals.
- FaVOR architecture: three-stage agentic pipeline
- Decomposition (Hypothesis → observable conditions → candidate factor formulas)
- Factor-level Validation (quantile-binned, distributional checks using OHLCV to verify each factor maps to its intended condition)
- Integration (construct joint signals from validated factors and test directional selectivity as entry strictness increases)
- Validation procedure: factor values are split into cross-sectional quintiles; for each quintile the framework computes descriptive statistics on intraday return, daily range, close position, and volume; an LLM maps textual condition to expected variable/direction but the pass/fail decision is deterministic via four explicit checks:
- systematic shift in central tendency across bins,
- non-trivial tail/spread variation,
- agreement across complementary statistics,
- semantic consistency with the observable condition.
- Integration uses Cartesian products of validated factors across conditions and a shared selectivity quantile σ. Directional Selectivity is required: as σ increases (more selective), realized average return and win rate should improve (fewer false positives). Combinations failing this structural test are discarded.
- Optimization & backtest: factor-specific entry thresholds are optimized on a validation set (Optuna with Calmar ratio objective), then fixed for an out-of-sample test period. Transaction costs are applied.
- Interpretability: factors are interpretable by construction (each maps back to an observable condition), which aims to reduce spurious correlations and improve regime robustness.
Data & Methods
- Data: daily OHLCV only. CSI 500 (Chinese A-share) from Baostock; S&P 500 from Yahoo Finance.
- Train / Validation / Test splits:
- Training: 2022-01-01 to 2023-12-31 (decomposition, validation, structural screening)
- Validation: 2024-01-01 to 2024-12-31 (threshold optimization)
- Test (out-of-sample): 2025-01-01 to 2025-12-31 (final evaluation)
- Agents in pipeline: Hypothesis agent (generates hypothesis from human insight), Observation agent (decomposes hypothesis), Factor agent (generates candidate formulas from a limited operator set), Validation agent (computes distributional evidence and applies pass/fail rules), Integration/Execution (combines validated factors, optimizes thresholds, backtests).
- Validation stats: per-bin mean/median/q10/q90/dispersion for variables X = {C−O (intraday return), H−L (range), (C−L)/(H−L) (close position), V (volume)}.
- Integration: candidate combinations are Cartesian products of validated factors for each observable condition; entry rule requires all constituent factors cross their selectivity quantile in the hypothesized polarity simultaneously.
- Optimization: Optuna to maximize Calmar ratio (annualized return / |MDD|).
- Transaction costs: CSI 500 buys 0.0005, sells 0.0015; S&P 500 sells 0.0005 (applied to backtests).
- Main out-of-sample results (test period 2025, excess returns vs benchmark; best values in bold in paper):
- CSI 500: Annualized Return (AR) = 0.2067, Information Ratio (IR) = 1.5295, Max Drawdown (MDD) = -0.0853, Cumulative Return (CR) = 0.2225
- S&P 500: AR = 0.1062, IR = 1.1315, MDD = -0.0443, CR = 0.1123
- Baselines: linear, XGBoost, LightGBM, MLP, Transformer, RL methods, and prior LLM-based systems (RD-Agent, AlphaAgent). FaVOR outperformed most baselines on IR and cumulative excess return in these tests.
- Code: authors provide repository (link in paper).
Implications for AI Economics
- Methodological: Demonstrates how LLMs can be integrated into systematic discovery pipelines while preserving domain-grounded validation. The approach shifts evaluation from pure performance optimization toward hypothesis‑level empirical consistency, addressing a major source of spuriousness in automated economic/financial discovery.
- Interpretability & robustness: By construction, factors produced by FaVOR map back to observable economic conditions, making them more interpretable and (empirically, in this study) more robust across market regimes—this supports better causal reasoning and more reliable generalization in automated economic modeling.
- Research scaling: FaVOR suggests a scalable route to expand the universe of economically meaningful factors without sacrificing econometric discipline; helpful for academic and industrial quant research where human validation is costly.
- Market effects: If widely adopted, hypothesis-verified automated factor mining could change alpha discovery dynamics—fewer transient, overfit signals and possibly slower alpha decay for structurally justified factors.
- Cautions & extensions:
- Reliance on LLMs for decomposition and semantic mapping still introduces model-dependence; quality of decomposition and semantic mapping matters.
- Validation is limited to OHLCV-based observable conditions here; broader fundamental or alternative data would require extending the validation toolkit.
- Experiments cover two markets and a single out-of-sample year (2025). Wider temporal and cross-market testing would be needed to confirm generality.
- Deterministic rule thresholds and choice of validation statistics could miss subtler economic constructs; adaptive or hierarchical validation rules may be a next step.
- Policy & reproducibility: The pipeline’s explicit hypothesis-memory and rule-based validation facilitate auditability relative to opaque return-optimized pipelines—relevant for model governance in finance.
Summary: FaVOR provides a principled middle ground—leveraging LLMs for scalable factor suggestion while enforcing deterministic, hypothesis-aligned empirical checks—yielding interpretable, economically faithful factors with improved out‑of‑sample performance in the presented tests.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper provides out-of-sample backtests on two major equity universes and compares against a range of baselines, includes transaction cost assumptions and a held-out validation period, which supports the claims about improved factor stability and returns; however, the evaluation window is short (single-year test), key appendices and exact factor counts/selection procedures are not shown in the excerpt, and backtests remain vulnerable to data-snooping, selection, and implementation biases, limiting causal or general claims.
Methods Rigormedium — The framework enforces hypothesis decomposition, explicit distributional checks, held-out optimization, and transaction-cost-aware backtesting, which are good practices; but important details (full hyperparameters, appendices, robustness checks across longer/historical regimes, sensitivity to LLM prompts/models, execution/slippage realism, and survivorship/lookahead controls) are either deferred to appendices or not evident in the supplied text.
SampleDaily OHLCV data for two equity universes: CSI 500 (Chinese A-share) from Baostock and S&P 500 (U.S.) from Yahoo Finance; data split into training (2022-01-01 to 2023-12-31), validation (2024-01-01 to 2024-12-31), and testing (2025-01-01 to 2025-12-31); backtests use only daily open/high/low/close/volume, include proportional transaction costs (CSI buys 0.0005, sells 0.0015; S&P sells 0.0005), and evaluate AR, IR, CR, and MDD.
Themesinnovation adoption
GeneralizabilityShort out-of-sample period (single year, 2025) limits robustness to different market regimes and longer horizons, Two equity universes only (CSI 500 and S&P 500), so performance may not generalize to other asset classes, smaller caps, or international markets, Use of daily OHLCV only excludes fundamentals, alternative data, and intraday microstructure effects, Dependence on specific LLM agents, prompts, and implementation details may limit reproducibility and transferability, Potential backtest risks: selection/data-snooping, survivorship bias, lookahead bias, and unmodeled execution slippage, Transaction cost assumptions may not capture real-world liquidity and market impact in large-scale deployment
Claims (8)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| FaVOR structures factor mining as a three-stage consistency loop consisting of hypothesis decomposition, factor-level validation, and factor integration. Other | positive | Consistency between generated factor formulas and their intended economic hypotheses |
Reading fidelity
high
Study strength
medium
|
not reported
|
| FaVOR validates candidate factors using factor-conditioned distributions of OHLCV-derived market variables rather than evaluating them solely by future returns. Other | positive | Empirical alignment of factor values with observable market states |
Reading fidelity
high
Study strength
medium
|
not reported
|
| The final pass/fail decision in FaVOR's factor validation stage is deterministic once the LLM maps the textual observation to relevant market variables and expected directions. Decision Quality | positive | Factor-validation pass/fail decision |
Reading fidelity
high
Study strength
medium
|
not reported
|
| FaVOR uses directional selectivity to retain factor combinations whose realized returns or win rates improve when entry conditions become more selective. Output Quality | positive | Average realized return and win rate under increasingly selective entry thresholds |
Reading fidelity
high
Study strength
medium
|
not reported
|
| On the 2025 CSI 500 test period, FaVOR achieved an annualized excess return of 0.2067, an information ratio of 1.5295, a maximum drawdown of -0.0853, and a cumulative excess return of 0.2225. Firm Productivity | positive | CSI 500 portfolio annualized return, information ratio, maximum drawdown, and cumulative excess return |
Reading fidelity
high
Study strength
medium
|
AR 0.2067; IR 1.5295; MDD -0.0853; CR 0.2225
|
| On the 2025 S&P 500 test period, FaVOR achieved an annualized excess return of 0.1062, an information ratio of 1.1315, a maximum drawdown of -0.0443, and a cumulative excess return of 0.1123. Firm Productivity | positive | S&P 500 portfolio annualized return, information ratio, maximum drawdown, and cumulative excess return |
Reading fidelity
high
Study strength
medium
|
AR 0.1062; IR 1.1315; MDD -0.0443; CR 0.1123
|
| FaVOR outperformed every listed baseline on all four reported portfolio metrics in both the CSI 500 and S&P 500 test results. Firm Productivity | positive | Relative portfolio performance across annualized return, information ratio, maximum drawdown, and cumulative excess return |
Reading fidelity
high
Study strength
medium
|
not reported
|
| The paper concludes that hypothesis-centered empirical validation can improve the economic reliability of automated factor mining. Other | positive | Economic reliability and out-of-sample performance of automatically mined factors |
Reading fidelity
high
Study strength
low
|
not reported
|