A 1,400-question benchmark finds modern LLMs can leverage retrieval to interpret company filings but largely fail at market trading-signal and time-series reasoning; this gap limits their reliability for real-world financial decision-making.

FinTradeBench: A Financial Reasoning Benchmark for LLMs

Yogesh Agrawal, Aniruddha Dutta, Md Mahadi Hasan, Santu Karmaker, Aritra Dutta · March 19, 2026

arxiv descriptive low evidence 7/10 relevance Source PDF

FinTradeBench is a 1,400-question benchmark combining company fundamentals and trading signals for NASDAQ-100 firms over ten years, showing retrieval helps LLM reasoning over textual fundamentals but current models struggle with numerical and time-series trading-signal reasoning.

Real-world financial decision-making is a challenging problem that requires reasoning over heterogeneous signals, including company fundamentals derived from regulatory filings and trading signals computed from price dynamics. Recently, with the advancement of Large Language Models (LLMs), financial analysts have begun to use them for financial decision-making tasks. However, existing financial question answering benchmarks for testing these models primarily focus on company balance sheet data and rarely evaluate reasoning over how company stocks trade in the market or their interactions with fundamentals. To take advantage of the strengths of both approaches, we introduce FinTradeBench, a benchmark for evaluating financial reasoning that integrates company fundamentals and trading signals. FinTradeBench contains 1,400 questions grounded in NASDAQ-100 companies over a ten-year historical window. The benchmark is organized into three reasoning categories: fundamentals-focused, trading-signal-focused, and hybrid questions requiring cross-signal reasoning. To ensure reliability at scale, we adopt a calibration-then-scaling framework that combines expert seed questions, multi-model response generation, intra-model self-filtering, numerical auditing, and human-LLM judge alignment. We evaluate 14 LLMs under zero-shot prompting and retrieval-augmented settings and witness a clear performance gap. Retrieval substantially improves reasoning over textual fundamentals, but provides limited benefit for trading-signal reasoning. These findings highlight fundamental challenges in the numerical and time-series reasoning for current LLMs and motivate future research in financial intelligence.

Summary

Main Finding

FinTradeBench is a new benchmark (1,400 historically grounded questions) that tests LLM financial reasoning over both company fundamentals (SEC filings) and market trading signals (price/volume time series). Evaluations of 14 LLMs show (1) retrieval-augmented LLMs substantially improve reasoning on textual fundamentals and hybrid questions (reported +37% and +55% relative gains), but (2) LLMs perform poorly on trading-signal/time-series reasoning and often receive limited or negative benefit from retrieval. Overall absolute accuracy remains low (~20–40% across models), highlighting current LLM weaknesses on numerical/time-series financial reasoning.

Key Points

Dataset and scope
- Covers NASDAQ‑100 firms across 2015–2025.
- 1,400 questions created by scaling 150 expert seed questions (50 per category).
- Three question types: Fundamentals-only (F), Trading-signal-only (T), Hybrid (FT).
Signals and grounding
- Company fundamentals from 10‑K/10‑Q (ROA, ROE, valuation ratios, leverage, etc.).
- Trading signals from OHLCV time series (momentum, moving averages, realized volatility, drawdowns, volume).
- All questions aligned by ticker & financial quarter so answers are verifiable against historical data.
Benchmark construction (calibration-then-scaling)
- Multi-model, multi-prompt candidate generation (TELeR taxonomy), intra-model self-selection of best candidate, automated numerical audit.
- Human experts score seed outputs; an independent LLM judge (Claude Sonnet 4.5) is calibrated to human judgments (MAE < 10%) and used to scale evaluation.
Retrieval & architecture
- Dual-track RAG: Track A for SEC text using hierarchical parent–child chunking + metadata injection; Track B for time-series chunks (temporal retrieval). Prompts assemble evidence within a global token budget.
- TELeR-guided generation + self-filtering used for robust candidate selection.
Empirical results
- Retrieval (RAG) yields large relative improvements on fundamentals-focused and hybrid questions (paper reports +37% and +55% respectively).
- For trading-signal (time-series) questions, RAG provides little to no benefit and can hurt performance for some models.
- Across the 14 evaluated models (proprietary and open-source, 1B–100B+ parameter ranges) absolute category accuracies are modest (many models in the ~20–40% range), indicating substantive room for improvement.
Diagnostic findings
- LLMs can leverage textual grounding much better than structured numerical/time-series evidence.
- Failures concentrated in numerical fidelity, temporal reasoning, and interpreting interactions between fundamentals and market dynamics.

Data & Methods

Data sources: SEC filings (10‑K/10‑Q) for fundamentals; daily OHLCV price/volume for trading signals.
Signal selection principles: interpretability, empirical relevance, liquidity. Full signal list included in paper.
Question taxonomy: F-type, T-type, FT-type (examples and golden indicators provided for seed questions).
Calibration pipeline:
- Phase 1: N=6 TELeR prompts per model → intra-model self-filter → numerical auditing (SUPPORTED/CONTRADICTED/NOT FOUND).
- Phase 2: Human expert double-blind evaluation on 150 seed Qs; LLM judge calibrated to humans.
- Phase 3: Automated scaling across tickers/time to produce 1,400 benchmark items; final scoring by calibrated LLM judge.
Retrieval architecture:
- Hierarchical indexing and metadata injection to reduce temporal hallucination.
- Dual-track retrieval: text (dense + BM25 + re-ranker) and time-series (temporal query).
- Assembly under token budget to balance long textual context with short time-series evidence.
Models evaluated: 14 LLMs spanning large proprietary (e.g., GPT-5-mini, Gemini 2.5 variants, DeepSeek-R1), mid and small open models (LLaMA variants, Qwen, Phi-4, Mistral, distilled models).
Metrics: absolute accuracy (scaled judge scores), relative retrieval delta (RAG vs No-RAG), golden-indicator F1 (precision/recall on key metrics), statistical testing via paired t-tests.

Implications for AI Economics

For practitioners and policymakers
- LLMs can be useful assistants for tasks relying on textual disclosures (earnings note interpretation, narrative summarization) if paired with reliable retrieval; however, they are not yet reliable for decisions that depend on precise time-series signal interpretation (trade timing, short-horizon strategy signals).
- Deploying LLMs alone for automated trading or high-stakes portfolio decisions is premature—human oversight and dedicated quantitative modules remain necessary.
For researchers
- There is a clear need to improve numerical fidelity and time-series reasoning in LLMs (hybrid architectures that combine LLMs with specialized time-series/quant modules, or neural-symbolic approaches).
- Benchmarks should evaluate cross-signal integration (fundamentals × market dynamics); FinTradeBench provides a reproducible testbed for this.
- Retrieval systems for finance need better mechanisms to incorporate structured numeric evidence (not just textual documents): improved temporal retrieval, differentiable calculators, and verification layers are promising directions.
For the economics of model adoption
- Models’ stronger performance on fundamentals suggests LLMs may accelerate text-heavy analyst workflows (coverage expansion, first-pass screening), potentially changing labor allocation in financial analysis.
- Weakness on trading signals limits near-term automation in quantitative strategies; gains are more likely in augmentation than replacement.
- Transparency and evaluation pipelines (human-aligned judges, numerical audits) are essential to reduce model-driven market risk and to meet regulatory expectations for model validation.

Takeaway: FinTradeBench demonstrates that current LLMs can be successfully grounded on textual financial disclosures with retrieval, but significant methodological advances are required before LLMs can reliably reason about, and act on, market trading signals and their interaction with fundamentals.

Assessment

Paper Typedescriptive Evidence Strengthlow — The paper presents a benchmarking dataset and reports model performance; it does not identify causal effects on economic outcomes (productivity, wages, firm performance) or evaluate real-world decision outcomes, so its evidence does not support causal claims about AI's economic impact. Methods Rigormedium — The authors employ a systematic 'calibration-then-scaling' process (expert seed questions, multi-model response generation, intra-model filtering, numerical auditing, and human–LLM judge alignment) and evaluate 14 LLMs across retrieval/no-retrieval settings, which is methodically thorough for benchmark construction; however, potential issues remain (question selection/design biases, reliance on specific retrieval implementations, lack of out-of-sample/real-world trading validation, and limited transparency about annotation disagreement rates), preventing a 'high' rating. SampleFinTradeBench: 1,400 questions grounded in NASDAQ-100 companies spanning a ten-year historical window; questions fall into fundamentals-focused, trading-signal-focused, and hybrid cross-signal reasoning categories; evaluated across 14 LLMs under zero-shot and retrieval-augmented setups with multi-model and human-LLM judging procedures. Themesadoption human_ai_collab GeneralizabilityLimited to NASDAQ-100 firms (large-cap, US-centric, sector-skewed sample), Ten-year historical window may not reflect current/other market regimes, Benchmark Q&A format may not map directly to real-world trading or decision-making outcomes, Results depend on specific retrieval systems and prompt designs used, Models evaluated are a snapshot in time; newer models/finetuning could change results

Claims (8)

Claim	Direction	Confidence	Outcome	Details
FinTradeBench contains 1,400 questions grounded in NASDAQ-100 companies over a ten-year historical window. Other	null_result	high	benchmark size and scope (number of questions; data grounding)	n=1400 0.3
The benchmark is organized into three reasoning categories: fundamentals-focused, trading-signal-focused, and hybrid questions requiring cross-signal reasoning. Other	null_result	high	benchmark organization / task taxonomy	0.3
To ensure reliability at scale, we adopt a calibration-then-scaling framework that combines expert seed questions, multi-model response generation, intra-model self-filtering, numerical auditing, and human-LLM judge alignment. Other	null_result	high	benchmark annotation and validation procedure	0.3
We evaluate 14 LLMs under zero-shot prompting and retrieval-augmented settings and witness a clear performance gap. Output Quality	mixed	high	model performance on financial reasoning benchmark (accuracy/score across models)	n=14 0.18
Retrieval substantially improves reasoning over textual fundamentals. Output Quality	positive	high	improvement in reasoning/performance on fundamentals-focused questions with retrieval	n=1400 0.18
Retrieval provides limited benefit for trading-signal reasoning. Output Quality	null_result	high	change in performance on trading-signal-focused questions with retrieval	n=1400 0.18
These findings highlight fundamental challenges in the numerical and time-series reasoning for current LLMs and motivate future research in financial intelligence. Other	negative	high	LLMs' numerical and time-series reasoning capability (qualitative conclusion from experiments)	n=14 0.18
Existing financial question answering benchmarks primarily focus on company balance sheet data and rarely evaluate reasoning over how company stocks trade in the market or their interactions with fundamentals. Other	null_result	high	scope of existing financial QA benchmarks (focus on balance sheet data vs. trading signals)	0.09