A new benchmark finds AI research agents can name financial formulas but struggle to build indicators end-to-end: across 3,350 tasks in US and Chinese markets agents reach only around 40% correct final answers, with data collection and macroeconomic indicators the biggest bottlenecks.

FinDeepIndicator: Benchmarking Deep Research Agents in End-to-End Financial Indicator Construction

Chaoqun Yang, Fengbin Zhu, Xinyu Lin, Long Bai, Xiaoluan Liu, Ke-Wei Huang, Roger Zimmermann, Tat-Seng Chua · August 01, 2026

arxiv descriptive n/a evidence 7/10 relevance Full text usable extracted full text Source PDF

Structured author observations

Linked only from stored provider relations; the raw author line above is never matched by name.

Arxiv

Latest observation: August 4, 2026

Chaoqun Yang unresolved corpus identity
Fengbin Zhu unresolved corpus identity
Xinyu Lin unresolved corpus identity
Long Bai unresolved corpus identity
Xiaoluan Liu unresolved corpus identity
Ke-Wei Huang unresolved corpus identity
Roger Zimmermann unresolved corpus identity
Tat-Seng Chua unresolved corpus identity

Semantic Scholar

Latest observation: August 25, 2026

Chaoqun Yang provider ID
Fengbin Zhu provider ID
Xinyu Lin provider ID
Long Bai provider ID
Xiaoluan Liu provider ID
Ke-wei Huang provider ID
Roger Zimmermann provider ID
Tat-Seng Chua provider ID

FinDeepIndicator is a process-level benchmark for end-to-end financial indicator construction (3,350 QA pairs, 234 indicators, US and China) showing that while LLMs often specify formulas correctly, they—and current DR agents—fail frequently on data retrieval and numeric execution, yielding only ~40% final-answer accuracy.

Citation observations

Cumulative provider counts captured on specific dates; providers are never combined.

0 cumulative citations

Semantic Scholar · Observed August 25, 2026

View corpus context

Financial indicators are essential tools for transforming raw financial data into interpretable measures for various downstream tasks, such as valuation, risk assessment, and economic analysis. However, existing financial benchmarks largely focus on answer-level accuracy and often assume that relevant data are already provided, leaving the assessment of the intermediate process of indicator construction underexplored. In this work, we propose FinDeepIndicator, the first benchmark dedicated to evaluating Deep Research (DR) agents in end-to-end financial indicator construction. Specifically, FinDeepIndicator evaluates DR agents across four stages in indicator construction: formula specification, data collection, indicator calculation, and answer generation, and covers fundamental, technical, and macroeconomic indicators organized into 21 fine-grained sub-categories. It contains 3,350 curated question-answer (QA) pairs derived from both U.S. and Chinese markets, 10 years of historical financial data, and 800 listed companies. Extensive experiments on search-equipped Large Language Models (LLMs) and DR agents show that, while LLMs generally perform well in formula specification, their accuracy drops substantially during data retrieval and numerical execution. DR agents consistently outperform search-equipped LLMs, yet remain unreliable in realistic financial analysis settings. These findings provide insights for developing more capable and trustworthy DR agents in finance.

Summary

Main Finding

FinDeepIndicator is a process-level benchmark for end-to-end financial indicator construction. It shows that contemporary search-equipped LLMs often understand indicator formulas (≫70% formula accuracy) but fail to reliably complete the full pipeline: best systems reach only ~40% final-answer accuracy. Data collection (raw-data retrieval and alignment) is the primary bottleneck; macroeconomic indicators and cross-market differences (U.S. vs China) pose extra difficulty. DR agents (agentic tool-using systems) improve results over single-pass LLM+search, but gains are model-dependent and overall reliability remains limited.

Key Points

Benchmark scope
- 234 distinct financial indicators organized into 3 major categories (fundamental, technical, macroeconomic) and 21 sub-categories.
- 3,350 curated QA pairs covering U.S. and Chinese markets.
- 10 years of historical data for 800 listed companies.
Process-level evaluation: decomposes the task into four stages
Formula Specification (identify/express correct mathematical definition).
Data Collection (retrieve and align raw inputs).
Indicator Calculation (apply formula, compute intermediate values).
Answer Generation (final aggregation/selection).
Dataset construction
- 170 templates with linguistic variants and three difficulty levels (Easy / Medium / Hard).
- Instances include reference formula, required raw data, stepwise calculations, and final numeric answer.
- Quality control: taxonomy-driven constraints, expert inspections.
Evaluation design & tooling
- Free-form model outputs parsed into structured components by an LLM; parsing validated by experts (Gwet’s AC1 scores > 0.6; high acceptance rates).
- Numeric comparisons use tolerance-aware matching (examples: 0.01, 0.005, 0.001).
- Formula correctness judged by LLM-as-judge with human consistency checks (Alt-test).
Models evaluated
- Search-equipped LLMs: Qwen3.6-Max, Claude-Sonnet-4.6, Gemini-3-Flash, Deepseek-V4-Flash, Grok-4.3, GPT-5, GPT-5-Mini (Google Search used as search API).
- Deep Research (DR) agents: minimal ReAct-style agents with iterative tool use (search + Python).
Key empirical findings
- Large gap between formula understanding (>70% accuracy) and end-to-end final answers (~40% top).
- Data collection accuracy and numerical execution cause major performance drop-offs.
- Macroeconomic indicators hardest (most models <30% final-answer accuracy), likely due to heterogeneous sources, aligning frequencies, geography/definitions.
- Cross-market gap: best agents perform better on U.S. data than Chinese (gaps up to ≈9.6% for some agents).
- Agentic approaches yield substantial improvements over single-step LLM+search, but gains vary by model and do not eliminate unreliability.

Data & Methods

Data
- 10 years historical financial and macro data; 800 public companies; U.S. and China markets.
- Structured metadata for each indicator: standardized name, formula, required raw data fields, and executable calculation program.
Benchmark generation
- Enumerate indicator × template variants; sample placeholders (company, date, etc.); generated instance pool pruned to 3,350 representative QA pairs.
- Difficulty control and constraints to ensure answerability.
Evaluation pipeline
- LLM-based parser converts free-form responses into four structured components (formula, raw data table, indicator calc table, final answer).
- Cell-level, tolerance-aware matching for numerical tables; LLM-as-judge for formula semantics; human expert sampling for validation.
Experimental setups
- Search setting: single-search-then-answer.
- Agent setting: ReAct-style iterative reasoning with search and Python tool calls.
- Multiple LLM backbones evaluated; Google Search used as retrieval source.

Implications for AI Economics

For research and model design
- Improve grounded retrieval and robust ETL: benchmarks show that understanding formulas is insufficient — models need reliable, structured data ingestion and alignment across heterogeneous sources.
- Emphasize tool use and executable pipelines: agentic architectures (iterative tool use, code execution) help but need better orchestration, verification, and error recovery.
- Invest in domain-specific grounding and adaptation: cross-market/reporting heterogeneity (e.g., accounting conventions, disclosure formats) meaningfully degrades performance.
- Develop numeric-robust reasoning and verification layers: tolerate rounding but detect misalignments, unit mismatches, and time-frequency mismatches in macro indicators.
- Prioritize interpretability and provenance: process-level outputs (intermediate tables, formulas, source citations) are essential for auditing financial analyses and for model adoption in economic practice.
For deployed financial systems and practitioners
- Current models are not yet dependable for unsupervised end-to-end financial indicator construction; human oversight remains necessary, especially for high-stakes decisions.
- Model errors concentrate in data retrieval/processing rather than conceptual understanding—operational controls (source whitelists, automated sanity checks) can reduce risk.
For policy and regulation
- Benchmarks like FinDeepIndicator enable finer-grained auditability and standardization for AI in finance; regulators should encourage process-level evaluation and provenance requirements for AI-driven analyses.
For the economics of labor and workflows
- Automated assistance may speed formulaic parts of analysis, but the persistent retrieval/verification gap suggests human analysts will still be required for data curation and validation in the medium term.
For future AI-economics research
- Use FinDeepIndicator to evaluate downstream economic models that rely on constructed indicators (e.g., macro forecasting, asset pricing) to quantify how construction errors propagate into economic inferences and policy recommendations.

If you want, I can: - Extract a short list of the most error-prone indicator sub-categories (from the paper) and suggested technical fixes for each; or - Produce a one-page slide-ready summary for policymakers or product managers.

Assessment

Paper Typedescriptive Evidence Strengthn/a — This is a dataset and benchmark paper evaluating model/agent performance; it does not attempt causal identification or provide causal empirical evidence about economic outcomes. Methods Rigormedium — The authors present a comprehensive, process-level benchmark with careful taxonomy, template design, QA generation, tolerance-aware numerical metrics, and expert validation (including inter-annotator agreement). Limitations include reliance on synthetic templates and sampled instantiations, potential circularity from using LLMs both for parsing and judging outputs, evaluation constrained to specific search/tool setups, and limited discussion of robustness to alternative data sources or real-time data variability. SampleFinDeepIndicator contains 3,350 curated question-answer pairs covering 234 financial indicators (organized into 3 major categories and 21 sub-categories), instantiated via 170 templates with three difficulty levels; uses 10 years of historical financial data for 800 listed companies across U.S. and Chinese markets; experiments evaluate multiple search-equipped LLMs and agentic (ReAct-style) systems using Google Search and Python tools. Themeshuman_ai_collab adoption GeneralizabilityBenchmark examples are generated from templates and sampled placeholders, which may not cover the full diversity of real-world analyst queries., Data limited to 800 listed companies and two markets (US and China); performance may differ in other markets or private/less-structured data sources., Evaluation depends on a particular search API and data accessibility; results are sensitive to changes in web sources, APIs, or data availability over time., Use of LLM-based parsing and LLM-as-judge introduces potential evaluator-model dependence and may not generalize to non-LLM evaluation pipelines., Tolerance thresholds and normalization rules affect measured accuracy and may hide some substantive errors in interpretation or data alignment.

Claims (10)

Claim	Direction	Outcome	Confidence & Evidence	Details
FinDeepIndicator contains 3,350 curated question-answer pairs covering U.S. and Chinese markets, 10 years of historical financial data, and 800 publicly listed companies. Other	positive	Benchmark coverage and dataset scale	Reading fidelity high Study strength medium	n=3350 0.18
FinDeepIndicator evaluates financial-indicator construction across four stages: formula specification, data collection, indicator calculation, and answer generation. Other	positive	Process-level evaluation coverage	Reading fidelity high Study strength medium	not reported 0.18
The benchmark taxonomy comprises 234 financial indicators organized into three major categories and 21 fine-grained sub-categories. Other	positive	Indicator taxonomy coverage	Reading fidelity high Study strength medium	n=234 234 distinct indicators 0.18
The best-performing evaluated system achieves approximately 40% final-answer accuracy, while most models exceed 70% formula accuracy. Output Quality	mixed	Final-answer accuracy and formula accuracy	Reading fidelity high Study strength medium	approximately 40% final-answer accuracy; most models exceeding 70% formula accuracy 0.18
Deep-research-agent settings generally outperform search-equipped LLM settings, particularly for data collection, indicator calculation, and final-answer generation. Output Quality	positive	Data-collection, indicator-calculation, and final-answer accuracy	Reading fidelity high Study strength medium	not reported 0.18
Two of the three strongest deep-research agents achieve higher final-answer accuracy in the U.S. market than in the Chinese market, by more than 9.6% and 3.1%, respectively. Output Quality	positive	Cross-market final-answer accuracy	Reading fidelity high Study strength medium	n=3 over 9.6% and 3.1% higher accuracy 0.18
Macroeconomic indicators are the most challenging category for the evaluated models, with final-answer accuracy below 30% for most models. Output Quality	negative	Final-answer accuracy on macroeconomic indicators	Reading fidelity high Study strength medium	below 30% final-answer accuracy 0.18
External-trade, fiscal, labor, and productivity indicators are particularly difficult for the evaluated models. Output Quality	negative	Final-answer accuracy by macroeconomic subcategory	Reading fidelity high Study strength medium	not reported 0.18
The LLM-based response-parsing procedure received expert acceptance rates above 0.9 for all four extracted components. Ai Safety And Ethics	positive	Accuracy and completeness of automatic response parsing	Reading fidelity high Study strength medium	n=100 all acceptance rates above 0.9 0.18
Data collection is identified as the primary bottleneck in end-to-end financial-indicator construction. Output Quality	negative	Data-collection accuracy and end-to-end construction reliability	Reading fidelity high Study strength medium	not reported 0.18