A real-world finance benchmark exposes gaps between lab scores and business needs: GPT-5 records 61.5% accuracy against an 84.8% operational bar, while commercial model DeepSeek-R1 leads on investment efficacy.

Main Finding

BizFinBench.v2 introduces the first large-scale, bilingual (China + US) financial benchmark based on authentic user query–response data and an integrated offline + online evaluation track. It contains 28,860 real-world QA pairs across 8 offline tasks and 2 online tasks (including a market-facing portfolio simulation). Evaluating 21 LLMs, the authors find commercial best-in-class models still fall short of practical business requirements (e.g., GPT-5 averages 61.5% accuracy vs. an asserted business target of 84.8%). Open-source model DeepSeek‑R1 yields the best investment efficacy in the portfolio allocation simulation. Error analysis highlights persistent shortcomings (temporal grounding, numeric precision, robustness to noisy inputs, and online decision-making).

Key Points

• Scope and novelty
  • First benchmark built from real user queries on financial platforms, covering Chinese and U.S. equity markets.
  • Dual-track evaluation: Core Business Capabilities (offline) + Online Performance (real-time tasks including simulated trading).
  • Bilingual and cross-market coverage; intended to close the gap between lab benchmarks and deployed business performance.
• Task taxonomy (4 scenarios → 10 tasks)
  • Business Information Provenance (BIP): Anomaly Information Tracing (AIT), Financial Multi-turn Perception (FMP), Financial Data Description (FDD).
  • Financial Logical Reasoning (FLR): Financial Quantitative Computation (FQC), Event Logic Reasoning (ELR), Counterfactual Inference (CI).
  • Stakeholder Feature Perception (SFP): User Sentiment Analysis (SA), Financial Report Analysis (FRA; China only).
  • Real-time Market Discernment (RMD): Stock Price Prediction (SPP), Portfolio Asset Allocation (PAA; live trading simulation).
• Scale & data characteristics
  • 28,860 questions total. Examples: 3,963 AIT, 4,497 FMP, 3,803 FDD; average input token lengths vary widely (e.g., FRA ~19,681 tokens).
  • All data sourced from real user interactions; desensitized and passed a three-stage quality control (platform clustering + frontline staff review + expert cross-validation).
• Evaluation protocol & metrics
  • Offline tasks: average accuracy (zero-shot and CoT settings).
  • SPP/SA: require prediction intervals; tolerance: SA 10%, SPP 1% (prediction interval must contain ground truth).
  • PAA: evaluated by financial outcomes (cumulative return, Sharpe ratio, max drawdown) via a trading simulator that models fees, latency, slippage; decisions hourly.
• Empirical results (selected)
  • GPT-5: average accuracy 61.5% (zero-shot). Gemini‑3: 61.3%. Many models perform well on some tasks (e.g., FMP) but poorly on others (e.g., FRA, CI).
  • DeepSeek‑R1 (open-source, 671B) shows superior investment performance in PAA relative to evaluated commercial models.
• Limitations called out by authors
  • Benchmarks’ rubrics for offline quality control not fully disclosed (privacy/compliance).
  • FRA only covers Chinese market.
  • PAA transaction counts and unlimited question potential depend on simulation time horizon.

Data & Methods

• Data sources
  • Real user queries and responses from financial business platforms covering institutional and retail users in China and the U.S. markets.
  • Data were desensitized (PII/sensitive corpora removed) and clustered automatically before manual and expert review.
• Quality control pipeline
  • Three-step screening: automated platform clustering + desensitization → manual screening by frontline professionals (≥5 years experience) → cross-validation by senior financial experts (≥10 years experience).
• Tasks and formulations
  • Offline: verifiable open-ended questions across AIT, FMP, FDD, FQC, ELR, CI, SA, FRA.
  • Online:
    • Stock Price Prediction (SPP): models produce a point prediction and an interval; success if ground truth lies in interval (1% tolerance).
    • Portfolio Asset Allocation (PAA): integrates real-time market data; LLMs submit hourly decisions to a trading simulator.
• Evaluation settings
  • Tested 21 LLMs (proprietary via API and open-source), under zero-shot and chain-of-thought prompting where applicable.
  • Metrics: task-level accuracy, interval containment for SPP/SA, and finance metrics (cumulative returns, Sharpe, drawdown) for PAA.
• Released resources
  • Data and code (including the PAA simulator) to be open-sourced at https://github.com/HiThink-Research/BizFinBench.v2 (paper states data/code will be released; some rubrics kept private for compliance).

Implications for AI Economics

• External validity and real-world deployment value
  • Benchmarks built from synthetic or generic data overestimate business performance. Real-user benchmarks like BizFinBench.v2 are essential for accurate ex-ante valuation of LLM deployment in finance.
  • Firms should re-assess expected returns from model deployment using realistic task distributions, long-context inputs, and online decision dynamics rather than relying on standard offline benchmarks.
• Economic consequences of model shortcomings
  • Deficits in temporal grounding and numeric precision can translate directly into financial loss (wrong trades, missed signals). The gap between academic metrics and business thresholds (e.g., 61.5% vs 84.8%) implies potentially large negative or suboptimal economic outcomes if used naively.
• Measuring ROI and risk-adjusted value
  • PAA-style online simulations produce financial metrics (returns, Sharpe, drawdown) directly interpretable for ROI estimation and risk assessment. Economists and practitioners should prefer these outcome-level metrics when deciding on procurement or investment in LLM-based tools.
• Open-source vs. proprietary trade-offs
  • Open-source models (e.g., DeepSeek‑R1) can deliver competitive or superior economic performance in trading tasks, suggesting lower-cost deployments may be viable with domain adaptation and integration.
• Policy and regulatory considerations
  • Real-user benchmarks reveal failure modes relevant to compliance (hallucinations, misattribution, erroneous risk calls). Regulators and firms need realistic stress-testing (including online simulation) before production deployment.
• Research and investment priorities
  • To increase economic value, prioritize improving:
    • Temporal grounding and access to verifiable real-time data (reduces stale inference losses).
    • Quantitative/numeric reasoning and calibration (reduces risk of incorrect computations).
    • Robustness to noisy, multi-turn, long-context user interactions (improves service quality and reduces manual oversight costs).
    • System integrations: tool use (market data APIs), latency-aware decision pipelines, and safe human-in-the-loop controls.
• Suggested next steps for economists and firms
  • Use benchmarks like BizFinBench.v2 when estimating model-driven alpha and when backtesting deployment strategies.
  • Conduct cost–benefit analyses that include model improvement costs, compliance overhead, monitoring, and expected changes in trading or advisory P&L.
  • Assess market impact and strategic responses — deploying models that trade or advise at scale may alter market dynamics; these general equilibrium effects should be considered in valuation.

If you want, I can: - Extract a short table of the top-performing models per task from the paper’s results. - Draft a checklist firms can use to evaluate LLM readiness for production in financial services using BizFinBench.v2 metrics.