A new point-in-time benchmark and harness for financial deep research reveals substantial gaps in current LLM agents—top systems score below 40% on expert rubrics—while the FinanceHarness pipeline yields a modest but measurable improvement (25.3% → 32.4%).

Main Finding

FinanceHarness introduces an end-to-end, point-in-time (PIT) framework and benchmark for automated, analyst-style financial deep research. Built on a large PIT web corpus and an entity graph, FinanceGym is a 400-question, expert-validated benchmark with pre-cutoff (evidence retrieval) and post-cutoff (forward-looking) rubrics. FinanceHarness is a layered agent harness that runs finance-focused tools, enforces PIT access, and trains/evaluates agents against FinanceGym. Existing LLMs and agents perform poorly (all <40% rubric score); FinanceHarness raises an open-backbone baseline from 25.3% to 32.4%, showing measurable but limited improvement and substantial headroom.

Key Points

• Point-in-time evaluation: Agents can only retrieve documents published on or before a question-specific cutoff date; any PIT leakage invalidates runs.
• FinanceGym benchmark:
  • 400 balanced, expert-annotated research questions.
  • 2,464 rubric items (pre-/post-cutoff split) with professional validation (82% pass rate during curation).
  • Questions span topics, sectors, and reasoning types; each question has a reference investment thesis and criterion-level groundings.
• FinanceHarness:
  • Layered harness exposing finance-specific tools, orchestration, runtime, and reward modeling; ties training and evaluation to the same PIT contract and tool stack.
  • Integrates PIT search, reading, citation-grounded synthesis, and auxiliary finance primitives (valuation, comparables, numeric extraction).
• Empirical results:
  • Built a PIT search sandbox from >100M web articles (publication dates extracted via htmldate, text via trafilatura).
  • Entity-relation graph: ~4.37M filtered edges, ~1.11M unique entities from 1.2M source articles.
  • Retrieval: Qwen3-Embedding-4B embeddings, FAISS IVF-SQ8 index, with text fetched separately for citation grounding.
  • Rubric scoring: LLM judge assigns 0–4 per criterion with explicit anchors; primary metric is per-question normalized average of rubric points.
  • Benchmarked many LLMs/agents — all score <40%; a FinanceHarness pipeline improves a shared open backbone from 25.3% → 32.4%.
• Release: Questions and submission/grading code released; underlying corpus not publicly released to protect PIT integrity and provenance.

Data & Methods

• Corpus construction:
  • Collected ~100+ million public web articles filtered for finance relevance.
  • Publication dates normalized with htmldate; text extracted using trafilatura.
• Embedding & retrieval:
  • Embeddings: Qwen3-Embedding-4B.
  • Index: FAISS IVF-SQ8; API enforces per-question cutoff date filtering.
• Entity graph & situation mining:
  • Entity-relation extraction via Gemini-3.5-Flash producing ~5.74M raw edges → 4.37M filtered edges.
  • Situation mining modes: cross-category linkages (multihop paths), temporal narratives, divergences/tensions (e.g., upgrades vs downgrades).
  • Cutoff-date selection: optimized for event volume, entity diversity, and relation entropy via a z-scored objective.
• Question & rubric generation:
  • LLMs generate unconstrained analyst-style questions + reference theses + two-tier rubrics given (situation, cutoff).
  • Bottom-up taxonomy assigned via LLM classification; balance enforced across topic/sector/reasoning/time.
  • Filtering pipeline reduced ~29.7k generations → 2,078 candidate pool → ILP-selected 500 → expert annotated → 411 pass → final balanced release of 400.
  • Expert annotation: ~1.2 hours per sample; annotators mark feasibility/clarity and rubric feasibility.
• Evaluation protocol:
  • LLM judge (with anchors) scores each rubric item 0–4; primary outcome metric is per-question fractional rubric score averaged over questions.
  • Pre-cutoff items test retrieval and evidence-grounded synthesis; post-cutoff items test forward-looking anticipations verifiable after cutoff.
  • Any detection of access to post-cutoff info invalidates the run.

Implications for AI Economics

• Evaluation design for temporally sensitive forecasts:
  • The PIT protocol addresses a central econometric/identification issue for forecasting tasks: preventing leakage of future information. This is a model for economic evaluation where temporality matters (macroeconomic forecasts, event studies).
• Labor and task reallocation:
  • Demonstrated capabilities (and current limitations) indicate potential partial automation of analyst tasks (evidence collection, drafting) while higher-level judgment and outcome anticipation remain hard. Expect complementarities: tools augment analysts’ throughput and change task mix (more monitoring, strategy, and adjudication).
• Market efficiency and information diffusion:
  • Better automated research agents could accelerate information synthesis and dissemination, potentially reducing frictions and time-to-price incorporation. That raises research opportunities on whether agent-generated analysis compresses price discovery, increases volatility, or alters liquidity.
• Model risk, incentives, and regulation:
  • PIT enforcement and verifiable rubrics highlight the importance of auditable evaluation for models used in finance. Regulators and firms need protocols to detect leakage, data contamination, and overfitting to benchmarks. Miscalibrated agent outputs could cause erroneous investment actions at scale—necessitating guardrails, provenance tracking, and human-in-the-loop oversight.
• Algorithmic narratives & systemic risk:
  • Agents trained to mine similar entity graphs and motifs may amplify common narratives (herding) and exacerbate correlated trading; economic research should study how agentic frameworks change cross-sectional return dynamics and systemic vulnerabilities.
• Research avenues in AI economics enabled by FinanceHarness/FinanceGym:
  • Quantify labor-substitution vs. complementarity effects for financial analysts using controlled deployments.
  • Measure how automated research affects trading behavior, price impact, and market efficiency in event-study settings with PIT constraints.
  • Study information asymmetry: do firms/market participants that adopt agentic research obtain measurable informational advantages?
  • Evaluate welfare tradeoffs of deploying agentic analysis across institutional vs. retail markets.
• Policy and marketplace incentives:
  • Benchmarks like FinanceGym create incentives for providers to optimize for temporally robust forecasting and provenance-aware retrieval. But they may also encourage overfitting to rubrics; governance and multiple evaluation modalities will be important.

Limitations and caveats - Underlying corpus and private grading logic are not publicly released; reproducibility outside their infrastructure requires effort. - The benchmark focuses on analyst-style research in public markets; findings may not generalize to private markets or non-public information contexts. - Agent performance gains reported (25.3%→32.4%) show progress but also that forward-looking, evidence-aware financial reasoning remains a hard problem.

If you want, I can extract specific empirical results (model-by-model scores, rubric breakdowns, or ablation details) from the paper and summarize them in a short table or bullet list.