A new public benchmark for investment-research agents finds that toolchains, not just base models, drive real-world performance; Samaya’s in-house system scores 56% while leading open-weight models nearly match proprietary accuracy at substantially lower cost.

Main Finding

FrontierFinance is a new, open benchmark for evaluating finance AI agents on end-to-end, professional investment research tasks. It contains 220 expert-crafted queries and 11,543 source‑attributed binary rubrics across six investor‑workflow use cases, and shows that (1) agent tooling and orchestration (the "harness") strongly shape performance and cost, (2) open‑weight models can be cost‑competitive with proprietary models when paired with good tooling, and (3) open‑ended use cases remain largely unsolved by current systems.

Key Points

• Dataset scale and scope

  • 220 queries, 11,543 rubrics (avg ~52.5 rubrics/query).
  • Six use cases spanning the full investor workflow: Financial Data Extraction; Company Research; Earnings & Events; Coverage & Catalyst Monitoring; Screening & Discovery; Sector, Industry & Macro.
  • Each rubric is binary (fulfilled/unfulfilled) and attributed to a public data source; every query is timestamped (agents answer as of that date); predictive queries are excluded.
  • Public release includes dataset and grading code.

• Benchmark design and novelty

  • Focus on open‑ended, long‑form research tasks rather than narrow extraction benchmarks that many models have saturated.
  • Rubric-level source attribution and expert-authored, checklist-style criteria enable objective, reproducible scoring (rubric qualification rate = fraction of rubrics satisfied; judged by majority of 3 independent LLM judges).
  • Difficulty calibrated with Bradley–Terry pairwise comparisons (∼77k internal pairwise judgments) and split into easy/medium/hard terciles.

• Data and evidence mix

  • Rubrics draw from a broad set of public sources; SEC filings are the largest single category but account for <40% of rubrics overall.
  • 74% of rubrics are factual data extraction; the remainder covers qualitative, analytical, forward‑looking and comparative requirements.
  • Use cases differ in source mix: Screening & Discovery and Sector/Industry/Macro are more driven by market data, professional knowledge, news and regulatory sources.

• Evaluation results (examples)

  • Samaya’s in‑house agent system achieved the top overall qualification rate at 56.0%.
  • Claude Fable 5 (frontier proprietary model, deployed under an open harness) scored 49.2%.
  • Best open‑weight model tested (Kimi K3) scored 46.4% and was materially cheaper to run.
  • Reported cost tradeoffs: Samaya’s system delivered higher quality at roughly 2.2× lower cost relative to Claude Fable 5 in the reported harness; Kimi K3 nearly matched best proprietary performance at ~4.5× lower cost.
  • Hardest use cases: Screening & Discovery (best systems ~33%) and Sector/Industry/Macro (~39%).

• Agent behavior and failure modes

  • Tool use tends to follow three phases across systems: (1) data gathering, (2) mid‑rollout synthesis & analysis, (3) final answer preparation.
  • High‑performing systems tend to use tools more efficiently and converge on similar token budgets despite different tool‑call volumes.
  • Failure pattern: models that shortcut to parametric knowledge of known sources (direct URL targeting) have higher URL error rates, causing token waste and context pollution.

Data & Methods

• Curation pipeline

  • Four-stage expert pipeline: (1) query drafting by finance professionals (buy‑side, sell‑side, IB), (2) dense rubric authoring with source attribution, (3) expert + AI‑assisted cleaning and multi‑stage auditing to ensure objectivity and public accessibility, (4) taxonomy‑based rebalancing and stratified sampling to produce the 220‑query public set (∼4,300 additional queries retained for internal use/future releases).

• Rubric taxonomy and tagging

  • Rubrics tagged by source (10 top‑level categories), essentiality (must‑have vs supplementary), and rubric category (8‑category content taxonomy).
  • Queries tagged with one use case and 1.9 average capability tags (e.g., temporal filtering, cross‑entity triangulation, qualitative synthesis).

• Difficulty calibration

  • Pairwise comparisons of queries scored along retrieval breadth, reasoning depth, entity scope, time scope, and qualitative ambiguity.
  • Aggregation via a confidence‑weighted Bradley–Terry model to produce per‑query difficulty scores and tercile bins (easy/medium/hard).

• Evaluation harness

  • Systems evaluated under a shared harness constrained to publicly available data; metrics measured include rubric qualification rate (quality), cost, and latency.
  • Scoring uses three independent LLM judges per rubric to form majority verdicts; checklist aggregation yields interpretable failure analysis by rubric/use case/source.

Implications for AI Economics

• Value of orchestration and tooling vs. model alone

  • The quality–cost frontier is strongly determined by the engineering of tool harnesses and retrieval orchestration, not just raw model capability. Firms that invest in retrieval pipelines, tool integration, and efficiency engineering can obtain substantial competitive advantage even with similar or smaller LLMs.
  • Economic implication: downstream value capture will favor organizations that bundle models with superior agent infrastructure (search, source‑handling, prompt/tool management), increasing returns to engineering and platformization.

• Cost sensitivity and open‑weight competitiveness

  • Open‑weight models (e.g., Kimi K3) can approach proprietary performance at materially lower operating costs. This reduces barriers to entry for smaller firms and may compress margins for model providers who charge on a per‑use basis.
  • Investors and firms should model total cost of ownership (model inference + agent orchestration + human supervision) rather than model quality alone.

• Labor and automation prospects

  • Even top systems perform well on narrower extraction tasks but struggle on open‑ended screening and macro/sector analysis. This suggests partial automation: routine data extraction and structured report drafting are nearer‑term substitution candidates, while high‑level discovery, thematic strategy, and nuanced macro judgment remain human‑centric in the near term.
  • However, improvements in tooling and orchestration could accelerate capability progression; the benchmark’s failure modes point to high‑leverage engineering improvements (e.g., robust URL handling, more efficient retrieval) that could materially raise effective automation.

• Measurement, standards, and market effects

  • A reproducible, publicly available benchmark with timestamped queries and source‑attributed rubrics enables transparent measurement of progress, helping buyers and regulators evaluate claims about agent competence and safety in finance.
  • Standardized, rubric‑level metrics shift evaluation toward what analysts need (verifiable evidence coverage), which may influence contracting and procurement: customers may demand agents be evaluated and certified on checklist‑style criteria tied to public sources.

• Research and investment implications

  • High value from improving agent orchestration suggests investment priorities: retrieval quality, source‑attribution, tool‑use policies, and robust error handling (to reduce token waste and downstream risk).
  • The benchmark highlights specific, economically relevant research areas—open‑domain screening, sector/macro synthesis, multi‑entity triangulation—that, if solved, could unlock automation of high‑value research tasks.

Caveats and considerations - Authors are from Samaya AI; reported results and system comparisons come from experiments run under a common harness but may reflect implementation choices or internal optimizations. - Scoring relies on LLM judges (3‑model majority); while checklist rubrics improve interpretability, judge sensitivity and prompt/model choice can still influence outcomes. - The benchmark restricts evidence to public sources, which approximates many real deployments but excludes paywalled datasets some firms rely on.

Links and provenance - Paper/report: August 13, 2026 (Samaya AI technical report). - Dataset and grading code: publicly released by the authors (see FrontierFinance project page: https://research.samaya.ai/benchmarks/frontier-finance). Contact: research@samaya.ai.