A new professional-grade benchmark finds leading LLMs still fall far short of equity analysts: the best agent matches analysts in just over half of cases and false-positive rates span roughly 1%–32%, revealing important trade-offs among accuracy, cost and reliability.

Main Finding

Frontier Financial Judgement evaluates agents on the hard, realistic task of detecting genuinely new, valuation-relevant company news and estimating its importance and directional impact. No agent reliably matches expert equity-analyst judgments: the best agent achieves 52.4% "all-label" accuracy (all three labels correct) and 71.1% atomic accuracy (individual-label accuracy). False‑positive behavior and operational costs vary widely across agents, producing substantial trade-offs that hinder practical deployment of automated news filtering.

Key Points

• Task and labels
  • Agents must assess whether an article contains information newness (new vs not new), expected importance (none/low/medium/high), and direction (positive/negative/neutral/unclear).
  • Evaluations count an answer correct if it matches the expert primary label or accepted secondary label in genuine boundary cases.
• Dataset and scope
  • v1 contains 82 expert-designed synthetic events embedded in 656 items (each case: 1 synthetic article + 5 recent real articles + 2 historical documents).
  • Focus area: semiconductor supply-chain companies (e.g., ASML, NVIDIA, Ciena, Infineon).
  • All data frozen as of 17 July 2026.
• Benchmark construction
  • Synthetic events are authored with professional equity analysts and rendered into realistic article formats by LLMs; web-page chrome added to mimic live pages.
  • Each case supplies agents a locked bundle and permits web search; this mixes control (preventing label leakage) with realistic distractors.
• Scoring and operational metrics
  • Reported metrics: atomic accuracy (per-label), all-label accuracy (all three labels), false-positive proxy (distractor articles flagged as new + importance>none), valid-output counts, token usage, cost, and search volume.
  • False-positive proxy rates vary dramatically across agents (paper reports a range roughly ~1% up to ~32–35% depending on agent).
• Results summary (selected highlights from Table 3)
  • Best agent: GPT-5.5 — all-label 52.4%, atomic 71.1%, 82/82 valid answers, cost ~$76 total for the evaluation run.
  • Close second: GPT-5.6 Sol — all-label 51.2%, atomic 69.1%, cost ~$83.
  • Other agents show steep drops: next best ~46.3% all-label (GPT-5.6 Terra), then a cluster down through ~30% and below.
  • Significant operational variance: e.g., Claude Opus 4.8 had comparable atomic accuracy to some top models but much higher false-positive proxy and a much higher evaluation cost (~$206 reported).
  • Some models exhibited poor reliability (many malformed or missing outputs), e.g., Nemotron 3 Super with only 28 valid outputs.
• Qualitative finding
  • High accuracy on synthetic target events does not guarantee low false-positive rates on surrounding live distractors — so precision in practice is not assured even for models that do well on labelled targets.

Data & Methods

• Expert generation and labeling
  • Professional hedge fund and equity analysts designed 82 plausible company events and provided gold labels + rationales.
  • Labels: Information newness; Expected importance (none/low/medium/high); Direction (positive/negative/neutral/unclear).
• Synthetic article generation
  • Analysts' event descriptions + supporting evidence + primer paragraphs were rendered into multiple journalistic styles by LLMs and wrapped in webpage chrome templates.
• Realistic context per case
  • Each case supplied: 1 labelled synthetic article, 5 live articles collected at or before cutoff, and 2 curated historical documents. All agents received the same locked bundle per case.
• Agent harness and evaluation
  • Agents evaluated as full configurations (model + harness + prompt + tool policy). Web search was allowed.
  • Execution and logging via Harbor in isolated containers. Agents returned structured JSON with labels, rationale, and evidence.
• Scoring
  • Atomic accuracy: proportion of the 246 label-classifications (82 cases × 3 labels) that were correct.
  • All-label accuracy: proportion of cases where all three labels were correct.
  • False-positive proxy: fraction of parseable distractor decisions classified as new and importance>none.
  • Operational measures: validity of outputs, token consumption, search calls, and dollar cost for the single evaluation pass.

Implications for AI Economics

• Practical readiness and economics of deployment
  • Current top agents are far from perfectly matching expert judgements — best-case: only ~50% full agreement. Deploying them as stand-alone news filters risks both missed signals and noisy alerts.
  • False-positive rates matter economically: high false positives impose analyst time costs and reduce utility of automated alerts. Models with similar target accuracy can differ greatly in false‑positive burden and cost.
  • Cost/throughput trade-offs are material. Some higher‑accuracy agents are substantially more expensive per unit of processing (token/compute/search). Firms must trade off per-alert cost versus precision and recall when scaling to large coverage universes.
• Market and behavioral effects
  • Automated but imperfect filtering could change information-processing bottlenecks in asset management firms, shifting human attention toward validation tasks. This may alter labor allocation and the value of human oversight.
  • If deployed publicly, inconsistent model judgments could affect short-term market reactions—especially if models systematically mislabel recombined or stale information as novel.
• Research and product priorities for AI + finance
  • Improve temporal grounding and retrieval: better retrieval of prior disclosures and rigorous point-in-time grounding are crucial to reduce false positives and stale-info misclassification.
  • Calibrated uncertainty and conservative alerting: models should output well-calibrated probabilities and abstain or downgrade alerts when uncertain to limit analyst overload.
  • Human-in-the-loop workflows: hybrid systems where model screening and human adjudication combine can capture scale benefits while limiting downstream costs from false positives.
  • Benchmark expansion: test other sectors, larger coverage universes, longitudinal evaluation (market outcomes), and adversarially recombined distractors to stress robustness.
  • Economic evaluation metrics: beyond accuracy, measure analyst-time saved, cost-per-actionable-alert, and false-alert externalities to align model objectives with real economic value.
• Broader policy/operational risks
  • Reliance on opaque agents for news-filtering without rigorous monitoring may lead to operational risk and misallocation of capital if systematic biases or failure modes (e.g., over-alerting from recombined content) are present.
  • Standards for evidence-grounded explanations and audit trails are important for compliance and for firms to validate why an alert was issued.

Summary takeaway: Frontier Financial Judgement shows substantial headroom before LLM-based agents can be trusted to replace or heavily automate expert news‑flow judgment. The benchmark demonstrates that accuracy, false-positive behavior, reliability, and cost must all be optimized together to unlock economically valuable automated financial news filtering.