A new benchmark of real-world finance workflows finds top AI agents complete fewer than 40% of tasks: GPT-5.1 spends nearly 17 minutes per workflow yet passes only 38.4% of cases, exposing persistent failure modes on messy, multimodal enterprise work.

Main Finding

FINCH (FinWorkBench) is a new, enterprise-grade benchmark of realistic finance & accounting (F&A) workflows that shows state-of-the-art frontier LLM agents—which perform strongly on curated tasks—still fail on the messy, long-horizon, multimodal, and collaborative workflows analysts actually do. Even the best evaluated agent (GPT‑5.1 Pro) completes only 38.4% of workflows under human evaluation, highlighting that real-world operational complexity (cross-file dependencies, irregular spreadsheets, multimodality, and long task chains) is a major barrier to productive automation.

Key Points

• Dataset scale and provenance

  • 172 annotated composite workflows (384 tasks) built from authentic enterprise artifacts.
  • 1,710 spreadsheets (956 distinct sheets in 301 Excel files), ~27 million cells total, plus 17 PDFs, 12 images, Word docs, CSV/JSON, etc.
  • Sources: Enron email/spreadsheets (≈15k spreadsheet files, 500k emails), EUSES, and public/industry reports (World Bank, government financial statements, investment firm models).
  • Public release: https://huggingface.co/FinWorkBench

• Workflow construction & annotation

  • Novel pipeline grounded in real collaborative context: (1) mining enterprise email threads with LLM assistance, (2) deriving workflows from versioned spreadsheet diffs using LLMs, (3) authored tasks from high-quality deliverables.
  • LLM-assisted drafts refined and validated by experts; annotators edit input/reference files to cleanly realize workflows.
  • Over 700 hours of expert annotation and iterative QC; each workflow reviewed by multiple experts.

• Task and data characteristics

  • Highly compositional: 78.5% of workflows contain multiple interdependent tasks (typical workflows interleave structuring, import, cross-file retrieval, calculation, modeling, validation, visualization, reporting).
  • Messy, large, and multimodal inputs: median workflow covers ~15K cells (mean 157K), formula-dense but heavy long-tail (largest workflow ~3.7M cells).
  • Task categories: Calculation (119 workflows), Structuring/Formatting (86), Data Entry/Import (44), Validation/Review (37), Cross-sheet/file Retrieval (36), Summary/Visualization (33), Financial Modeling (15), Web Search (11), Translation (3).
  • Tool-call complexity: median multi-call interactions are high (case studies report median ~16 tool calls; many workflows require dozens of calls).

• Evaluation methods and results

  • Dual evaluation: human expert judgement (gold standard) and a scalable multimodal LLM-as-judge pipeline using structured diffs, snapshots, and multimodal renderings.
  • Systems evaluated: GPT‑5.1 (GPT 5.1 Pro), Claude Sonnet/Opus (4.5/4.6), Gemini 3 Pro, Grok 4, Qwen 3 Max.
  • Top performance: GPT‑5.1 Pro averaged 16.8 minutes per workflow and achieved 38.4% pass rate (human eval). All evaluated systems passed <50% of workflows.
  • Performance degrades sharply with composition and multimodality: GPT‑5.1 Pro pass rate falls from 44.3% on workflows with ≤2 tasks to 23.5% on those with >2 tasks; workflows involving PDFs/images had pass rate ≈35%.

Data & Methods

• Workflow discovery

  • Email-thread grounding: GPT-5 identifies threads that state business goals and reference attachments; LLM drafts normalized into task instructions; annotators verify/complete.
  • Version-diff derivation: collect version chains from repositories; LLM proposes diffs and workflow descriptions; experts validate and produce paired input/reference artifacts.
  • Deliverable-based tasks: domain experts author tasks from final reports and high-quality spreadsheets (e.g., valuation models, World Bank reports), sometimes adapting prior datasets (WideSearch, DABStep).

• Annotation & Quality Control

  • Team of five expert annotators (mix of finance/CS backgrounds, industry experience); workflows created and cross-reviewed.
  • Iterative QC: ~40% of workflows revised at least once; some >3 rounds; LLMs used as secondary checkers but human verification required.
  • Annotation cost: >700 expert hours.

• Evaluation pipeline

  • Human evaluation: experts judge end-to-end completion and fidelity (structural and semantic correctness).
  • Automated LLM-as-judge: multimodal comparisons using structured diffs and rendered snapshots to scale evaluation; validated against human judgements.

Implications for AI Economics

• Productivity estimates for LLM-driven automation in F&A should be tempered

  • Benchmarks built from isolated, clean tasks overstate real-world gains. FINCH shows frontier agents frequently fail on real enterprise workflows, so projected labor displacement or efficiency gains may be optimistic unless tooling and integration improve.

• Economic value depends on robust, compositional agent capabilities and tooling

  • The bottlenecks are cross-file retrieval, robust spreadsheet understanding (irregular layouts, merged cells, latent formulas), long-horizon planning, multimodal parsing (PDFs, images), and faithful editing/formatting. Investments that improve these specific capabilities (specialized parsers, tools for provenance, file-system-aware agents, verifiable tool chains) will likely yield the greatest productivity returns.

• Labor and skill impacts

  • Near-term: humans will remain essential for oversight, validation, and handling edge cases; demand may shift toward higher-skill work (prompt/tool orchestration, auditing, model-assisted verification).
  • Training/upskilling: organizations must invest in employee skills to use LLM agents safely (e.g., validation protocols, prompt engineering, spreadsheet hygiene).

• Risk, compliance, and governance implications

  • High rates of partial or incorrect automation in financial workflows raise operational and regulatory risks (misreporting, valuation errors). Economic adoption will require stronger audit trails, deterministic tool integration, and human-in-loop checks—creating demand for compliance tooling and explainability features.

• Market and research signals

  • There is economic space for specialized enterprise agents and tool stacks that natively understand spreadsheet ecosystems, versioning, and multimodal financial documents.
  • Benchmarks like FINCH create clearer product-market fit signals: firms that target these failure modes (cross-file reasoning, multimodal grounding, deterministic execution, verifiable outputs) can capture high-value enterprise use cases.

• For economists and policymakers

  • Empirical, task-grounded benchmarks (not just isolated benchmarks) are necessary to forecast labor-market impacts of AI in professional sectors.
  • Regulation and procurement for mission-critical automation should require evaluation on realistic, compositional benchmarks (e.g., FINCH) to assess systemic risk.

If you want, I can: - Extract key numeric statistics into a one-page quick-reference. - Summarize failure modes with illustrative examples from the paper for a policy brief.