LLMs write fluent financial prose but fall short of institution-grade reports; a new 244-task benchmark and 35-item rubric identify report identity, source discipline and visual delivery as the main failure modes. Applying benchmark-guided generation and self-review rules lifts key rubric scores (G1 by ~34 points, G2 by ~14 points) across multiple model families.

Main Finding

FinReportBench is a purpose-built, expert-grounded benchmark and improvement pipeline that measures whether LLM-generated financial reports meet institution-grade delivery standards. The authors show that fluency alone is insufficient: while basic deliverability is nearly saturated across evaluated models, critical failures in report identity and institutional completeness (traceability, information density, source/ chart discipline, and visual/institutional framing) remain the main barriers to institutional use. They also demonstrate a failure‑grounded skill-evolution method that converts recurring item-level failures into compact generation + self-review constraints, producing substantial gains in identity/completeness scores without degrading deliverability.

Key Points

• Problem framed: LLM outputs can be fluent yet rejected by professionals because they omit institutional conventions (publisher, analyst identity, page system), provenance discipline, risk/disclosure elements, and consistent visual delivery.
• Rubric and hierarchy: FinReportBench introduces a hierarchical rubric with three layers (G0–G1–G2):
  • G0 (deliverability): 4 preflight checks (visible, readable, continuous, non-broken). Any G0 fail → total = 0.
  • G1 (report identity): 4 items testing whether the artifact is recognizable as institutional research.
  • G2 (institutional completeness): 27 items covering front-page framing, institutional identity, compliance, page systems, information density, source/chart discipline, and generation-artifact control.
  • Total rubric = 35 observable items; items scored pass/partial/fail (1/0.5/0) with hierarchical aggregation.
• Rubric induction: created via a two-stage, expert-guided process. Experts provide partial orders on small output sets; a multimodal model mines contrastive evidence; candidate items are induced, then experts audit and refine items. This makes scarce expert effort efficient and yields bounded, observable criteria.
• Dataset: 244 bilingual tasks (122 Chinese, 122 English) curated from a balanced 10,000-task reference pool (5k Chinese analyst reports + 5k English institutional publications). Tasks cover company, industry, and macro objects; T0 (open-ended) and T1 (thesis-guided) inputs. The selected suite covers 95.65% of the 10k semantic space at cosine ≥ 0.60. Each task contains a reconstructed research trajectory: public query, search plan, retrieved evidence, structured findings, research note, and a hidden source packet (the source artifact is held out from the generator).
• Multimodal judging protocol: rendered HTML reports + extracted text evaluated under the 35-item rubric. Validation shows three independent judge families reproduce expert partial orders at near-ceiling rates.
• Benchmark evaluation: nine model families were run (including MiniMax M2.7, GLM-5.2, DeepSeek variants, Qwen variants, Kimi). Basic deliverability (G0) was nearly saturated; major deficits concentrated in G1 and G2. Example provisional totals (no-skill runs): MiniMax M2.7 top with provisional overall 22.6 (G1 40.0, G2 46.6); other models scored lower (reported but interpreted cautiously).
• Skill evolution: a pipeline turns recurrent item failures observed on external, query‑disjoint cases into compact plan/write/review constraints (a candidate skill). Candidates are validated on held-out external cases and accepted only if they improve targeted items without regressions. The frozen best skill K* was then paired with no-skill runs on the locked FinReportBench test set. Across five model families, the evolved skill increased mean G1 by 33.85 points and mean G2 by 13.83 points while preserving G0 for every paired comparison.
• Practical outputs: full artifacts, code, and benchmark materials are released (link in paper).

Data & Methods

• Source pool and task synthesis:
  • Collected 5,000 Chinese analyst reports (Eastmoney) and 5,000 English institutional publications (World Bank).
  • Reverse-synthesized plausible client requests from each source; LLM-based audit filtered unnatural/ leaking tasks.
  • For selected tasks, separated public query card from hidden source mapping (sources withheld during generation).
• Coverage-oriented curation:
  • Embedded tasks into a multilingual semantic space; selected 244 tasks balancing language, research object, domain, and query specificity.
  • Coverage statistics: 122 Chinese + 122 English; 61 company / 92 industry / 91 macro tasks; 87 T0 / 157 T1.
• Research trajectory construction:
  • For each task, scripted a deep-research trajectory: public query, search plan, retrieved evidence, structured findings, research note. Tasks exclude benchmark identifiers and the hidden source artifact is not provided to models.
• Rubric induction and evaluation design:
  • Experts supplied within-query partial orders on small anonymized report sets.
  • A multimodal contrastive model mined visual and textual differences between preferred/dispreferred outputs to suggest candidate observable items.
  • Experts audited and revised items in a second round; resulting frozen rubric has 35 items with explicit pass/partial/fail boundaries and item weights.
  • Evaluation protocol renders HTML outputs, extracts text, and requires judges to provide item decisions plus brief evidence citations.
  • Hierarchical scoring formula: G0 pass needed; G1 aggregated into S1, which scales G2 contribution (see paper for formal equation).
• Skill evolution pipeline:
  • External discovery: run models on separate external cases, convert rendered outputs into item-level failure/evidence data.
  • Optimizer crafts candidate compact constraints (when applicable; what to do; how to check).
  • Critic prunes task-specific or disallowed instructions; candidate validated on held-out external validation cases, accepted only if no regressions and targeted improvements.
  • Best accepted candidate frozen as K and then evaluated on the locked FinReportBench set under paired no-skill vs K runs.
• Judges: evaluation used GPT-5.6 Luna at medium reasoning effort to assist item-level judgments in the main run; human expert partial-order validation used for calibration.

Implications for AI Economics

• Evaluation economics: FinReportBench illustrates the value of task- and deliverable-specific benchmarks for economically meaningful evaluation. Aggregate fluency or generic presentation scores can mask the institutional readiness problems that determine commercial adoption and liability exposure. Firms and researchers should budget for finer-grained, expert-grounded evaluation when assessing model readiness for regulated/high-stakes domains.
• Productivity vs. oversight trade-off: the work suggests that LLMs already meet basic deliverability (file integrity, readable output) but fail on institutional requirements that enable unassisted deployment. Economically, LLMs can reduce report drafting costs but will still require higher-skilled human oversight (ensuring provenance, compliance, and identity), implying a shift in labor from drafting to review and governance.
• Market differentiation and adoption: firms that operationalize failure‑grounded skills (compact plan/write/review constraints) may gain competitive advantage by achieving institutional readiness more quickly and with lower oversight cost. This can accelerate automation for lower-risk report classes while preserving premium human review for sensitive outputs.
• Risk management and compliance: the rubric emphasizes provenance, transparency, and generation-trace control—features important for auditability and regulatory compliance. Benchmarks like FinReportBench can thus be used to measure and certify models or workflows against regulatory expectations, reducing information asymmetry between model vendors and institutional buyers.
• Incentives for modular improvements: the success of a compact, transferable skill distilled from recurring failures points to an economic strategy: invest in small, generalizable production rules and review toolchains (rather than full retraining) to capture outsized gains across model families. This can be cost-effective for institutions that integrate multiple model suppliers.
• Research & policy priorities: results highlight priorities for investment—traceability, structured provenance, information density, chart/source discipline—rather than further marginal gains in prose fluency. Regulators and standard-setting bodies may adopt similar itemized checklists to define minimum acceptable standards for automated reporting.
• Measurement of externalities: by making hidden sources and trajectories part of the evaluation design, the benchmark reduces incentives for models to hallucinate plausible-seeming but unverified claims; this can reduce reputational and systemic risk in financial markets if widely adopted.

If you want, I can: - Extract the 35 rubric items and summarize them by subcategory. - Produce a concise checklist for integrating the K* skill into a report-generation pipeline. - Map these findings to potential regulatory audit requirements or compliance checklists.