A specialized IPO benchmark finds modern LLMs can handle long, complex S-1 filings: Alibaba Qwen 3.7 Max scores 79.4% on SpaceX IPO diligence questions, while Xiaomi MiMo-2.5 Pro offers near-comparable accuracy at a fraction of the cost; improvements rely on contextual retrieval tuned for long documents.

Main Finding

IPO Finance Agent extends Finance Agent v2 to evaluate LLM financial analysts on IPO (S-1) due-diligence tasks by (1) introducing a 1,000-question IPO-diligence benchmark (70 SpaceX S-1 questions publicly released), (2) replacing naive chunk retrieval with contextual, context-enriched embeddings for long, cross-referential S-1 filings, and (3) automating rubric generation via an evaluator-optimizer pipeline. Using this harness, several models outperform the Finance Agent v2 frontier: Alibaba Qwen 3.7 Max achieves 79.4% accuracy at $0.30/query and Xiaomi MiMo-2.5 Pro achieves 76.8% at $0.05/query, demonstrating that retrieval architecture and rubric methodology materially affect measured capability and cost-efficiency for financial research tasks.

Key Points

• Motivation

  • S-1 IPO prospectuses differ from periodic filings (10-K/10-Q): long documents, pro forma/common-control accounting, governance disclosures, underwriting-sensitive risk language, and evidence dispersed across distant sections.
  • Finance Agent v2’s public harness (unenriched fixed-size chunk retrieval) failed on the SpaceX S-1, motivating architectural changes.

• Contributions

  • Dataset: 1,000 IPO-diligence questions across six domains; 70 SpaceX S-1 questions released publicly, rest held private to limit contamination.
  • Retrieval: Contextual retrieval—each chunk augmented with document-aware context before embedding—designed to surface evidence scattered across long S-1s.
  • Evaluation: Automated evaluator-optimizer pipeline that (a) extracts candidate facts from ensemble model answers, (b) consolidates them into draft rubric items, (c) iteratively audits and repairs coverage/structure via LLM feedback, and (d) ends with human expert review.
  • Public code and data: GitHub release (link in paper).

• Empirical results

  • Best absolute accuracy: Alibaba Qwen 3.7 Max — 79.4% (cost $0.30/query).
  • Best cost-efficiency on Pareto frontier: Xiaomi MiMo-2.5 Pro — 76.8% (cost $0.05/query).
  • Both outperform Finance Agent v2 leaderboard ceiling (Google Gemini 3.5 Flash: 57.9% at $2.51/query) and undercut FABv2’s cheapest entry on cost-efficiency.
  • Demonstrates that retrieval/harness changes can yield large gains relative to prior benchmark scores.

• Design choices to limit bias/contamination

  • Private/public split modeled on Finance Agent v2 to limit benchmark memorization.
  • Professional-workflow labels for questions (investment banking, public markets, VC, credit, securities counsel, accounting advisory) to enable slicing by intended audience.

Data & Methods

• Case study and corpus

  • Flagship public case: SpaceX (SPCX) S-1 — chosen for complex segment structure (Starlink, launch business, xAI exposure), common-control recasts, and reliance on secondary-market pricing.
  • Private dataset: additional S-1 filings across sectors to reach 1,000 questions total.

• Question taxonomy (six IPO domains)

  • Segment economics & operating performance
  • KPI quality & monetization
  • Governance & control structures
  • Accounting & common-control recast mechanics
  • Execution risk & capital requirements
  • Valuation & underwriting analysis

• Retrieval & agentic harness

  • Tooled agent environment with EDGAR and web search (like Finance Agent v2).
  • Contextual retrieval: prepend chunk-specific, document-level context (generated by an LLM) to each chunk prior to embedding and indexing, improving retrieval when relevant evidence is lexically distant from the question.
  • Motivated by failure of fixed-window chunking on long S-1s where footnotes and narrative sections must be linked.

• Automated rubric generation pipeline

  • Ensemble of independently generated model answers → candidate fact extraction.
  • Consolidation/normalization into draft rubric items.
  • Iterative evaluator stage (LLM critiquing) to find omissions, hallucinations, mistiered items, redundancy.
  • Route defects to scope-limited repair or targeted re-extraction (enrichment).
  • Final deduplication and human expert review.

• Evaluation metrics & cost accounting

  • Accuracy measured against multi-item, dealbreaker-gated rubric-style grading.
  • Cost-per-query computed for models; Pareto frontier constructed to show accuracy/cost tradeoffs.
  • Models evaluated included frontier and open models (examples: Alibaba Qwen 3.7 Max, Xiaomi MiMo-2.5 Pro, Google Gemini 3.5 Flash).

• Reproducibility & release

  • Public release: 70 SpaceX S-1 questions and code on GitHub; remaining questions held private.

• Limitations noted by authors

  • Public benchmark portion focuses on a single, complex S-1 (SpaceX) — public slice is intentionally small to avoid contamination.
  • Rubric automation still ends with human expert review; fully unsupervised grading not claimed.
  • Specific technical hyperparameters for contextualization and embedding choices are in the codebase (implementation detail).

Implications for AI Economics

• Benchmark and measurement effects

  • Demonstrates that benchmark outcomes can be dominated by retrieval/harness design, not only by model capability. For economic analysis of LLMs, reported model improvements may partially reflect engineering/harness gains rather than pure model advances.
  • Validates that domain-tailored retrieval (contextualized embeddings) is critical when economic decisions rely on long, cross-referential documents (e.g., IPO prospectuses).

• Cost-performance tradeoffs and deployment

  • Pareto frontier results show low-cost models can approach the accuracy of higher-cost frontier models when paired with improved retrieval and evaluation harnesses. This matters for firms deciding which LLMs to deploy for financial due diligence under budget constraints.
  • Cost-per-query metrics tied to accuracy provide a quantitative basis for productization and procurement decisions in financial services (which care about both runtime cost and legal/regulatory risk from errors).

• Evaluation automation and scaling

  • The evaluator-optimizer rubric pipeline points to scalable ways to construct graded benchmarks across specialized disclosure regimes, reducing expert annotation costs. For researchers and firms, this lowers barrier to creating tailored evaluations for regulatory filings, tax disclosures, or sector-specific reports.
  • However, reliance on LLMs to synthesize rubrics introduces a feedback loop: if training data or model biases influence rubric construction, evaluations risk reinforcing systematic errors—important for economic analyses of AI-driven auditing tools.

• Policy and risk considerations

  • Improved LLM performance on IPO diligence tasks increases the feasibility of automating due-diligence workflows, with implications for labor (analyst roles), market efficiency, and regulatory oversight.
  • Regulators and firms should note that harness engineering (retrieval/context) materially affects outputs; disclosures or model audits that ignore retrieval architecture may understate systemic risk.
  • Private/public split to avoid contamination highlights an ongoing challenge: benchmarks can be gamed by model training. Economic research relying on benchmarks must account for the temporal and access dynamics of evaluation data.

• Research directions for AI economics

  • Evaluate broader generalization across S-1s and other long regulatory documents to quantify how much harness changes vs model scaling contribute to real-world performance gains.
  • Study labor–capital substitution: how improved, cheaper LLM+retrieval stacks change analyst staffing, turnaround times, and pricing of research services.
  • Investigate market effects if intermediaries adopt LLM-based due diligence at scale (e.g., faster IPO pricing, changes in underwriting spreads, or altered information asymmetry).

If you want, I can (a) extract the 70 publicly released SpaceX questions from the repository and summarize them by domain, (b) produce a short checklist for deploying a similar contextual-retrieval + evaluation pipeline in a production setting, or (c) highlight specific failure modes observed in the paper with examples. Which would you prefer?