Large language models often read but do not use long financial disclosures: while retrieval accuracy stays high as filings grow, the factual content's influence on model investment judgments vanishes with longer context — a targeted, structured restatement placed next to the decision (not generic summarization) restores the facts' effect.

Main Finding

When large language models are used as AI analysts of long financial filings, they can accurately retrieve specific disclosures but often fail to integrate those facts into downstream investment judgments as document length grows. This "retrieval–integration gap" widens with context size: retrieval remains accurate up to 128k tokens while the marginal influence of a focal disclosure on a model’s investment judgment falls to noise (or is greatly reduced). Model capability postpones but does not eliminate the gap. Workflow architecture — how read information is routed to the decision point — determines whether retrieved facts actually affect the judgment; a targeted, structured restatement placed adjacent to the decision restores influence, whereas generic chunk-and-summarize pipelines and extended reasoning do not.

Key Points

• Definition: Marginal decision influence = change in the model’s investment judgment when a specific disclosure is present versus when the same location contains neutral text, holding all other firm information fixed.
• Core empirical pattern:
  • At 2,000 tokens, a constructed quantitative risk disclosure raises the model’s sell probability by ~3.2 percentage points (economically meaningful).
  • Between ~8k and 32k tokens the disclosure’s influence becomes statistically indistinguishable from neutral insertions; it remains at that empirical noise floor up to 128k tokens.
  • Direct retrieval of the disclosure remains accurate even at 128k tokens (primary model retrieved for all 12 test firms; no false retrievals on neutrals).
  • More capable models push the failure point outward (e.g., the largest open-weight model preserves ~3.4-pp influence at 128k), but do not eliminate the gap.
• Replication and robustness:
  • Pattern replicates across three independently trained model families and in an exploratory experiment on real quantified disclosures from 10-K filings (removing the passage changed judgments in excerpted short contexts but not in the full filing).
• Mechanism (internal channels):
  • Two transmission channels carry read information into the decision: (1) a compressed recurrent/summary state and (2) attention-based lookup over source text.
  • Causal interventions:
    • Transplanting the compressed state between matched documents removes roughly two-thirds of a disclosure’s influence when the disclosure is absent from that state.
    • Implanting the disclosure into the compressed state recreates nearly half the influence in documents lacking the source passage.
    • Matching attention-blackout experiments identify a disclosure-specific lookup effect comparable in magnitude to the compressed-state effect. The two channel effects are statistically indistinguishable.
  • Conclusion: The disclosure is encoded and retrievable, but the channels that transmit it into the judgment weaken as competing context grows — an integration failure rather than a comprehension failure.
• Workflow remedies:
  • Enabling extended chain-of-thought/reasoning does not restore influence and can reduce it at short lengths.
  • Generic chunk-and-summarize pipelines eliminate the disclosure’s influence at all lengths (even 2k) because bounded notes omit the target information before decision stage.
  • Verbatim repetition of the disclosure near the decision is insufficient.
  • A targeted, structured restatement (either model-extracted or experimenter-written) that summarizes decision-relevant facts and is placed immediately before the investment judgment while the source filing remains available restores influence — e.g., raising the disclosure’s effect at 128k to ~8.5 percentage points with consistent directional responses across firms.
• Conceptual contributions:
  • Introduces and operationalizes the retrieval–integration gap.
  • Proposes marginal decision influence as an evaluation metric aligned to delegated decision tasks.
  • Offers a "decision-proximal representation" principle: extract and place crucial facts in a structured form adjacent to the decision point rather than relying on capacity-limited generic summaries.

Data & Methods

• Experimental setup:
  • Twelve U.S. registrants (firm-level experiments).
  • For each firm, authors constructed a coherent, quantitative, decision-relevant disclosure (examples: covenant threshold, settlement payment, indemnification cap) embedded in the filing text.
  • Treatment variation: hold focal-firm information fixed; vary only the surrounding, genre-matched but economically unrelated text length from 2,000 up to 128,000 tokens. For each focal location, compare the model’s judgment with the true disclosure present versus five equal-length neutral replacements.
  • Separate retrieval tests: in independent calls scored against frozen answer keys, measure whether the model can correctly retrieve the target fact.
  • Judgment metric: change in model’s sell probability (and other decision scaling tests across a seven-step severity ladder).
• Models tested:
  • Multiple model families and sizes, including a primary open-weight model, larger open-weight variants, and a higher-capability production model (commercial). The paper reports consistent qualitative patterns across families; capability shifts the context-length where integration fails.
• Probing and causal interventions:
  • Representation analyses (probes) measured encoded disclosure content at the original read location and at the decision position.
  • State-transplant experiments: swap compressed recurrent/summary states between matched documents to measure contribution of compressed state.
  • Implantation experiments: inject disclosure content into compressed state.
  • Attention-blackout experiments: selectively disable attention-based lookup to measure lookup contribution.
• Workflow experiments:
  • Compared architectures: chunk-and-summarize pipeline; enabling extended reasoning; verbatim repetition of disclosure near decision; targeted structured restatement placed immediately before judgment while keeping source available.
• Real-document robustness:
  • Exploratory tests on 20 complete 10-K filings with self-contained quantitative disclosures: removing the original passage affected judgment in short excerpts but not in the full filing, although retrieval remained accurate.

Implications for AI Economics

• Evaluation and validation:
  • Evaluations that rely only on retrieval accuracy or on intermediate, observable behaviors (e.g., quoting a passage) can be misleading. Systems should be validated by outcome-aligned metrics (e.g., marginal decision influence) that measure whether retrieved facts meaningfully change the delegated judgment.
  • Advertised context window (e.g., 100k+ tokens) is not a guarantee that the effective decision context equals that window; capability and workflow jointly determine effective decision scope.
• Product and workflow design:
  • Model capability improvements help but are not sufficient. Workflow architecture matters: design pipelines to produce decision-proximal, structured representations of critical facts and place them adjacent to the decision step.
  • Avoid naive chunk-and-summarize pipelines for delegated decision tasks; they can drop crucial facts before the decision stage.
  • Practical remedy: include targeted restatements or structured summaries of decision-relevant facts at the decision point while retaining source access.
• Market and disclosure economics:
  • Generative AI shifts disclosure-processing frictions from acquisition to integration. Firms and regulators should recognize that making disclosures machine-retrievable does not ensure machine decision usefulness.
  • Firms may have incentives to restructure filings (or the placement/formatting of key facts) if AI analysts are sensitive to proximity and structure — creating potential strategic effects on disclosure design and information diffusion.
  • Investors and intermediaries using AI analysts should not equate a model’s ability to quote a fact with the model actually using that fact in its recommendation; misplaced trust in retrieval signals could lead to misinformed decisions and trading responses.
• Policy and governance:
  • Regulators evaluating machine readability should complement it with tests for machine decision usefulness in task-relevant configurations.
  • Organizational management of AI should treat models as embedded systems whose outputs depend on both model internals and workflow architecture; governance should include pipeline audits (not just model audits).
• Research directions:
  • Explore generality across decision tasks (e.g., forecasting, credit risk), other model families and architectures, and interactions with human oversight (acceptance/override).
  • Investigate architectural remedies (beyond workflow) that preserve integration across long contexts (memory architectures, retrieval-augmented generation with decision anchors).
  • Study market-level responses to disclosure placement strategies aimed at influencing AI readers.

Limitations (as noted by the authors) - Focuses on AI processors (models) after delegation; does not study human users’ trust, acceptance, or override behavior. - Constructed disclosures are used for controlled identification; exploratory real-document tests are limited in scope. - Exact model identities and hyperparameters are not fully specified in the draft; effects are reported in relative terms across sizes/families.

Overall, the paper highlights a critical, practically relevant failure mode for AI analysts: being able to "read" (retrieve) does not imply "use" (integrate into decisions). Correct evaluation, design, and governance must span both model capability and workflow architecture to ensure that retrieved information actually changes delegated economic judgments.