State-of-the-art agentic LLMs become less accurate and far costlier when evidence is buried: across six flagship models accuracy fell (4–18 percentage points), tool calls and cost-per-correct rose several-fold, and models often maintained high confidence even when wrong—underscoring the need for claim-level provenance and auditing.

Main Finding

When frontier agentic models must find answers in realistic, noisy “data-room” folders (buried evidence) rather than being given only the answer-bearing documents (clean), their accuracy falls, they make more tool calls, they more often reach forced-decision budgets, and the retrieval/API cost per correct answer rises substantially. Confidence scores and standard benchmark calibration do not reliably flag many wrong answers; agents can pair accurate numeric outputs with fabricated or unsupported explanations. The paper argues that claim-level receipts (source-linked evidence for each statement), condition-aware scoring, and adversarial human verification are necessary for trustworthy deployment and audit of agentic QA.

Key Points

• Experimental contrast: each flagship model from six leading labs (OpenAI, Anthropic, Zhipu, Alibaba, Google, Meta) was run on a 41-item synthetic financial “room02” data-room benchmark in two conditions:
  • clean: only gold documents mounted; and
  • buried: full room including noise mounted.
• Buried vs clean effects (high-level):
  • Accuracy declines across all six models when evidence is buried. Example deltas (clean→buried accuracy change, Table 3b): OpenAI -4.1 pp, Anthropic -11.6 pp, Zhipu -8.9 pp, Alibaba -13.0 pp, Google -17.8 pp, Meta -8.9 pp.
  • Cost per correct answer increases dramatically (multiplicative): range 1.6× (Meta) to 7.0× (Anthropic), with OpenAI rising ~6.7× (from $5.63 to $36.09) and Google ~6.6× (from $18.81 to $99.31).
  • Mean tool calls per item increase (×1.6 to ×3.2); being wrong consumes similar retrieval effort as being right (nontrivial economic waste).
  • Forced-declaration (budget-exhaustion) behavior increases in buried condition for some models (e.g., Anthropic forced +54.9 pp).
  • Declared-hop depth: 1–2 hop items remain near ceiling; multi-hop (d4/d5) accuracy degrades, and degrades further under buried evidence.
• Confidence does not reliably distinguish right vs wrong; committed/“confident-wrong” errors occur and can remain underreported by benchmark-level calibration.
• Receipts: the audit requires claim-level receipts (file-level provenance for each claim) rather than only answer-level scores or stated confidence. This enables a signer to check the chain before taking responsibility.
• Instrument limitations and grader sensitivity:
  • Scores are produced by a frozen containment grader (not exact-match); the author audited the grader and found some conclusions sensitive to grader choices. The core buried-effect on accuracy, tool calls, and cost per correct answer is presented as robust; the paper is explicit where grader-sensitivity matters.
• Scope: focused on synthetic financial due diligence scenarios (room02); multimodal shallow quiz shows high 1-hop image-reading accuracy for many models but is a separate, shallower instrument. The paper does not test remediation strategies (e.g., explicit evidence-linked training) — it establishes a baseline and auditing requirements.

Data & Methods

• Corpus and items:
  • room02: second-generation synthetic finance-and-insurance data room authored for the project (no proprietary data), with known provenance for every fact.
  • 41 items, declared hop counts distributed as d2=8, d3=9, d4=17, d5=7; each item run with 3 seeds.
• Panel:
  • One flagship run per lab: GPT-5.6-sol (OpenAI), Fable 5 (Anthropic), GLM-5.3 (Zhipu), Qwen3.8-Max1 (Alibaba), Gemini-3.1-Pro (Google), Muse-Spark 1.2 (Meta). (Open-weight frontier releases excluded by design.)
• Tools and agent setup:
  • Models run as agents with shell-style tools (bash, python, read_file, grep, find). Retrieval is done by the agent through tool calls, not a fixed retriever.
  • Forced-declaration JSON output required: answer, confidence (0–100), list of evidence files (receipts).
• Metrics:
  • Accuracy (frozen containment-based grader), committed-error rate (confident wrong), forced-declaration rate, tool calls per item, API spend, and cost per correct / wrong answer.
• Main quantitative results (selected):
  • Table 2 aggregated accuracies (clean → buried): panel examples — GPT-5.6-sol 82.9% → 78.9%; Fable 5 89.4% → 77.9%; GLM-5.3 89.4% → 80.5%; Qwen3.8-Max 87.0% → 74.0%; Gemini-3.1-Pro 85.2% → 67.5%; Muse-Spark 82.9% → 74.0%.
  • Tool calls (mean per item) rose across arms; Table 3 shows increases from ~5.8→18.4 (OpenAI) up to ~9.2→25.7 (Zhipu).
  • Cost per correct answer and cost-per-wrong both increase; being wrong is not cheap.
• Multimodal shallow quiz:
  • Separate image-based 1–2 hop reading quiz: most image-capable models score 100% at one hop, slight drop at two hops; shows reading per se can be high while buried multi-hop QA still degrades.
• Auditing methodology:
  • Claim-level receipts recorded so a third party with the frozen corpus can verify which document a claim relied on.
  • The grader is frozen and its limitations are audited; the paper flags which findings are robust to grader-sensitivity and which are not.

Implications for AI Economics

• Cost accounting for agentic workflows must include retrieval/search costs, verification, and human-in-the-loop auditing, not just per-token model costs.
  • Realized cost per correct answer can be multiple times higher under realistic (buried-evidence) conditions. Procurement and ROI models that use clean-benchmark numbers will understate operating costs.
• Product-market signaling and procurement:
  • Leaderboard-style comparisons on clean, curated contexts compress model differences and can mislead buyers about real-world performance and costs. Buyers should demand multi-hop, buried-evidence evaluations and cost-per-correct metrics.
• Risk and liability economics:
  • Because agents can pair accurate numeric outputs with fabricated or unsupported explanations, institutional liability remains with human signers. Organizations must budget for human verification and legal risk mitigation (audit trails, liability transfer instruments, insurance).
• Pricing & contracts:
  • Vendors and integrators should price solutions to include evidence-tracing infrastructure, human-review workflows, and auditing tools. Flat per-token pricing hides variability in downstream verification costs.
• Investment and R&D priorities:
  • There is economic value in research that reduces evidence-location and verification costs (e.g., architectures that return explicit source-linked receipts or training on audited task trajectories). If such methods yield better accuracy/cost tradeoffs under matched budgets, they change the marginal value of compute vs verification labor.
• Evaluation & regulation:
  • Regulators and standards bodies aiming to ensure safe deployment in regulated domains should require claim-level provenance, condition-aware testing (buried vs clean), and human-adversarial verification as part of compliance regimes.
• Open questions with direct economic impact:
  • Will explicit, source-linked evidence structures and training on audited trajectories materially improve accuracy and reduce verification cost? If so, the market will value models and stacks that provide verifiable receipts; if not, the economic model will emphasize human verification capacity.
  • How do open-weight models compare under buried-evidence, cost-aware, receipt-based evaluation? (Open-weight frontier models were excluded here; their inclusion could change supplier/price dynamics.)

Practical takeaways for economists and decision-makers: - Don’t rely on clean-benchmark accuracy to price agentic systems for real work; model selection and procurement must be conditioned on buried-evidence, multi-hop evaluations and measured verification costs. - Require claim-level receipts and budget explicit human verification when agents are used in liability-bearing roles. - When modeling the economics of adoption, include the expected multiplier on verification and search costs implied by buried-evidence: empirical multipliers in this study ranged roughly 1.6×–7× on $/correct across models.