Frontier LLMs seldom recover a paper's idea from pre-publication references on their own—single-model match rates are only ~3–15%—but a reference-only multi-agent review with Swiss selection boosts recovery to ~23–42%, about 2.4× the best single-model baseline.

Reconstruction: A Blind Benchmark for Recovering Research Ideas from Pre-Publication Bibliographies

Shaolong Chen, Yanlin Fei, Nazhou Liu, Xinmiao Yu, Lei Li, Rahul Thapa, Madalina Ciobanu, Qingqing Mao, Ritankar Das · August 17, 2026

arxiv descriptive medium evidence 7/10 relevance Full text usable extracted full text Source PDF

Structured author observations

Linked only from stored provider relations; the raw author line above is never matched by name.

Arxiv

Latest observation: August 18, 2026

Shaolong Chen unresolved corpus identity
Yanlin Fei unresolved corpus identity
Nazhou Liu unresolved corpus identity
Xinmiao Yu unresolved corpus identity
Lei Li unresolved corpus identity
Rahul Thapa unresolved corpus identity
Madalina Ciobanu unresolved corpus identity
Qingqing Mao unresolved corpus identity
Ritankar Das unresolved corpus identity

Semantic Scholar

Latest observation: August 24, 2026

Shaolong Chen provider ID
Yanlin Fei provider ID
N. Liu provider ID
Xinmiao Yu provider ID
Lei Li provider ID
Rahul Thapa provider ID
M. Ciobanu provider ID
Q. Mao provider ID
R. Das provider ID

Reconstruction is a blind benchmark that tests whether LLMs can recover a published paper's research idea from only its pre-publication bibliography, finding single-model match rates of roughly 3–15% while a reference-only multi-agent pipeline raises matches to about 23–42%.

Citation observations

Cumulative provider counts captured on specific dates; providers are never combined.

0 cumulative citations

Semantic Scholar · Observed August 24, 2026

View corpus context

Can a language model recover the true research idea of a published paper when given only that paper's pre-publication bibliography? We introduce Reconstruction, a blind idea-recovery benchmark that withholds the seed paper and all contemporaneous or future literature, and asks models to propose hypotheses that an independent large language model judge matches against the held-out ground-truth idea. A strict anti-leakage protocol-temporal citation cutoff, anonymous reference IDs, and frozen per-paper bibliographies, which prevents prompt-time leakage of the seed idea. Across six scientific domains and 643 evaluated papers, seven frontier models achieve only modest Match rates (approx. 3-15%). We then evaluate a reference-only multi-agent (top 4) pipeline that combines cross-model review with a Swiss tournament over aligned hypothesis slots, without external web search. Cross-model review plus tournament selection raises Match rates to approx. 23-42% across all six domains, which is an observed approx. 2.4x lift over the best single-model baseline. This draft reports the protocol, anti-leakage design, and current results as an arXiv timestamp.

Summary

Main Finding

Reconstruction introduces a blind, anti‑leakage benchmark that tests whether LLMs can recover a published paper’s actual research idea using only its pre‑publication bibliography. Single‑model proposers achieve modest Match rates (~3–15%; best six‑domain average 13.3% for Claude‑Opus‑4.8). A reference‑only multi‑agent pipeline (top 4 models, cross‑model reference review + Swiss tournament selection) substantially improves recovery to per‑domain Match rates of ~23–42% and a six‑domain average of 36.0% ± 6.1% — an observed ≈2.4× lift vs. the best single‑model baseline (measured as the ratio to the best top‑4 single model per domain).

Key Points

Task (Reconstruction): Given only the seed paper’s bibliography resolved to title+abstract (only works published strictly before the seed’s publication date T0), models propose ns=5 hypotheses (each must cite anonymous reference IDs). An independent LLM judge (different from proposers) compares each hypothesis to the held‑out seed title+abstract and returns a binary Match label.
Anti‑leakage protocol: temporal cutoff (published < T0), anonymous reference IDs, frozen per‑paper bibliographies, evidence binding (each hypothesis must cite refs), and judge recusal to avoid self‑evaluation.
Dataset and scope: 643 evaluated seed papers across six domains (ML—ICML 2026 orals; Nature Astronomy, Chemistry, Materials, Medicine, Physics), collected and resolved via Semantic Scholar / Crossref / OpenAlex / arXiv.
Models evaluated (OpenRouter snapshots, July 2026): Claude‑Opus‑4.8, GPT‑5.6‑Sol‑Pro, Kimi‑K3, GLM‑5.2, Gemini 3.1 Pro Preview, DeepSeek‑V4‑Pro, Qwen3.7‑Max. Judges are LLMs (leave‑one‑out panels; multi‑agent judges recuse on their own origins).
Protocols:
- Default (single‑model): each model produces 5 hypotheses from frozen blind refs; scored by other models as judges.
- Multi‑agent (top 4): top 4 models each produce 5 hypotheses; slot alignment (kth hypothesis from each model → four candidates per slot); reference‑only cross‑model review (no web search); Swiss tournament per slot (recusal & debiasing); five slot champions become the multi‑agent output.
Main quantitative outcomes:
- Single‑model cell means: ~3.4%–15.0% by domain; best average 13.3% ± 2.3% (Claude‑Opus‑4.8).
- Multi‑agent Match rates: ML 22.9%, Astronomy 36.5%, Chemistry 38.4%, Materials 40.1%, Medicine 41.6%, Physics 36.4%; Avg 36.0% ± 6.1%.
- Observed lift vs best top‑4 single per domain: ≈2.3–2.6× (mean 2.4×); pooled bootstrap 95% CI for overall lift ≈ [2.3, 2.6].
- Success@5 (fraction of papers with at least one matching hypothesis): multi‑agent 57.1% overall vs best dagger single 55.1% (smaller gap than per‑hypothesis Match).
Additional observations:
- Multi‑agent hypotheses are longer than single‑model (mean ~114 words vs ~56 for Default), closer to seed length (~191), so length/presentation is a potential confound.
- Some cases dropped due to insufficient pre‑T0 references or generation failures; final reported set is n=643 papers with ns=5 champions.

Data & Methods

Seed sourcing: 879 titles from ICML 2026 orals (ML) and Nature family journals (Astronomy, Chemistry, Materials, Medicine, Physics). After eligibility (resolvable title/abstract, publication date, ≥3 pre‑T0 refs) and multi‑agent alignment filters, 643 seeds retained.
Reference resolution: Merge and deduplicate references from Semantic Scholar, Crossref, OpenAlex, and extracted PDFs when available; include only refs with published(r) < T0; assign anonymous IDs (ref‑001...).
Evaluation metric: Match(s; P, J) = fraction of ns=5 hypotheses matching the seed under judge J; domain and overall Match rates aggregate over seeds and judge panels. Multi‑agent aggregates handle judge recusal and weight judges by eligible coverage.
Infrastructure: LLM calls via OpenRouter (July 2026 snapshot); reference resolution with concurrency; frozen bibliographies ensure Default and multi‑agent see identical blind contexts.
Statistical checks: Paper‑level bootstrap (B=2000) for confidence intervals on Match and lift; paired per‑paper comparisons vs per‑paper max among top‑4 single models (multi‑agent favored on more papers; sign test p < 1e‑15).
Limitations called out by authors: multi‑agent is post‑selected on top‑4 roster (no held‑out validation for selection rule); some observed lift may come from inference‑time selection (20 candidates → choose 5) rather than model collaboration per se; judges are LLMs (not human experts); hypothesis length differences and presentation could confound results.

Implications for AI Economics

Valuation of LLM‑based research tools:
- Multi‑model ensembles and selection pipelines materially increase measured recovery of published ideas relative to single models (≈2.4× observed lift). For product managers and pricing strategists, this suggests ensembles and compositional selection logic can be a differentiator worth premium pricing.
- However, absolute Match rates remain limited (multi‑agent average ~36%), so LLMs are currently partial complements to human researchers rather than full substitutes for discovery tasks that require precise literature synthesis.
Productivity and labor economics:
- The success@5 (~57% multi‑agent) and per‑hypothesis lift indicate ensembles improve the chance of producing at least one useful lead per paper. This implies potential productivity gains for literature‑based ideation workflows, but the gains are heterogeneous across domains (Materials/Medicine higher).
- Firms estimating labor cost savings should model both (a) ensemble inference and API costs and (b) still‑required human validation time; naive extrapolation from relative lift to full automation would overestimate substitutability.
Market structure and competition:
- The top‑4 post‑selection approach reflects a competitive market for best‑in‑class models; purchasers may prefer bundled multi‑provider access or intermediary platforms that orchestrate cross‑model review and tournament selection.
- A market for “derivation services” that curate frozen bibliographies and run multi‑agent pipelines could arise (value: provable anti‑leakage contexts, reproducible ideation).
Cost vs. performance tradeoffs:
- Multi‑agent pipelines incur higher inference cost (4× generation + review + tournament) and likely higher latency. Economic decisions should weigh marginal utility of higher Match rate against API/compute costs and downstream human time saved.
- Some of the multi‑agent gain comes from sampling/selection (20→5). Economists and procurement teams should consider whether cheaper scale‑up of a single strong model (more sampling from one model) could capture part of the lift versus maintaining heterogeneous model access.
Incentives, IP, and disclosure:
- Anti‑leakage protocol (temporal cutoff, anonymous refs) highlights how evaluation design affects the attribution of idea originality. Platforms delivering “idea recovery” capabilities must consider IP, attribution, and researcher incentives if models increasingly surface near‑ground‑truth ideas from bibliographies.
- Policymakers and institutions should note that LLMs can reconstruct research directions from pre‑publication citations to a non‑trivial degree — this affects norms around preprints, embargoes, and commercial exploitation of emergent insights.
Measurement and externalities:
- The paper’s protocol provides a template for rigorous, temporally grounded benchmarking that avoids prompt‑time leakage; this is valuable for regulators and purchasers seeking reliable performance metrics.
- Remaining limitations (LLM judges, post‑selection, length confounds) mean estimates of economic value should be treated cautiously and stress-tested with human evaluation and cost modeling.
Research & product recommendations for economists and strategy teams:
- Run cost‑benefit analyses comparing: multi‑provider orchestration vs single‑provider scaled sampling; inference costs vs human validation savings; productized frozen‑bibliography services vs live retrieval with provenance.
- Consider piloting multi‑agent ideation for domains with higher observed multi‑agent success (Materials, Medicine) where ROI on R&D ideation assistance may be larger.
- Track downstream execution outcomes (not just Match) to estimate conversion rates from ideation to actionable experiments — crucial for monetization estimates.

Limitations and caution: the multi‑agent lift is an observed association in this experimental setup (top‑4 roster post‑selected on the same papers). Judges are LLMs (not human experts), and hypothesis length/presentation differ between Default and multi‑agent outputs — all factors that can affect interpretability and economic extrapolation.

Assessment

Paper Typedescriptive Evidence Strengthmedium — Large N (643 evaluated papers) across six scientific domains with a strict temporal anti-leakage protocol and bootstrap CIs provides substantive empirical evidence about model behavior on the benchmark; however results rely on LLM judges (not human adjudicators), post-selection of a top-4 roster for the multi-agent pipeline, and potential bibliographic resolution/selection biases that limit causal or fully general conclusions. Methods Rigormedium — The protocol is carefully designed (temporal cutoff, anonymous refs, frozen bibliographies, recusal rules), uses multiple models and bootstrap inference, and documents retention/filtering; nevertheless, important limitations remain (automated LLM judging rather than humans, post-hoc top-4 selection, some domain-specific early stopping and filtering, potential unresolved bibliographic errors, and unaddressed confounds like hypothesis length and candidate-as-selection-scaling). SampleSeed set of 643 published papers (ICML 2026 or Nature-family journals: ML, Astronomy, Chemistry, Materials, Medicine, Physics) retained after bibliographic resolution and eligibility filtering; for each seed the frozen blind context R<T0 contains resolved title/abstracts of references published strictly before the seed's publication date; seven frontier LLM snapshots were used as proposers and judges (OpenRouter snapshot, July 2026), with a post-selected top-4 used for a multi-agent pipeline evaluated on aligned seeds. Themeshuman_ai_collab productivity innovation GeneralizabilityDomains limited to ICML and five Nature-family journals—may not generalize to other fields, venues, or less curated literatures, Seeds exclude papers with poorly resolved bibliographies or undated references, biasing toward well-indexed work, Model-suite restricted to OpenRouter July 2026 snapshots; results may change with other model families or later versions, Final multi-agent pipeline is post-selected on the same papers (top-4 chosen after Default runs), limiting out-of-sample claims about ensemble selection rules, Evaluation uses automated LLM judges rather than human experts, which may not reflect human judgment of idea recovery, Hypothesis length/presentation differences and selection-scaling (5 of 5 vs best-of-20) may confound comparisons

Claims (7)

Claim	Direction	Outcome	Confidence & Evidence	Details
The Reconstruction benchmark evaluates whether language models can recover a published paper's research idea using only its pre-publication bibliography, while withholding the seed paper and contemporaneous or future literature. Research Productivity	positive	Research-idea Match rate	Reading fidelity high Study strength medium	n=643 0.18
Single-model systems recovered the held-out research idea at only modest rates, with observed Match rates ranging from 3.4% to 15.0% across model-domain cells. Research Productivity	negative	Per-hypothesis Match rate with the held-out seed-paper idea	Reading fidelity high Study strength medium	n=643 3.4%–15.0% 0.18
Claude-Opus-4.8 achieved the highest six-domain average single-model Match rate, at 13.3% ± 2.3%. Research Productivity	positive	Six-domain average Match rate	Reading fidelity high Study strength medium	n=643 13.3% ± 2.3% 0.18
The reference-only multi-agent pipeline achieved a 36.0% ± 6.1% average Match rate across the six domains, with domain-specific rates from 22.9% in ML to 41.6% in Medicine. Research Productivity	positive	Multi-agent Match rate for recovering the held-out research idea	Reading fidelity high Study strength medium	n=643 36.0% ± 6.1% average; 22.9%–41.6% by domain 0.18
Relative to the best top-four single-model baseline, the full multi-agent pipeline produced an observed average Match-rate lift of approximately 2.4× across the six domains. Research Productivity	positive	Relative Match-rate performance of multi-agent versus best single-model baseline	Reading fidelity high Study strength medium	n=643 2.4× lift; 95% CI [2.3, 2.6] 0.18
The multi-agent pipeline's advantage should not be interpreted as an isolated causal effect of collaboration, because it also selects the best five hypotheses from 20 candidates rather than retaining all five hypotheses from one model, and the conditions use different judge panels. Research Productivity	mixed	Interpretation of the multi-agent versus single-model Match-rate difference	Reading fidelity high Study strength high	n=643 0.3
Multi-agent success@5 was only slightly higher than the domain-best single-model baseline: 57.1% versus 55.1% overall. Research Productivity	positive	Fraction of papers with at least one matching hypothesis	Reading fidelity high Study strength medium	n=643 57.1% versus 55.1% 0.18