Task-level benchmarking shows LLMs reliably handle screening and metadata extraction for multi-source AI-and-work evidence but falter on interpretive analysis and cross-source synthesis; a human‑supervised routing framework preserves speed without sacrificing expert judgment.

Main Finding

The authors introduce the Knowledge Synthesis Review (KSR), a human-in-the-loop, task-decomposed framework that benchmarks LLM-based systems on four core evidence‑synthesis tasks (screening, extraction, analysis, synthesis) and routes each task to the best-performing model under continuous expert validation. Across a 1,893‑document corpus (244‑document benchmark subset) and four LLM systems (GPT‑5, Claude Sonnet 4, Gemini 2.5 Pro, NotebookLM), no single model dominated all tasks. LLMs performed well on structured tasks (screening, metadata extraction) but performance fell substantially on interpretive analysis and cross‑source synthesis, where expert judgment remained essential. Applying KSR end‑to‑end revealed cross‑source blind spots in the AI-and-work evidence base (e.g., worker well‑being, small firms, Global South).

Key Points

• KSR decomposes evidence synthesis into four tasks: screening, extraction, analysis, synthesis, and benchmarks models on each task against expert gold standards.
• Corpus and benchmark:
  • Full corpus: 1,893 documents (324 research papers, 30 industry reports, 30 policy briefs, 1,509 news/media/blog).
  • Benchmark subset: 244 documents (103 RPs, 30 IRs, 30 PBs, 81 NMBs).
• Human evaluation:
  • Two field experts independently labeled screening decisions; inter-rater reliability = 92.2% agreement, Cohen’s kappa = 0.804.
  • After reconciliation and deduplication, 183 documents retained for extraction/analysis/synthesis (RP=79, IR=21, PB=21, NMB=62).
• Models evaluated: GPT-5, Claude Sonnet 4, Gemini 2.5 Pro, NotebookLM.
• Task-level performance highlights:
  • Screening: Claude Sonnet 4 highest overall screening accuracy (82.8%); GPT‑5 achieved highest recall (91.8%) but with lower specificity (more false positives).
  • Extraction: >90% agreement for titles and source fields; substantially lower accuracy for author and reference fields.
  • Analysis & synthesis: largest performance drop — models struggled with interpretive judgments, cross-source integration, identifying methodological limitations and inferential caveats.
• Operational points:
  • Models often could not follow external links consistently (GPT‑5 with web search was an exception).
  • Prompts, input formatting (JSON), and standardized outputs (XLSX/JSON) were iteratively refined to improve consistency.
  • A contamination check on post‑cutoff documents found no evidence that prior model exposure inflated results.
• Applying routed, task‑specific model usage at scale surfaced asymmetries and blind spots in the evidence base that single‑source syntheses would miss (e.g., sparse attention to Global South, small firms, worker well‑being).

Data & Methods

• Three‑phase KSR implementation:
• Phase I — Corpus construction: curated multi‑source corpus (2020–2025), preserving real-world source imbalance.
• Phase II — Benchmarking & protocol development: 244‑document benchmark, task decomposition, standardized prompts, structured outputs, human‑in‑the‑loop evaluation.
• Phase III — Model‑agnostic orchestration: route each task to the empirically best model while maintaining expert validation.
• Task definitions and protocols:
  • Screening: binary include/exclude using title/abstract/keywords or executive summaries; inputs provided in JSON; outputs standardized. Screening criteria: substantive link between AI and labor market outcomes (automation, employment, wages, skills, productivity, inequality, policy, etc.).
  • Extraction: convert un/semi‑structured sources to metadata records; tailored schema per source type (titles, authors, publisher, year, counts of tables/refs, links).
  • Analysis: source‑level interpretive summaries (e.g., methodology, limitations, recommendations) returned as structured JSON.
  • Synthesis: cross‑source integration addressing six dimensions (labor‑market impacts, stakeholder narratives, cross‑source integration, temporal patterns, governance/ethics, challenges + blind spots).
• Evaluation:
  • Two expert reviewers assessed model outputs (anonymized, randomized) against predefined rubrics: accuracy, completeness, clarity, relevance, source fidelity, reasoning quality, usefulness; quantitative ratings + qualitative comments; disagreements reconciled by consensus.
  • Extraction fields graded as exact/partial/absent matches; analysis/synthesis scored with rubrics.
• Model access & constraints:
  • Systems queried via API/web UIs; external link reading inconsistent; earlier prompt versions revealed failure modes (title‑only judgments, formatting inconsistencies) prompting refinement.

Implications for AI Economics

• Practical workflow for multi‑source evidence synthesis:
  • KSR provides an auditable, model‑agnostic blueprint that decomposes review work into tasks where LLMs can be selectively applied and monitored. For economists synthesizing heterogeneous evidence (academic, industry, policy, media), this reduces risk of error propagation by embedding human checkpoints early (screening/extraction).
• Speed vs. reliability tradeoffs:
  • LLMs can accelerate structured parts of literature review (screening, extracting titles/sources) and improve recall (e.g., GPT‑5), but they introduce false positives and struggle with nuanced interpretive judgments. Economic synthesis requiring causal inference, methodological critique, and nuanced policy implications currently requires sustained expert oversight.
• Evidence coverage and policy relevance:
  • Routed multi‑source synthesis revealed thematic blind spots (worker well‑being, small firms, Global South) that are relevant for policy design and distributional analysis. Relying on single source types (e.g., industry reports or media) risks skewed public/policy narratives; economists should prioritize multi‑source aggregation to surface these asymmetries.
• Best practices to adopt in research and policy workflows:
  • Use task decomposition and model‑specific routing (assign models to tasks they empirically excel at).
  • Standardize inputs (JSON), outputs (structured records), and evaluation rubrics for auditability and reproducibility.
  • Maintain human‑in‑the‑loop checkpoints, especially for analysis and synthesis stages where inferential judgments matter.
  • Continuously re‑benchmark as model capabilities evolve; contamination checks and provenance tracing are essential for credibility.
• Caution for automated meta‑analysis and forecasting:
  • Given models’ weakness on interpretive integration, automated meta‑analytic claims or policy prescriptions derived without expert curation risk overconfidence and misrepresentation. For policy and economic forecasting (e.g., labor displacement estimates), preserve human accountability and transparent documentation of how model outputs were routed and validated.

Short recommendation for AI economists: adopt KSR’s task decomposition, use LLMs selectively for high‑recall screening and structured extraction, but retain experts for interpretive synthesis and causal or policy conclusions; regularly re‑benchmark models and report routing provenance to ensure transparency and reliability.