A Gemini-based AI substantially speeds complex scientific synthesis: 13 climate scientists produced a 79-paper synthesis in roughly 46 person-hours with most AI-generated content retained; yet expert oversight remained essential, and humans supplied much of the final rigor and content.

Main Finding

An integrated Gemini-based AI Assistant substantially accelerated a verification-poor scientific assessment (AMOC stability) while leaving human expertise central to rigor. In a 5-week, 13-expert case study the hybrid workflow produced an 8,000-word synthesis of 79 papers through 104 revision cycles in ~46.5 logged person-hours (≈3.5 h/author). Participants estimated the same task typically takes 100–200 person-hours; all reported at least 2–3× speedups (38% estimated ≥7×). Most AI-proposed content was retained, but scientists contributed the majority (58.3%) of final content and were essential to elevate the draft to rigorous assessment. The study introduces simple, sentence-level metrics (Retention and Intervention) and embedding-based convergence measures to quantify human–AI co-authoring dynamics.

Key Points

• Task and setting

  • Topic: Atlantic Meridional Overturning Circulation (AMOC) stability — deliberately verification-poor and contested.
  • Participants: 13 climate scientists with diverse subfield backgrounds and institutional-report experience.
  • Corpus: curated AMOC collection (~1,660 papers) plus search via OpenAlex and web.
  • Tooling: Gemini LLMs (Gemini 2.5 Pro then Gemini 3 Pro Preview) via Vertex AI embedded into an editor + evidence panel + chatbox workflow.

• Workflow and outcomes

  • Three phases: outline → section drafting/curation → final collation and sign-off.
  • Output: ~8,000-word report synthesizing 79 papers; 104 version iterations; 46:33 total logged hours.
  • Efficiency: Participants judged substantial speedups (≥2–3× common; some much larger).

• Quantitative contributions

  • Human vs AI content: scientists authored ~58.3% of final content; AI contributed substantially and much AI text was retained.
  • Revision dynamics: AI-generated revisions were used ~3× more often than manual revisions; AI credited for ~25% of revisions content.
  • Convergence: embedding-based similarity between intermediate versions and final version showed monotonic convergence with identifiable drafting → refinement → finalization stages.

• Evaluation metrics & methods

  • Sentence-level alignment used to define:
    • Retention (precision-like): fraction of AI-proposed sentences/themes present in final text.
    • Intervention (recall-oriented / 1 − recall): fraction of final content that is new human-authored material.
  • Document convergence measured via embeddings (gemini-embedding-001) and bounded inverse Euclidean distance.

• Limitations & failure modes

  • LLM issues: hallucinations (e.g., citation numbering), difficulty synthesizing quantitative material, sycophancy/obsequiousness under adversarial prompting, and limited holistic reasoning (identifying caveats).
  • Study limits: single case study, proprietary tool, model switch during experiment (Gemini 2.5 → 3 Pro) — potential confound for pure model-performance attribution.

Data & Methods

• Data

  • Precompiled AMOC corpus (~1,660 papers) converted to Markdown; assistant had access to this corpus and to external search/OpenAlex metadata.
  • Evidence panel entries included auto-generated summary snippets, bibliographic metadata, and retrieval scores (BM25 + dense retrieval).
  • Final report references: 79 papers synthesized, final reference list consolidated in Phase 3.

• Models & system

  • LLMs: Gemini 2.5 Pro (Phase 1) and Gemini 3 Pro Preview (Phases 2–3).
  • Embeddings: gemini-embedding-001 for semantic similarity/convergence measures.
  • Retrieval: BM25 and dense retrieval (Karpukhin-style) over document chunks; evidence selection iterated during drafting.

• Workflow mechanics

  • Editor supported manual and AI edits; AI edits could be triggered at document- or text-span-level.
  • Assistant functions: draft generation, source selection/summarization, argumentative/rhetorical review (Toulmin + discourse frameworks), conflict resolution suggestions, version-history summaries.
  • Human curation: experts edited, expanded, checked numeric and conceptual claims, and performed final sign-off.

• Evaluation & metrics

  • Time: logged-in elapsed time with a timeout parameter; interpreted as approximate effort.
  • Convergence: sequence of document embeddings → bounded inverse Euclidean distance to final version; step-wise similarity between successive versions.
  • Sentence alignment: many-to-many matching between draft sentences (D) and final sentences (F) used to compute Retention = |D_M|/|D| and Intervention = 1 − |F_M|/|F| (where M is alignment set).

Implications for AI Economics

• Productivity shock in expert synthesis tasks

  • The case suggests a meaningful productivity gain in verification-poor, high-expertise tasks (reports, assessments, meta-analyses). If generalizable, lower effective costs and faster turnaround for consensus products (e.g., intergovernmental assessments, institutional white papers) are plausible.
  • Effect on labor demand: complementary shift — less time on first-pass drafting, more demand for high-skill curatorial, evaluative, and adjudicative labor (expert oversight, cross-validation, methodological critique). Routine drafting tasks could be partially automated, reducing low-level drafting hours but raising value of verification/credibility roles.

• Valuation and attribution

  • The paper’s sentence-level Retention and Intervention metrics provide a practical framework to apportion credit/value between human and AI contributions. These can inform payment, authorship conventions, and productivity accounting in institutions and funding agencies.
  • Firm-level advantages: organizations with privileged access to high-performing LLMs and curated corpora may capture disproportionate productivity gains, increasing concentration in research-production capacity.

• Institutional and market effects

  • Faster, cheaper assessments could increase the supply of syntheses and policy-relevant reports, altering competition among consultancies, NGOs, and academic groups. This may compress timelines for policy cycles and reshape consultancy pricing.
  • Reputation and trust capital become more valuable: because AI can produce fluent but potentially flawed drafts, demand will grow for high-reliability human validators and for institutional certifications that a synthesis was rigorously vetted.

• Risks, externalities and regulation

  • Quality and signaling: greater throughput risks dilution of scrutiny if oversight is weak; markets may discount outputs unless provenance, traceability, and human-signoff conventions are standardized.
  • Bias and informational cascades: AI-assisted convergence toward consensus could amplify stylistic or evidentiary homogeneity (echo chambers). Economic models of information aggregation should account for reduced diversity of initial drafts when many actors use similar LLMs.
  • Access inequality: proprietary models and curated datasets create barriers; unequal access can generate asymmetric productivity gains across institutions and countries.

• Metrics for economic assessment and policy

  • Use embedding-based convergence and sentence-level attribution to:
    • Measure productivity gains attributable to AI at project and sector levels.
    • Inform compensation models that allocate value between human expertise and AI tooling.
    • Monitor quality externalities (e.g., rate of hallucination corrections per unit time) as a regulatory metric.

• Open questions for economic research

  • Generalizability: How do observed gains scale across domains and larger teams? Are speedups similar in other verification-poor fields (macroeconomics, epidemiology, legal synthesis)?
  • Labor reallocation dynamics: What is the long-run impact on demand for mid-skill vs high-skill scientific labor? Will labor markets bifurcate into AI-supervisory roles vs model operators?
  • Market structure: How will concentration of access to high-quality LLMs affect competition among research producers and the pricing of assessment services?

Summary conclusion The study demonstrates a practical hybrid model where LLMs materially increase efficiency in producing high-level, verification-poor scientific assessments, while retaining the necessity of human expertise for rigor. For AI economics, this implies a productivity shock concentrated in drafting-intensive expert goods, leading to complementarities that raise the value of human verification and institutional trust mechanisms, create new metrics for attributing AI value, and produce distributional and regulatory challenges that merit further empirical and theoretical study.