Large language models produce one-page research plans that expert reviewers judge comparable to human-authored drafts, but state-of-the-art AI reviewers both perfectly detect authorship and assign substantially higher quality scores to AI-written proposals, flagging risks for AI-assisted proposal preparation and review processes.

Main Finding

Contemporary LLMs (mid‑2025 generators; mid‑2026 frontier evaluators) can produce one‑page scientific project plans that human expert reviewers judge about as good as human‑written plans. However, state‑of‑the‑art LLM evaluators systematically favor AI‑generated proposals (≈+1 point on a 1–5 scale), and the frontier LLMs can perfectly identify authorship in this corpus, while human reviewers identify authorship only ~72–79% of the time. This combination implies both opportunity (AI as a capable assistance tool for planning) and risk (systematic bias in AI-based evaluation that could distort funding allocation).

Key Points

• Corpus and experiment
  • 8 expert‑conceived research projects across physics, astrophysics, cosmology.
  • For each project: 1 human‑written proposal + 3 AI‑generated proposals = 32 one‑page proposals total.
  • AI generators (mid‑2025): ChatGPT, Claude, DeepSeek. AI evaluators (mid‑2026): Claude Opus 4.8 and ChatGPT Pro 5.5.
  • Four human reviewers (senior academics) and two AI reviewers assessed each anonymized proposal.
• Evaluation procedure
  • All proposals followed identical 1‑page template: Title, Background (1 sentence), Goal (1 sentence), Methodology (≤5 steps, timeline); reviewers judged origin (human/AI) and scored 4 rubric aspects (1–5): Clarity/Structure; Appropriateness of Methods; Resource/Tool Planning; Feasibility/Timeline/Risk.
  • Human proposals were lightly copyedited to remove superficial cues.
• Authorship classification
  • Human reviewers: correctly identified AI‑written proposals 79% and human‑written proposals 72% of the time (varied across reviewers).
  • Frontier AI reviewers (Claude Opus 4.8, ChatGPT Pro 5.5): 100% correct on all 32 proposals in this study.
  • Smaller AI (Claude Sonnet 4.6) performed worse (67% on AI proposals).
  • Cues used by humans: presence of citations, idiosyncratic jargon, first‑person use → human; overly clean template structure, generic tool lists, round timelines, buzzwords → AI. AI evaluators used similar cues but applied them consistently.
• Quality scoring and bias
  • Human reviewers: rated AI‑ and human‑written proposals similarly (mean ≈3.5/5 across authors).
  • AI reviewers: systematically scored AI‑written proposals ~1 point higher than human ones (AI mean ≈4.2–4.5 vs human ≈3.5).
  • The pro‑AI bias was strongest on Clarity/Structure and Resource/Tool Planning; smaller but present for Appropriateness of Methods.
• Robustness notes
  • Bias persisted across several LLM evaluators and evaluation orderings; not an artifact of telling reviewers the class balance.
  • Proposals were intentionally short (one page), which affects generalizability to full grant applications.

Data & Methods

• Design
  • Controlled, blinded study: same expert seed (title/background/goal) given to human planner and AI prompter; all outputs standardized and anonymized.
  • Human planners were domain experts (grad students/postdocs); AI prompters were non‑specialist students who used a fixed prompt template to generate outputs from three assistants.
  • Human reviewers were four coauthor experts (faculty/senior postdocs).
• Models and timing
  • AI generation: ChatGPT, Claude (Sonnet 4), DeepSeek (outputs from June–Sept 2025).
  • AI evaluation: Claude Opus 4.8 and ChatGPT Pro 5.5 (mid‑2026).
• Rubric
  • Four aspects (1–5 each): Clarity/Structure; Appropriateness to Goal; Resource/Tool Planning; Feasibility/Timeline/Risk.
• Analysis
  • Reported mean scores and standard deviations across the eight projects; origin classification accuracy; reviewer qualitative justifications were analyzed for cues.

Implications for AI Economics

• Market and supply effects
  • Lower cost and faster generation of credible project plans could expand supply of proposals and ideas, reducing marginal cost of proposal preparation and increasing competition for limited funding.
  • Greater volume and higher baseline quality of submissions may pressure funders to raise thresholds or change selection criteria, potentially disadvantaging those without access to high‑quality AI tools.
• Signaling, information asymmetries, and quality inference
  • If AI evaluators are used in screening or triage, their pro‑AI bias could create a positive feedback loop: AI‑crafted proposals get preferential scores → applicants adopt AI tools to compete → AI style becomes the dominant signal irrespective of true scientific merit.
  • Human reviewers’ imperfect ability to detect AI authorship creates asymmetric information: applicants could hide AI use or mimic human idiosyncrasies; AI reviewers can exploit style cues, but that may not correlate with scientific quality.
• Incentives and strategic behavior
  • Applicants and institutions may invest in AI tooling and training (an “arms race”) to gain scoring advantage rather than to improve substantive science.
  • Strategic manipulation: proposals could be engineered to maximize features favored by automated scorers (template clarity, exhaustive tool lists), possibly at the expense of originality, novelty, or realistic risk assessment.
• Labor and organizational impacts
  • AI tools could substitute for parts of the proposal‑writing labor market (grant writers, proposal consultants), shifting labor to higher‑value interpretation and project execution tasks.
  • Funding agencies might reduce human reviewer workloads by delegating triage to LLMs, but this risks systematic biases and may distort portfolio outcomes.
• Policy and evaluation system design
  • Reliance on AI evaluators without calibration risks changing allocation of public research funds in non‑meritocratic directions.
  • Measures to mitigate risk: mandatory disclosure of AI assistance; mixed human+AI panels; randomized audits comparing AI and human scores; recalibration of automatic scorers against human‑value objectives; redesign rubrics to reward originality, uncertainty, and domain‑specific specificity rather than boilerplate polish.
• Resource allocation and public investment
  • Funders may need to invest in infrastructure to ensure equitable access to high‑quality AI tools (to avoid concentration of advantage) and to support research on robust, bias‑aware evaluation algorithms.
  • New evaluation norms (e.g., requiring detailed reproducibility checks, provenance for data and codes) could shift resource needs toward verification and monitoring.

Recommendations (practical, short)

• For funders and grant panels:
  • Do not substitute AI evaluators for humans without calibration and transparency; run pilots and randomized controlled comparisons before operational use.
  • Require disclosure of AI assistance in proposal drafting; track correlations between disclosed AI use and funding outcomes.
  • Use mixed evaluation workflows (human + multiple independent AI scorers) and test for systematic biases.
  • Update rubrics to reward domain‑specific originality and uncertainty management, not only polished structure.
• For institutions and applicants:
  • Treat AI as an assistance tool, not a substitute for substantive experiment/theory work; document AI contributions.
  • Be aware of potential strategic incentives to "optimize for the scorer" and avoid gaming that undermines scientific quality.
• For researchers and policy makers:
  • Fund studies that (a) replicate and extend this work to full proposals and larger samples, (b) measure long‑run effects on funding distributions, and (c) develop bias‑aware evaluators and detection/provenance tools.

Limitations & Open Questions

• Short, one‑page proposals: results may differ for full grant proposals with technical appendices and data/figures.
• Small sample (n=8 projects, 32 proposals) and a limited set of models—caution in generalizing.
• Perfect AI authorship classification here may not hold at scale or with different sanitization; adversarial prompt engineering could defeat classifiers.
• Long‑term effects on research agendas, novelty, and scientific progress remain to be empirically evaluated.

If you want, I can (a) produce a concise one‑page brief for funders summarizing policy actions, or (b) draft a short experimental design to test interventions (e.g., mixed panels, disclosure rules) to measure and mitigate the pro‑AI bias.