The spread of large language models is changing how grant ideas are framed and selected: proposers who use LLMs produce less semantically distinctive submissions and—at NIH but not NSF—are more likely to win funding and generate early, modest-impact papers.

Main Finding

The rapid adoption of large language models (LLMs) in federal grant writing since 2023 is detectable and uneven (bimodal). Higher LLM involvement is consistently associated with proposals and awards that are semantically closer to recently funded work (lower distinctiveness). Consequences differ by agency: at NIH higher LLM involvement is associated with greater proposal success (~+4 percentage points) and ~+5% more early-stage publications (driven by non-hit papers), whereas at NSF no systematic association with funding success or publication output is observed.

Key Points

• Temporal pattern
  • Detectable LLM use rose sharply beginning in 2023 (coinciding with ChatGPT availability).
  • Individual-grant LLM involvement (α) shows a bimodal distribution in 2023–2025: one mode near zero, another around ~10–15% α.
• Positioning of ideas
  • Greater LLM involvement is associated with lower semantic distinctiveness (proposals/awards move closer to the agency’s recent funded portfolio).
  • Magnitude: moving an award from the 25th to the 75th percentile of α corresponds to ~5 percentage-point decrease in distinctiveness for NSF and ~4 points for NIH (within-year percentile rank).
  • Rewriting 2021 abstracts with an LLM did not materially change distinctiveness measures, suggesting substantive convergence—not only surface-level stylistic smoothing.
• Selection and outputs (agency-dependent)
  • NSF: no statistically significant relationship between α and proposal success or publication counts.
  • NIH: higher α is associated with higher funding probability (~+4 percentage points from 25th→75th α) and ~+5% more early publications for grants (2023–2024 cohorts).
  • The NIH publication gain is concentrated in non-hit papers; associations with top 5% (or top 1%) high-impact papers are not significant.
• Robustness
  • Results robust to controls (grant-year, field, investigator fixed effects; funding amount).
  • LLM-associated proposals score as more readable (Flesch reading ease), and removing promotional words does not change the LLM-use signal.

Data & Methods

• Data
  • Private submissions: complete NSF and NIH proposal abstracts (funded, unfunded, pending) from two large U.S. R1 universities (request start dates 2021–2025).
  • Public awards: full population of publicly released NSF and NIH award abstracts (same period) plus linked publications for awards.
• Measuring LLM involvement (α)
  • Based on Liang et al. detection approach: estimate word distributions for human-written text (public 2021 abstracts) and for LLM-generated text (GPT-3.5-turbo-0125 rewriting of the same abstracts).
  • Applied detection at corpus level (3-month rolling windows) and per-grant level to get α (fraction of LLM-modified sentences).
• Semantic distinctiveness
  • Embed abstracts using SPECTER2 embeddings and compute average cosine distance to all agency-funded abstracts in the prior year.
  • Convert distances to within-year percentile ranks (higher = more distinctive).
• Statistical analysis
  • Regressions of distinctiveness, funding success, and publication outcomes on grant-level α.
  • All regressions include start-year, field, and investigator (PI/co-PI) fixed effects and controls for requested/awarded funding amount.
  • Analyses exploit within-investigator variation in α (estimates are conditional correlations, not causal effects).
• Robustness checks
  • Rewrote 2021 abstracts with LLMs to test whether surface-level changes explain distinctiveness shifts.
  • Evaluated reading-ease (Flesch) and removed promotional language; results held.

Implications for AI Economics

• Upstream effects of LLMs matter: LLMs are already reshaping the funding pipeline (positioning and selection), not just downstream manuscript production.
• Tradeoff: individual-level gains vs. system-level diversity
  • LLMs can raise submission quality/readability and may increase funded outputs (NIH), but they also appear to pull proposals toward established funded norms, reducing semantic novelty. This suggests a shift from exploration toward exploitation in public research portfolios.
• Agency heterogeneity matters for outcomes and policy
  • Differences between NIH and NSF outcomes indicate that institutional review processes, norms, and evaluation criteria interact with LLM effects. Policy responses likely need to be agency-specific.
• Implications for portfolio governance and evaluation
  • Funders should monitor LLM use and its impact on portfolio diversity and long-run innovation. Consider requiring disclosure of generative-AI use, explicitly evaluating novelty/distinctiveness in review, and designing mechanisms to preserve exploratory projects.
• Caution on long-run impact
  • Early publication gains at NIH are concentrated in lower-impact papers; there is no current evidence that LLM use raises the probability of producing highly cited, transformative work. Long-term effects on discovery and scientific impact remain uncertain.
• Research and policy priorities
  • Need causal studies to separate writing-quality vs. substantive-idea mechanisms.
  • Extend analyses to more universities, additional agencies, longer follow-up for citation outcomes, and evolving LLM architectures and detection methods.
  • Consider experiments or pilot policies (disclosure, reviewer training, modified evaluation rubrics) to manage potential homogenization while preserving productivity benefits.

Limitations to bear in mind: LLM-detection is imperfect and model-dependent; private-proposal data come from two institutions (generalizability limit); observational design cannot establish causality; publication windows are short (1–2 years) for long-run impact assessment.