Automated detectors suggest LLM-generated code is common but falling and concentrated in tests and duplicated snippets, while LLM-like comments stay steady; company repositories show higher detected usage, and only a small fraction of human-labeled bugs are associated with flagged LLM code.

Main Finding

Using a detector-based proxy analysis on eight active repositories (2021–2025), the authors find that code flagged as likely LLM-generated has declined over time while comments flagged as likely LLM-generated remained stable. Much of the flagged code appears in test files and exhibits high rates of intra-repository file-level cloning; flagged comments have high punctuation rates but lower grammatical accuracy. Company-maintained repositories show higher percentages of flagged code/comments and more clones. Only a small share of human-labeled post-2021 bugs appear associated with LLM-flagged code (≈10.8% in NVD, 5.6% in OSS-Fuzz). All findings are presented as detector-output proxies, not ground-truth labels.

Key Points

• Scope and datasets
  • 8 repositories analyzed (Go-github, Guava, Liquid, Zap, Act, Jadx, Kafka, Pandas), snapshots Oct 2021–Oct 2025.
  • Bug analysis used PreciseBugs (human-labeled bugs introduced after 2021).
• Detection methods
  • Employed a suite of zero-shot detectors: Binoculars, Log-Likelihood, Entropy, Rank, Log-Rank, LRR, DetectGPT, Fast-DetectGPT, and DetectCodeGPT.
  • Comment-detection thresholds calibrated via the AISE dataset filtered to files last modified before 2021; code used DetectCodeGPT default threshold.
• Quantitative findings
  • Temporal trend: proportion of code flagged as likely LLM-generated decreased over 2021–2025; flagged comments relatively stable.
  • Location: flagged code concentrated in test cases; flagged comments mainly categorized as Explanation/Meta.
  • Cloning: majority of repositories have >70% file-level clones among flagged code; method-level clones generally low (max ≈34.2%). Compared with GPTCloneBench, many flagged code pieces had no external clones.
  • Comment quality signals: ~90% proper punctuation on average, but relatively low grammatical correctness and limited use of AI-specific vocabulary.
  • Company vs community: company-maintained repos show higher percentages of flagged code/comments and more clones.
  • Bugs: only a small fraction of human-labeled bugs in investigated datasets likely associated with LLM-flagged code (NVD: 10.79%; OSS-Fuzz: 5.56%).
• Limitations
  • No ground-truth labels — results are proxy-based, dependent on detector choices and thresholds.
  • Detector performance and thresholds may vary by domain (code vs comment) and by detector type; errors (false positives/negatives) possible.
  • Watermarking and supervised detection methods were not used.

Data & Methods

• Repositories: selected 8 active company- and community-maintained projects, covering Oct 2021–Oct 2025 (motivation: key LLM releases/open-source since 2021).
• Detection pipeline:
  • Applied multiple zero-shot detectors spanning three methodological classes:
    • Statistical-based (log-probability, rank, entropy variants, Fast-DetectGPT),
    • Perturbation-based (DetectGPT, DetectCodeGPT),
    • Model-comparison-based (Binoculars).
  • Calibration:
    • Comments: thresholds chosen via AISE dataset (filtered to pre-2021 file edits to improve the human-written baseline).
    • Code: used DetectCodeGPT default threshold (benchmarked on developer repos).
• Post-detection analyses:
  • Manual coding to categorize detected code/comments (e.g., test vs production files; Explanation/Meta comments).
  • Code-clone analysis: intra-repository, cross-repository, and comparison to GPTCloneBench.
  • Comment quality measures: punctuation rate, grammatical correctness, AI-vocabulary frequency.
  • Bug association: matched flagged code to PreciseBugs (NVD and OSS-Fuzz subsets) to compute proportions of bugs likely associated with flagged code.
• Reproducibility: scripts and experiment code made available by authors.

Implications for AI Economics

• Adoption and measured use vs claims
  • Firms report high LLM usage (public reports ~20–30% LLM-generated code in some companies), but detector-based repo evidence shows a declining share of flagged code over 2021–2025 and a concentration in test code. This suggests heterogeneity in how LLMs are actually integrated into production engineering workflows and potential overstatement or different measurement definitions across sources.
• Productivity vs quality trade-offs
  • Concentration of flagged code in tests and limited association with bugs implies firms may be using LLMs for lower-risk, productivity-focused tasks (e.g., test scaffolding, examples, boilerplate). Economically, this supports a model where LLMs substitute low-complexity developer tasks first, delivering quick productivity gains while limiting exposure to high-cost bug risk.
• Firm strategy and differential adoption
  • Company-maintained repos having higher flagged rates suggests firms may more aggressively adopt LLMs (internal tools, premium access, integration into CI/CD). This points to a bifurcated market: firms investing in LLM tooling and workflow integration vs community projects being more conservative.
• Markets for detection, auditing, and compliance
  • Detector-based uncertainty and the lack of ground-truth highlight opportunity for markets providing robust provenance, watermarking, and auditing services. Demand drivers: legal/IP attribution, compliance (secure code provenance), and procurement/assurance for critical systems. Vendors that can credibly attribute LLM involvement or certify low-risk use will have economic value.
• Liability, security, and maintenance costs
  • Small but nonzero proportions of bugs linked to LLM-flagged code indicate potential exposure to security and liability costs. Firms will need to internalize testing and code review costs when deploying LLM-generated artifacts. Economic analyses should account for post-deployment debugging, longer-term maintenance costs from cloned/boilerplate LLM outputs, and potential legal risk (copyright, licensing).
• Labor and task reallocation
  • Findings are consistent with LLMs displacing or reshaping routine coding tasks (tests, boilerplate), increasing demand for higher-skill tasks (design, verification, system integration). This suggests short-term productivity gains but shifting labor composition—greater emphasis on code review, tooling, and evaluation roles.
• Investment and R&D priorities
  • Given detector uncertainty and repository patterns, investments that increase the reliability of LLM outputs (better fine-tuning for code, integrated testing, model auditing) or reduce downstream costs (automated test generation, clone-detection tools) are economically attractive.
• Policy and regulation
  • Empirical uncertainty underscores the need for standards (e.g., provenance labeling, watermarking mandates) if policymakers want to regulate LLM usage in safety-critical software. Regulations would create market demand for detection/compliance solutions but also raise compliance costs for firms.
• Research & evaluation needs for economic models
  • For accurate cost-benefit analyses, economists need better ground-truth datasets on LLM usage and longitudinal measures of maintenance costs, bug-remediation expenditures, and productivity metrics across firms and open-source projects.

Suggestions for stakeholders - Firms: treat current LLMs as productivity multipliers for low-risk tasks but invest in code-review and testing pipelines; consider adopting provenance/watermarking where liability matters. - Tool vendors: opportunity to offer robust detection, watermarking, auditing, and integration with CI/CD; products that reduce false positives/negatives will command premium. - Researchers/policymakers: prioritize building ground-truth datasets and standardized benchmarks to make economic assessments of LLM adoption reliable.

Limitations to carry into economic interpretation - Results are detector-proxy based (no ground truth), so economic conclusions should account for detection uncertainty and potential measurement bias. Further work should combine detector outputs with firm-internal logs and controlled audits for robust valuation.