Dependence on LLMs risks eroding engineers' ability to diagnose and fix complex systems, creating hidden 'epistemological debt' that can amplify outages like Amazon's 2026 failure; firms should pair AI tools with rigorous human-in-the-loop training and governance to preserve system resilience.

Main Finding

Widespread reliance on LLMs for code generation creates an invisible, accumulating liability—“epistemological debt”—by decoupling code production from engineers’ mental models. This drives cognitive atrophy among practitioners, narrows the diversity of the global codebase through recursive synthetic-data feedback (“polluted well”), and increases systemic fragility (higher incidence/severity of failures and security regressions). Short-term productivity gains can therefore produce long-term negative externalities across software ecosystems and the broader AI economy.

Key Points

• Epistemological Debt
  • When engineers accept AI-generated code without deriving it themselves, they hold code they do not understand. This gap (the “why”) impedes root-cause analysis and recovery after failures.
  • The common failure mode is a repeating human–model “iteration rabbithole” where fixes compound defects instead of resolving root causes (Shukla et al., 2025).
• Cognitive Atrophy
  • LLMs automate not only execution but derivation, replacing the “cognitive gym” of debugging and design with verification tasks, reducing development of algorithmic thinking and tacit knowledge (Polanyi, 1966; Lee et al., CHI ’25).
  • Empirical signs include higher acceptance rates of AI-generated reviews (e.g., Amazon Q Developer reported 79% acceptance) and weaker critical scrutiny.
• Polluted Well / Mechanized Convergence
  • Recursive training on AI-generated code biases models toward mean patterns, shrinking variance and losing “tails” (novel or optimal solutions). Shumailov et al. (2024) show model collapse under recursive training.
  • This feedback loop can entrench common vulnerabilities and stifle innovation in code patterns.
• Case study: 2026 Amazon outages
  • Paper cites Amazon’s 2024 Q Developer adoption (large productivity/cost savings) and links gen-AI assisted changes to outages in 2026 that prompted reinstatement of senior human approvals (TechHQ, 2026). Used as an illustration of systemic risk materializing.
• Proposed mitigations
  • Ban or restrict generative-AI use in core CS education to preserve foundational skills.
  • Treat AI-generated code as untrusted third-party components with elevated review standards—mandatory senior sign-off.
  • Preserve human-generated code diversity (“data hygiene”) to avoid model collapse and maintain innovation capacity.

Data & Methods

• Approach: conceptual/theoretical argument supported by a targeted literature review, references to modeling/theoretical results, prior empirical studies, and a case study narrative.
• Evidence sources cited:
  • Theoretical/modeling: Shumailov et al. (2024) on model collapse from recursive training.
  • Empirical/observational studies: Lee et al. (CHI ’25) on reduced critical thinking with GenAI; Shukla et al. (2025) quantifying increased security vulnerabilities after iterative AI code generation.
  • Industry reports/anecdotes: Amazon Q Developer adoption figures (AWS DevOps blog, 2024), Amazon outages and subsequent policy responses (TechHQ, 2026), executive commentary (Jassy, 2024).
  • Other literature on AI-driven software engineering dynamics (Cito & Bork, 2025; Gerstgrasser et al., 2024).
• Methods limitations
  • Largely conceptual and synthetic: no primary longitudinal dataset tracking individual cognitive decline or causal inference linking LLM use to system-wide collapse.
  • Case study evidence (Amazon) is suggestive but not a controlled causal analysis; multiple confounders plausible.
  • Some quantitative claims come from secondary studies with their own methodological constraints; generalizability across firms, domains, and model types is uncertain.

Implications for AI Economics

• Short-term productivity vs. long-term human-capital depreciation
  • Measured gains (developer-years saved, faster commits) can mask depreciating skills that reduce long-run productivity and increase expected outage/security costs. Production function models should include human-capital depreciation as a dynamic cost.
• Negative externalities and market failure risks
  • The “polluted well” is a negative externality: individual firms or developers committing AI-generated, potentially suboptimal/vulnerable code impose learning and security costs on the whole ecosystem (models trained on that code degrade).
  • Left unpriced, these externalities can lead to under-provision of verification, governance, and training.
• Labor-market distortions and skill-biased effects
  • Demand shifts toward senior/architect roles for oversight may raise wages and create a bottleneck, increasing organizational costs. Junior roles may experience skill atrophy and lower long-run employability/value.
• Systemic risk and insurance/operational costs
  • Increased probability of large-scale outages or security incidents raises expected loss for firms and insurers; this could produce higher premiums, stricter SLAs, and regulatory scrutiny.
• Platform concentration and governance implications
  • Model providers and major code hosts gain leverage: they control model updates, data curation, and thus ecosystem diversity. This concentration can amplify systemic fragility and raise entry barriers.
• Policy and firm-level responses with economic effects
  • Interventions (education restrictions, mandatory human approval, data-hygiene requirements, audits) impose compliance costs but internalize risk externalities; these will affect adoption rates and the net social benefit of LLMs in software engineering.
• Research and measurement agenda for economists
  • Track metrics that capture these dynamics: frequency/severity of AI-linked outages, acceptance rates of AI-generated code, vulnerability incidence correlated with AI iterations, measures of codebase diversity/variance, longitudinal skill measures for engineers, and the ratio of verification to production effort.
  • Model extensions: incorporate human-capital atrophy and epistemic debt into task-based models of automation; analyze equilibrium adoption when externalities and oversight costs are internalized; simulate how recursive-data feedback affects model performance and social welfare.

Suggested near-term actions for researchers and policymakers - Empirically quantify epistemological debt: longitudinal cohorts comparing skill trajectories of engineers with/without LLM reliance. - Require transparency/audit trails for AI-assisted code changes (to link incidents to provenance). - Incentivize curated human-generated datasets and provenance-aware training to reduce polluted-well risks. - Consider temporary curricular rules and certification for production use of AI-assisted code to preserve foundational skills.

Overall, the paper frames important, plausibly systemic costs of indiscriminate LLM adoption in software engineering. Its core claims merit targeted empirical follow-up and incorporation into economic models of automation that account for human-capital dynamics, externalities, and systemic risk.