Growth can blind societies: expansion of a dominant technology may destroy the practices that would reveal and fix its failures, making correction arbitrarily slow even as cross-action evidence accumulates; durable pilots or capitalized cancellation liabilities can substitute for cultural safeguards but require enforcement and funding.

Main Finding

The paper shows that unchecked technological or institutional integration can mechanically destroy the very practices needed to detect and correct a future systemic failure. Whether growth creates “self‑blinding” depends on two scale elasticities: how scale raises the hazard of a systemic breakdown (a) and how scale accelerates cultural/practice erosion (c). If erosion is more scale‑sensitive than breakdown (c > a, or c = a with a sufficiently large erosion coefficient), then the expected time to correct a breakdown is infinite (even though correction almost surely eventually occurs). The model also identifies a mass–share wedge: absolute numbers producing diagnostic evidence can grow with scale while the political share able to implement correction vanishes. Finally, culture can be replicated by fully funded, enforceable pilots, but at explicit capitalized cost and with a rebuilding lag; the paper proposes commitment devices (rolling performance bonds) to finance and protect such substitutes.

Key Points

• Model primitives
  • Total connected scale: mt = m0 e^{g t}.
  • Breakdown hazard: λt = η m_t^a (increases with scale).
  • Outside‑practice intensity per unit scale: s_t, with erosion ˙s = −b m_t^c s_t (cultural/practice capital erodes as the productive regime spreads).
  • After breakdown at time T, correction is possible at rate μ s_T (sequential timing: diagnosis precedes rebuilding).
• Scale–attrition trichotomy (central theorem)
  • Expected correction delay E[D] is finite iff c < a, or c = a and b < η.
  • E[D] is infinite iff c > a, or c = a and b ≥ η.
  • Intuition: if breakdown risk rises faster with scale than erosion (a > c), growth tends to bring breakdown earlier but leaves more corrective capacity at that hazard-clock time; if erosion dominates, rare long successful spells eliminate corrective capacity before breakdowns occur, producing heavy tails (infinite mean delay).
• Mass–share separation
  • Distinguishes the absolute mass of outside practice (which produces diagnostic comparisons/evidence) from the demographic or political share (which determines ability to implement correction).
  • In a constant‑erosion benchmark χ_t = χ_0 e^{−β t} with mt ∝ e^{g t}, absolute mass k_t = m_t χ_t ∝ e^{(g−β)t}. If g > β, absolute mass → ∞ while χ_t → 0. Thus evidence can be abundant while political power to act vanishes.
• Political and economic boundaries
  • Majoritarian unraveling: any fixed non‑unanimous population rule q < 1 is crossed in finite time as skeptical share decays, so repeal of protections can occur even while skeptical mass grows.
  • Buyout threshold: with transferable utility, repeal becomes privately profitable when integration surplus Γ > χ_t DR (compensation bill proportional to share χ_t).
• Culture as capital and funded replication
  • A fully enforced external pilot can replicate the counterfactual practice exactly if the discounted reproduction factor δ M < 1; its present cost B = κ_E / (1 − δ M).
  • If δ M ≥ 1, no finite fund can replicate the descendant path (the pilot would require infinite funds).
  • Cancelling a practice exposes an unfunded rebuilding liability; the paper proposes a rolling performance bond forfeited to an independent trust to fund replacement and block opportunistic repeal while allowing valuable reforms.
• Asymmetry and rebuilding lag
  • Worldviews or policies can reverse quickly, but restoring the tacit practice takes time proportional to the destructive successful spell: R_T = A(T)/ρ, where A(T) accumulates erosion during the productive period and ρ is maximal reconstruction speed.
• Policy characterization
  • There is an optimal cap on integration in a stationary benchmark that trades productive gains versus the depreciation of future epistemic capacity.
  • Atomistic adopters will over-adopt relative to that social optimum (familiar externality form but acting on a distinctive state variable: scale that depreciates epistemic capital).

Data & Methods

• Analytical, continuous‑time theoretical model:
  • Deterministic exponential growth for connected scale; Poisson arrival of breakdowns with scale-dependent hazard.
  • Differential equation for cultural/practice erosion driven by scale.
  • Change-of-clock (hazard-clock) technique: characterize erosion measured at the random hazard time using Λ(t) and its inverse.
  • Formal results: lemmas, propositions, and theorems deriving the trichotomy, mass–share separation, political boundaries, and capitalized cost formulas.
• Benchmarks and extensions:
  • Constant‑erosion demographic benchmark (χ_t exponential decay) used to illustrate mass–share wedge and political thresholds.
  • Stationary/regenerative benchmark for an optimal integration cap (appendix).
  • Complete‑contracting benchmark to price a fully‑funded pilot; limited‑commitment discussion motivates rolling bonds.
• No empirical data or calibration—pure theory. The results rely on power‑law scaling for transparency of boundaries, but the core comparisons are generalized via the hazard‑clock mapping and conditions on asymptotic ratios of erosion measured in hazard time.

Implications for AI Economics

• Framing for AI adoption
  • Widespread deployment of a dominant AI architecture or governance practice can raise social output but may simultaneously erode alternative operational practices (human oversight, manual procedures, decentralized testing regimes) that serve as counterfactuals and enable diagnosis and recovery from catastrophic failures.
  • Whether AI-driven expansion creates “self‑blinding” depends on whether practice erosion scales faster with adoption than the hazard of an adverse regime (e.g., systemic misalignment, capture, or catastrophic failure).
• What to measure empirically
  • Estimate the scale elasticity of systemic hazard (a): how the probability of a major AI failure increases with integrated scale (network size, interconnected deployments, concentration).
  • Estimate the scale elasticity of practice erosion (c): how fast operational diversity, skills, and alternative processes decline as AI adoption scales.
  • Measure erosion coefficients (b), reproduction factors for practice capital (M), and discounting (δ) to price replication costs.
  • Track absolute counts of alternative-practice units (mass) versus their political/organizational shares.
• Policy and institutional design suggestions
  • Preserve operational diversity: design institutions and incentives that maintain jurisdictional or organizational pockets of alternative practice (human-in-the-loop modes, offline testing labs, manual fallback procedures).
  • Fund and enforce pilot programs: where culture cannot be preserved organically, set up fully funded, audited pilots that maintain the full bundle (action, records, training) and capitalize their replacement cost (B) or bridging cost (B_R) before canceling existing practices.
  • Use commitment devices: rolling performance bonds forfeited to an independent trust on cancellation can finance replacement liabilities and reduce opportunistic repeal while keeping policy flexibility.
  • Decentralization and representation: maintain political or jurisdictional protections (representation not tied to cultural conversion) so that declining shares of counterfactual practitioners still retain blocking power where warranted.
  • Monitor the mass–share wedge: be alert that rising absolute numbers producing diagnostic evidence do not guarantee corrective capacity if their relative share and organizational power shrink.
  • Cap integration when needed: where erosion elasticities imply a high risk of self‑blinding, optimal policy may require slowing integration (e.g., phased rollouts, regional segmentation) to preserve epistemic capital.
• Research directions for AI economics
  • Empirical calibration of elasticities (a, c) using historical episodes (financial networks, industrial standardization, safety transitions) or sectoral AI deployment data.
  • Simulations and agent‑based models mapping adoption dynamics, cultural transmission, and political institutions to quantify probabilities of long‑tail correction delays.
  • Policy experiments estimating κ_E (cost to sustain a practice unit), M (practice reproduction), and δ in settings like regulated pilot programs.
  • Institutional design work to operationalize rolling bonds and trust structures to finance practice maintenance and rebuilding.

Limitations and caveats - The analysis is theoretical and stylized (continuous time, Poisson arrivals, specific functional forms for clarity). Practical implementation requires empirical parameter estimates. - The ability to “replicate” culture via contracts requires that tacit components are verifiable and enforceable; if key tacit elements are non‑contractible, replication cost is effectively infinite. - Sequential timing (diagnosis before rebuilding) is central; mechanisms where practice auto‑rebuilds at failure would alter the trichotomy.

Overall, the paper provides a clear formal mechanism by which successful AI diffusion can erode epistemic capacity needed to recognize and correct systemic AI failures, quantifies the conditions under which this erosion becomes dangerous (infinite expected correction time), and supplies institutional instruments (funded pilots, bonds, decentralization) to preserve or replicate that epistemic capital—important concepts for AI safety policy and the economics of large‑scale AI integration.