U.S. law tends to tolerate web scraping for AI training if data are publicly accessible, while the EU's GDPR makes scraping personal data legally risky unless a clear lawful basis exists, creating regulatory friction for AI developers and prompting calls for reform.

Main Finding

Data scraping is central to AI model training but conflicts with core data‑privacy principles. Under U.S. law, scraping publicly accessible information is generally permissible; under EU law, scraping publicly available personal data will usually breach the GDPR unless a lawful basis under Article 6 applies. Although scraping is not categorically illegal in the EU, GDPR’s strict requirements make most large‑scale scraping of personal data difficult to justify in practice. Enforcement across EU member states is uneven. The authors conclude that existing privacy regimes inadequately balance AI innovation and individual privacy and call for legal reform to reconcile these objectives.

Key Points

• Importance of data: AI model performance heavily depends on large, diverse, and (ideally) high‑quality datasets; this drives large‑scale automated scraping.
• Forms of training data: public web data, licensed datasets, user‑provided data, third‑party contractual data, and data donations/altruism.
• Nature of scraping: automated scraping (bots, crawlers) enables massive, low‑cost extraction distinct from manual copying; often occurs without data‑subject or platform consent.
• U.S. legal position: courts and practice tend to allow scraping of publicly accessible information; absence of comprehensive federal privacy law creates permissive environment.
• EU legal position: GDPR applies to personal data even if publicly available; lawful processing requires an Article 6 basis (consent, contract, legal obligation, vital interests, public task, or legitimate interests with balancing). Data scraping of personal data will typically fail GDPR tests unless designed to fit a lawful basis and meet principles (transparency, purpose limitation, minimization, security).
• Privacy principles undermined by scraping: fairness; control and individual rights (access, correction, deletion); transparency and consent; purpose specification and restrictions on secondary use; data minimization; onward transfer/contractual protections; data security.
• Ripple effects and harms: enables mass and shadow profiling, targeted cyberattacks, identity fraud, mass surveillance, and algorithmic/personalized price discrimination.
• Practical caveats: GDPR enforcement varies across member states, producing heterogeneous compliance burdens and regulatory risk.
• Policy recommendation (authors): rethink and reform privacy laws to better reconcile the needs of AI development with privacy protection.

Data & Methods

• Methodology: doctrinal legal research (legal and jurisprudential analysis) and literature review of academic work, policy reports, and case law.
• Legal sources and cases referenced: GDPR (esp. Article 6 and Article 5 principles); EU Data Governance Act (data altruism); U.S. cases and decisions cited historically include Authors Guild v. Google Inc. (2015) and Authors Guild v. HathiTrust (2014) (copyright contexts that informed scraping debates); Clearview AI litigation and regulatory scrutiny; Andrea Bartz, Charles Graeber, and Kirk Wallace Johnson v. Anthropic (2025) referenced as an AI-related dispute over data sourcing; regulatory guidance from DPAs and reports from OECD, national authorities, and privacy scholars (Solove & Hartzog, etc.).
• Empirical content: mainly legal interpretation and normative argumentation rather than econometric or experimental data.

Implications for AI Economics

• Data access and cost effects:
  • Regulatory differences (U.S. permissive vs. EU restrictive) create cross‑jurisdictional asymmetries in effective access to training data.
  • Stricter EU requirements increase firms’ compliance costs (legal review, consent mechanisms, data audits, DPIAs) and transaction costs for acquiring lawful datasets (licenses, contracts).
  • Higher compliance costs favor incumbents with legal, financial, and data assets — increasing barriers to entry and potential market concentration in AI.
• Innovation vs. privacy trade‑off:
  • Blanket bans or excessive restriction on scraping can slow model improvement, especially for smaller firms and researchers who rely on publicly available web data.
  • Conversely, permissive regimes risk negative externalities (privacy harms, fines, reputational risk) that could reduce social welfare and trust, ultimately undermining AI adoption.
• Market design and data markets:
  • Pressure for regulated data access models: standardized licensing, APIs with clear terms, compensated data markets, data trusts, or safe‑harbors for research use could lower transaction costs while embedding privacy safeguards.
  • Data altruism / donation frameworks (e.g., EU Data Governance Act) may supply some lawful data but scale and representativeness are uncertain — affecting model performance and bias.
• Algorithmic harms and economic welfare:
  • Scraped data enabling profiling and personalized pricing can produce redistributive harms and welfare losses (consumer surplus extraction, discrimination), prompting potential regulatory intervention that affects pricing strategies and market structure.
• Compliance uncertainty and investment:
  • Heterogeneous enforcement (within EU and between US/EU) creates regulatory uncertainty that can deter investment in cross‑border model training and push firms to localize datasets or tailor models per jurisdiction.
• Practical mitigation strategies with economic consequences:
  • Technical measures (differential privacy, synthetic data, federated learning) can reduce privacy risks but add development costs and may reduce model accuracy — an economic trade‑off between privacy protection and performance.
  • Licensing and negotiated data sharing shift costs to buyers and may create new revenue streams for data holders (platforms, publishers), potentially changing incumbents’ incentives.
• Policy implications for regulators and policymakers (economically salient):
  • Consider harmonized, proportionate legal clarifications (e.g., narrow, supervised research exceptions; standardized lawful bases for AI training) to reduce fragmentation and lower compliance costs while protecting privacy.
  • Promote market mechanisms (data marketplaces, standardized APIs, certified processors) and technical standards (privacy‑preserving ML) to internalize privacy externalities without killing beneficial innovation.
  • Monitor distributional impacts: support smaller firms/researchers (grants, access programs) to avoid undue concentration of AI capabilities.

Concise takeaway: Large‑scale scraping materially lowers the marginal cost of collecting training data, accelerating AI development but producing significant privacy externalities and regulatory risk. Absent careful reform (legal, market, and technical), the economics of AI will favor established firms and create social harms (profiling, discrimination), so policy design should aim to align data access incentives with robust privacy safeguards.