The rush to create and hoard internet-scale datasets for frontier AI hides mounting environmental and social costs: storage and curation of hundreds of thousands of datasets drive rising carbon burdens, while data-labour risks and representation harms disproportionately fall on workers in the Global South.

Main Finding

Hyper-datafication — the industrialised expansion of data produced, recombined, and synthetically generated to scale frontier AI — creates large, growing sustainability costs that are frequently omitted from model-centric accounting. Using a large Hugging Face metadata study, a survey of Kenyan data workers, and public infrastructure data, the authors show that data volume growth drives substantial environmental footprints (storage, energy, water, embodied emissions), redistributes labour risks to the Global South, and concentrates economic power in a few countries. They propose the Data PROOFS framework (Provenance, Resource awareness, Ownership, Openness, Frugality, Standards) to mitigate these harms.

Key Points

• Definition: Hyper-datafication = (i) large-scale collection/recombination of existing data, (ii) AI-generated synthetic data, (iii) creation of purpose-built datasets produced primarily to train AI.
• Empirical scale:
  • Metadata snapshot (1 Dec 2025) of the Hugging Face Hub: 570,802 datasets available; final analysed sample = 554,300 datasets (2.89% retrieval failure).
  • Top 1% of datasets (≈5,543 repos) account for concentrated downloads.
  • Examples: DCLM-Pool >240 trillion tokens; Phi-4 pretraining ≈10 trillion tokens with ≈40% synthetic; Ego4D = 3,670 hours of purpose-designed egocentric video.
• Environmental indicators:
  • IEA projects global data-centre electricity will more than double by 2030 (≈945–1,300 TWh), driven largely by AI workloads.
  • Large hyperscale projects illustrate scale (e.g., Microsoft’s US$106B, 3.33 GW Wisconsin data‑centre).
  • Storage and preprocessing of datasets (not commonly accounted) add non-trivial energy and carbon emissions beyond reported training footprints (e.g., Llama‑3.1 training footprint excludes dataset curation/storage).
• Social / labour findings:
  • Survey of 134 Kenyan data workers (Dec 2025): evidence of low pay, unpaid/rewarding gaps for harmful-content exposure, direct employment by big tech, and gendered disparities (57 female, 77 male respondents).
  • Data labour risks (psychological harm, precarious pay, surveillance/metrics regimes) are offloaded disproportionately to Global South workers.
• Economic / infrastructure:
  • Data-centre investments and capacity are geographically concentrated, reinforcing economic asymmetries and platform concentration.
• Recommendations:
  • Data PROOFS: Provenance, Resource awareness (carbon/energy metadata), Ownership (compensation/rights for data generators), Openness (transparent accounting), Frugality (minimise redundant storage/processing), Standards (reporting & lifecycle metrics).

Data & Methods

• Primary quantitative source: Hugging Face Hub metadata (retrieved 1 Dec 2025).
  • Sample: 554,300 datasets after 2.89% retrieval failures.
  • Collected attributes: repository identifiers, timestamps, download counts (all-time and last 30 days = Nov 2025), Hub-side storage, dataset size, modality, language, task, region (self-declared).
  • APIs used: Hugging Face Hub REST API, datasets-server API (for dataset size stats), and Hugging Face Python client.
  • Storage measures: (a) dataset size (raw bytes) as proxy for local storage footprint, (b) Hub-side storage (platform footprint including metadata/auxiliary files).
  • Language representation: mapped language tags to ISO-639 codes; compared dataset volumes against global speaker distributions and web presence (Common Crawl page counts) to assess representational skew.
• Energy / carbon estimates:
  • Storage-related energy and carbon footprint estimated for both provider (platform storage) and user sides (downloads/replication); combined with literature/I EA projections for context (data-centre electricity forecasts).
• Qualitative survey:
  • Instrument: 10-question online questionnaire (Appendix B).
  • Participants: 134 voluntary respondents in Kenya recruited via a local data workers’ collective (administered Dec 2025).
  • Topics: demographics, employment relationships, pay, exposure to graphic content, protections/compensation.
  • Limitations: non-random sample, self-report bias, limited geographic scope (Kenya case study).
• External contextual sources:
  • IEA energy forecasts, Data Center Map, corporate project disclosures, and prior literature on AI footprints, data labour, and e-waste.
• Limitations noted by authors:
  • Hugging Face metadata is public and broad but not representative of all proprietary datasets (sampling bias toward hosted public datasets).
  • Self-declared metadata fields have incomplete coverage.
  • Survey sample is localized to Kenya and not globally representative.
  • Estimates for storage energy/carbon rely on modeling assumptions (PUE, regional grid mix, replication factors).

Implications for AI Economics

• Total-cost accounting must include the data lifecycle:
  • Economic analyses of AI projects that focus solely on model training/deployment understate costs. Storage, transfer, preprocessing, synthetic-data generation, and repeated reuse materially increase energy consumption and should be internalised in cost models, carbon pricing, and project appraisals.
• Redistribution of costs and externalities:
  • Hyper-datafication externalises social and health costs (content exposure, precarious labour) and environmental burdens (water use, embodied emissions) disproportionately onto workers and regions in the Global South. Economic models and policy should account for distributional externalities and consider corrective mechanisms (e.g., labour protections, transfer payments, stricter procurement standards).
• Market concentration and platform rents:
  • Data and infrastructure concentration create winner-take-most dynamics; platform owners capture disproportionate economic surplus from data while generators/annotators receive little. This calls for antitrust, data-governance, or taxation policies that address rents from aggregated datasets and infrastructure advantages.
• Investment and localisation trade-offs:
  • Building local data-centre capacity could reduce transfer-related emissions and create local jobs, but may also lock in energy/water-intensive infrastructure if local grids are carbon-intensive. Economic decisions should weigh long-run operational emissions, grid decarbonisation trajectories, and equity considerations.
• Policy and regulation signals:
  • Requiring provenance and resource (carbon/energy) metadata for datasets would enable better market pricing of externalities, allow purchasers to choose lower-impact datasets, and enable regulators to enforce sustainability standards.
• Research and marketplace design:
  • Incentivising frugal data practices (deduplication, selective retention, synthetic data when lower-impact) and standards for dataset lifecycle accounting can change incentives away from “more data at any cost.” Markets (or public procurement) could reward lower-carbon, ethically sourced datasets.
• Modelling recommendations for economists:
  • Incorporate storage/transfer energy and embodied emissions into life‑cycle cost models.
  • Model general equilibrium effects of concentrated data-infrastructure investment (capital flows, labour markets in outsourcing regions).
  • Value non-monetary harms (mental health, representational exclusion) via shadow pricing or regulated compensation mechanisms.
  • Evaluate taxation or levy schemes on large aggregators to fund remediation, worker protections, and infrastructure decarbonisation.

Short takeaway: data is not free. Hyper-datafication materially alters the environmental, social and economic accounting of frontier AI — economists, industry, and policymakers should incorporate dataset lifecycle costs, redistribute accountability, and adopt the Data PROOFS measures to align incentives with sustainable, equitable outcomes.