Language AI is concentrated in a handful of tongues, leaving thousands of communities underserved and gaps widening rapidly; the EQUATE index maps where infrastructure exists but AI access remains unrealized.

Main Finding

AI-driven language technologies are concentrating ever more tightly in a small set of languages, producing a new, accelerating global linguistic hierarchy. The authors document a process they call “Zipfianisation”: resource availability (models, datasets) follows a power-law and has become hyper-concentrated during the LLM era, with English and a handful of other lingua francas far outpacing the rest. This inequality is widening over time, diffuses in atypical hype-driven bursts rather than a gradual S-curve, and is driven by a mix of web-data availability, historical/institutional factors, and top-down industrial priorities. The paper introduces EQUATE, a Language AI Readiness Index, to identify where investments would most efficiently close the divide.

Key Points

• Scope: Analysis covers 6,003 attested languages (those with at least one documented lexical item).
• Resource concentration: Number of language models and datasets on Hugging Face (2020–2024, Wayback Machine snapshots) follow a power-law/Zipf distribution; English is an extreme outlier with orders-of-magnitude more models than expected.
• Zipfianisation: Disparities have intensified during the LLM era (examples: English annual increases up to ~50,000 new models vs. average under-resourced languages gaining ~4.15 models/year).
• Representation washing: Broad coverage statistics can mask poor performance and superficial inclusion of low-resource languages (datasets/models that do not yield practical capability).
• Geography & bias:
  • Weak positive correlation between speaker population and model counts (OLS on log-log: β1 = 0.312, p < 0.001, R2 = 0.304). Large deviations expose systemic bias.
  • Under-resourced examples: Nigerian Pidgin (~85M speakers, 10 models), Chittagonian (~13M, 1 model), Wu Chinese (~80M, 10 models).
  • Over-representation concentrated in Europe (including small/minority or dead languages such as Latin), reflecting institutional/academic investment rather than present-day need.
• Diffusion dynamics:
  • Language-model coverage (speakers with at least one ready-to-use conversational AI) does not follow classic S-shaped adoption; instead shows early hyper-growth followed by lock-in.
  • Gompertz fit parameters for language models: displacement rate b ≈ 0.927, growth constant c ≈ 1.31, R2 ≈ 0.866 (contrasting with S-curve fits for phones, PCs, EVs).
• EQUATE index: First open-source Language AI Readiness Index combining AI resources, digital infrastructure, and socioeconomic indicators (25 features). PCA identifies two main components (AI resources vs. socio-digital infrastructure) explaining ~58.4% of variance.
• Predictors of model availability (mixed-effects, stepwise): existence of a Bible in the language (β = 3.287), number of speakers (β = 1.690), GBs in OPUS (β = 0.974) and XEUS (β = 1.232) — highlighting the role of available corpora and legacy text resources.

Data & Methods

• Core data sources:
  • Hugging Face model & dataset pages (monthly Wayback Machine snapshots for December of each year 2020–2024).
  • Web-corpus measures (CommonCrawl-derived GBs, OPUS, XEUS), Wikipedia activity, ACL Anthology validation.
  • Socioeconomic and infrastructural indicators from peer-reviewed and international organization datasets (25 total features).
• Validation: Representativeness of Hugging Face collection validated against ACL Anthology language coverage.
• Statistical methods:
  • Descriptive analysis showing power-law (Zipf) distributions and computing exponents.
  • Longitudinal analyses of model/dataset growth per language (2020–2024).
  • OLS regression (log-log) of number of models vs. speaker population; identification of large residuals to flag under-/over-resourced languages.
  • Mixed-effects stepwise linear regression (random effects: language family, primary country, macroarea) to identify predictors of model counts.
  • Principal Component Analysis (varimax rotation) to uncover main dimensions underlying the 25 features.
  • Diffusion analysis: fitted Gompertz curves to compare adoption dynamics of language models vs. technologies with classic S-shaped diffusion (mobile phones, PCs, EVs).
• EQUATE construction:
  • Two-stage process: (1) correlational/dimensionality analysis to select non-redundant features and grouping; (2) global expert survey to assign weights to index dimensions.
  • Languages scored and categorized into readiness tiers; index decomposable into three domains (AI resources, digital infrastructure, socioeconomic conditions).
• Limitations acknowledged (and to be considered in use): dependence on Hugging Face as a visibility proxy, web-archival completeness, speaker-count uncertainties, and that “existence of a model” does not guarantee usable capability for speakers.

Implications for AI Economics

• Market concentration and public-good failure:
  • Language AI shows classic “winner-takes-most” dynamics amplified by data-driven positive feedbacks; private incentives concentrate investment in a few languages while social returns from broader linguistic inclusion are uninternalized.
  • Overinvestment in dominant languages (English et al.) yields diminishing marginal social returns relative to high-need, under-resourced languages — suggesting market failure and a role for public intervention.
• Allocation of scarce resources:
  • EQUATE provides a principled instrument for funders, governments, and NGOs to prioritize languages where additional investment (data collection, compute, localization) would yield high marginal welfare gains (i.e., languages with high readiness but low current resources).
  • Economic policy options: targeted subsidies or procurement for under-resourced languages, prize mechanisms, public funding for corpus creation and annotation, and conditional requirements for deployed multilingual products.
• Regulatory and industrial policy:
  • Antitrust/competition and R&D policy could encourage wider linguistic coverage (e.g., mandating minimum coverage in public-interest deployments, funding inter-operable open datasets).
  • Procurement by public-sector actors (health, education, governance) could generate demand-side incentives to develop high-quality models for low-resource languages.
• Productivity and inequality effects:
  • If left unchecked, Zipfianisation will channel AI-enabled productivity gains into already-advantaged language communities, exacerbating global income and knowledge-access inequalities.
  • Inclusive language-AI development can unlock local economic opportunities (education, health, entrepreneurship) and increase aggregate economic welfare; cost-effectiveness depends on selecting languages where infrastructure and socioeconomic readiness make impact more likely — exactly what EQUATE identifies.
• Design of interventions:
  • Short-term: finance corpus building, community-driven data initiatives, and open model benchmarks targeted by EQUATE scores.
  • Medium-term: invest in local computational infrastructure and capacity building (engineers, annotators, domain experts), and support multilingual fine-tuning approaches that prioritize under-served languages rather than incremental improvements to dominant-language models.
  • Long-term: restructure incentives so platform owners and model-hosting ecosystems internalize social returns from linguistic inclusion (e.g., licensing, reporting, matching grants).
• Research and evaluation:
  • Economic evaluation should measure returns not only in developer metrics (model count) but in service usage, welfare gains in education/health, cultural preservation, and downstream labor market effects.
  • Monitor for “representation washing”: counting models/datasets is insufficient — assessments must include functional performance and community uptake.

Summary recommendation: Treat linguistic inclusion as a targeted public-good problem. Use EQUATE to prioritize languages where investments are most likely to translate into economic and social value, and design policy interventions (subsidies, procurement, public-data programs) that correct the market incentives currently producing Zipfianisation.