← Papers

Subword tokenizers shrink effective context for Indian-language users: under OpenAI's cl100k_base tokenizer Indian languages use on average 8× more tokens than English (Malayalam 13×), meaning much smaller usable context and higher per-token cost, but multilingual and newer tokenizers cut the gap by roughly three-quarters.

The Tokenizer Tax: Quantifying and Explaining the Cross-Lingual Cost of Subword Tokenization for Indian Languages

Priyansh Srivastava · July 27, 2026

arxiv descriptive high evidence 8/10 relevance Full text usable extracted full text Source PDF

Structured author observations

Linked only from stored provider relations; the raw author line above is never matched by name.

Arxiv

Latest observation: August 4, 2026

Priyansh Srivastava unresolved corpus identity

Semantic Scholar

Latest observation: August 25, 2026

Priyanshu Srivastava provider ID

Common subword tokenizers impose a large tokenizer 'tax' on Indian languages — averaging 8× (up to 13× for Malayalam) more tokens than English under cl100k_base — largely explained by unmerged single-byte tokens, and multilingual/newer tokenizers reduce this tax substantially (≈73%).

Citation observations

Cumulative provider counts captured on specific dates; providers are never combined.

0 cumulative citations

Semantic Scholar · Observed August 25, 2026

View corpus context

Large language models (LLMs) process text through subword tokenizers rather than directly reading characters or words. Because these tokenizers are trained predominantly on English-centric corpora, they introduce a systematic and often overlooked disadvantage for many non-English languages. In this work, we quantify this tokenizer tax for Indian languages using the FLORES-200 parallel corpus, measuring tokenization fertility across six widely used tokenizers and fourteen languages. Under cl100k_base (used by GPT-3.5 and GPT-4), Indian languages experience an average 8.0x tokenization tax relative to English, reaching 13.0x for Malayalam, reducing the effective context window to as little as 12% of that available to English users for equivalent semantic content. We identify the primary mechanism behind this disparity: failed byte-pair merges that leave text fragmented into single-byte tokens, with merge failure strongly correlating with tokenizer tax (Pearson r = 0.89). We further show that this phenomenon is not an inherent property of Indic scripts but a consequence of tokenizer design. Multilingual tokenizers such as XLM-R and OpenAI's o200k_base reduce the average Indic tokenizer tax by 73%, demonstrating that the disparity is largely remediable. Beyond token statistics, we quantify a practical consequence by showing that, under fixed context budgets, Indian-language documents preserve substantially less original content than equivalent English documents. Finally, we examine the relationship between tokenizer fertility and reading comprehension performance on the Belebele benchmark, finding that the apparent correlation is largely explained by language resource availability rather than tokenizer behavior alone.

Summary

Main Finding

Tokenization trained predominantly on English creates a large, systematic "tokenizer tax" on Indian languages: under OpenAI’s cl100k_base tokenizer Indian languages use on average 8.0× more tokens than English for the same content (up to 13.0× for Malayalam). This is primarily caused by BPE merges failing and leaving many single-byte tokens, and it is largely a remediable design choice—newer and multilingual tokenizers already cut most of the gap.

Key Points

Magnitude and structure of the tax
- Mean tax across ten Indian languages under cl100k_base (GPT-3.5/GPT-4 tokenizer): 8.0× relative to English.
- Worst case: Malayalam ≈ 13.0×. Dravidian languages (Tamil, Telugu, Kannada, Malayalam) are taxed most heavily.
- European Latin-script languages pay ~1.3–1.6×; Urdu/Arabic (Perso‑Arabic script) ≈ 3.4×.
Mechanism
- Dominant mechanism: failure of BPE merges leaving tokens as unmerged single bytes. The single‑byte unmerged rate correlates with the per‑language tax at r = 0.89.
Remediability
- Tokenizer choice matters: multilingual tokenizers (XLM‑R) and OpenAI’s newer o200k_base dramatically reduce the tax.
- Moving from cl100k_base → o200k_base reduces mean Indic tax from 8.0× to 2.1× (≈73% reduction).
- Multilingual tokenizers show much lower inequality (Gini coefficients reported: cl100k 0.35 → o200k 0.19 → XLM‑R 0.14).
Model-free harm
- For a fixed token budget (e.g., 8,192 tokens) Indian-language users receive a much smaller effective context in characters — as low as ~12% of what an English user receives for identical content.
Downstream performance
- Raw correlation between fertility (tokens per word) and Belebele reading‑comprehension accuracy (InfoXLM) is moderate negative (r = −0.61), but this relationship is substantially confounded by resource level. Controlling for resource (log proxy) yields a small positive partial correlation (r = 0.25). The paper does not claim a clean causal effect of fertility on accuracy.
Practical recommendations in the paper
- For model builders: improve vocabulary coverage for Indic scripts; monitor unmerged single‑byte rate as a diagnostic.
- For API providers: recognize per‑token pricing is regressive across languages.
- For practitioners: choose tokenizers deliberately for Indic applications.

Data & Methods

Parallel, content‑controlled corpus: FLORES‑200 dev set (997 professionally translated and aligned sentences across 200+ languages) to ensure identical content across languages.
Languages studied: ten Indian languages (Hindi, Marathi — Devanagari; Bengali — Bengali; Punjabi — Gurmukhi; Gujarati — Gujarati; Urdu — Perso‑Arabic; Tamil, Telugu, Kannada, Malayalam — Dravidian scripts) plus English, Spanish, French, Arabic as comparisons.
Tokenizers evaluated: cl100k_base (GPT‑3.5/GPT‑4), o200k_base (GPT‑4o), GPT‑2, Qwen2.5, mBERT, XLM‑R.
Metrics computed per (tokenizer, language):
- Word fertility: tokens per whitespace‑delimited word.
- Character fertility: tokens per character.
- Bytes per token.
- Unmerged single‑byte rate: fraction of emitted tokens that are single raw bytes (practical indicator of failed BPE merges).
- Tax multiplier: word fertility relative to English under the same tokenizer.
- Context shrinkage/usable characters: characters usable within a fixed token budget B (B / char_fertility), reported relative to English.
Downstream analysis: correlated cl100k_base fertility with Belebele reading‑comprehension accuracy (InfoXLM Translate‑Train‑All scores) for 13 languages, controlling for a log‑resource proxy.
Reproducibility: methods are CPU‑runnable and the authors intend to publish code/pipeline post peer review.

Implications for AI Economics

Pricing and access inequality
- Per‑token billing creates a regressive price structure: speakers of high‑fertility languages pay many times more (e.g., ~8× median, ~13× worst case under cl100k_base) to transmit identical content. This raises affordability and access concerns in large markets (India).
Effective resource allocation and product design
- Fixed token budgets (e.g., context windows, free tiers, rate limits) are effectively smaller for speakers of taxed languages, reducing utility and increasing friction for those users even when nominal limits are the same.
- API providers and platform designers should consider compensatory measures (e.g., language‑aware quotas, adjusted pricing, or larger context windows for affected languages) to avoid regressive outcomes.
Low‑cost fairness interventions
- Tokenizer retraining or adopting multilingual vocabularies is a high‑leverage, low‑cost intervention: expanding vocabulary coverage or using multilingual tokenizers substantially reduces token inequality without changing model architecture or training data scale.
- Monitoring metric: unmerged single‑byte rate is a simple diagnostic that predicts tokenization tax and can be operationalized in evaluations and SLAs.
Market and regulatory considerations
- If per‑token pricing remains standard, markets with many speakers of high‑fertility languages could face higher costs for LLM usage; this could slow adoption, change competitive dynamics, or justify regulatory attention to pricing fairness.
- Policymakers or procurement agents (e.g., governments, NGOs) may want to require language‑equitable tokenization as part of procurement or subsidize costs for under‑tokenized languages.
R&D and investment signals
- Investment in tokenizer and vocabulary improvements (including targeted data collection for underrepresented scripts) is a cost‑effective way to reduce downstream economic inequities from LLM deployment.
- Companies offering multilingual products should prioritize tokenizer choices as part of fairness-by-design, since tokenizer improvements can produce immediate economic and UX benefits for large language communities.

Caveats and limitations relevant to economics - The study uses FLORES translations (possible translationese) and whitespace word segmentation for word‑level metrics; some scripts may require character‑level measures. - The downstream performance link (fertility → accuracy) is confounded by resource level; claims about economic impacts via accuracy should be made cautiously until controlled interventions or further studies are performed.

If you’d like, I can: - produce a one‑page slide with the key numbers and recommended actions for product/ pricing teams, or - simulate the per‑token cost impact for a given API price and sample content in English vs. a chosen Indian language.

Assessment

Paper Typedescriptive Evidence Strengthhigh — Deterministic, content-controlled measurements on a parallel corpus (FLORES-200 dev, 997 sentences) across multiple languages and six real-world tokenizers produce reproducible token-count and byte-level statistics; strong correlation (r=0.89) between unmerged single-byte rate and fertility supports the mechanistic claim. Methods Rigorhigh — Uses a parallel, content-controlled dataset to isolate tokenizer effects from content differences; evaluates multiple tokenizers (including multilingual and newer variants) and reports multiple fertility metrics; transparently acknowledges limitations (translationese, tokenizer-specific heuristics, limited n for downstream accuracy). No experimental intervention on tokenizers or models, but descriptive/mechanistic claims are well supported. SampleFLORES-200 development set: 997 professionally translated and aligned sentences used as a content-controlled parallel corpus across 13 languages (10 Indian languages: Hindi, Marathi, Bengali, Punjabi, Gujarati, Urdu, Tamil, Telugu, Kannada, Malayalam; plus English, Spanish, French, Arabic). Six tokenizers evaluated: OpenAI cl100k_base and o200k_base, GPT-2, Qwen2.5, mBERT, and XLM-R. Downstream accuracy correlation used Belebele reading-comprehension scores (InfoXLM Translate-Train-All) for 13 languages (n=13). Main token-budget analyses used an 8,192-token budget to compute usable characters. Themesproductivity inequality GeneralizabilityResults derived from a single parallel dataset (FLORES-200 dev, 997 sentences) and may vary across domains, longer documents, or colloquial text, Fertility metrics depend on whitespace-based word segmentation for some scripts; character-fertility mitigates but does not eliminate this, Unmerged single-byte detection is tokenizer-specific and best-effort; different implementations might yield slightly different rates, Downstream accuracy correlation uses accuracy scores computed under a different tokenizer (InfoXLM) and a modest sample (n=13), limiting causal claims about fertility→accuracy, Tokenizers, models, and deployed vocabularies evolve; measured tax magnitudes may change for other tokenizer generations or retrained vocabularies

Claims (9)

Claim	Direction	Outcome	Confidence & Evidence	Details
Under cl100k_base, the ten Indian languages have a mean tokenizer tax of 8.0× relative to English, with Malayalam reaching 13.0×. Consumer Welfare	negative	Word-fertility ratio relative to English, representing the number of tokens required for equivalent content.	Reading fidelity high Study strength high	n=997 mean tax 8.0×; Malayalam tax 13.0× 0.3
At a fixed 8,192-token context budget, every Indian language studied retains less than one quarter of the usable characters retained by English; Kannada and Telugu retain approximately 12%. Consumer Welfare	negative	Usable real-text characters within an 8,192-token context window relative to English.	Reading fidelity high Study strength high	n=997 12–23% of the English window 0.3
The rate of unmerged single-byte tokens is strongly associated with the tokenizer tax across languages, with a correlation of r=0.89. Consumer Welfare	negative	Tokenizer tax multiplier and unmerged single-byte token rate.	Reading fidelity high Study strength medium	r=0.89 0.18
High-tax Indic languages emit unmerged single-byte tokens at rates of approximately 27–43%, compared with under 10% for English and European comparison languages. Consumer Welfare	negative	Unmerged single-byte token rate.	Reading fidelity high Study strength high	n=997 27–43% for high-tax Indic languages versus under 10% for English and European languages 0.3
The tokenizer tax follows a typological gradient: Latin-script European languages have taxes of 1.3–1.6×, Urdu and Arabic are near 3.4×, and Dravidian languages range from 9.9× to 13.0× under cl100k_base. Consumer Welfare	negative	Word-fertility tax multiplier relative to English by language and script family.	Reading fidelity high Study strength high	n=997 1.3–1.6×; approximately 3.4×; 9.9–13.0× 0.3
The tokenizer tax is substantially reduced by multilingual and newer tokenizers: XLM-R reduces Malayalam's tax to 2.0×, while o200k_base reduces the mean Indic tax from 8.0× to 2.1×, a 73% reduction. Consumer Welfare	positive	Language-specific and mean Indian-language tokenizer tax multiplier.	Reading fidelity high Study strength high	n=997 73% reduction; Malayalam reduced from 13.0× under cl100k_base to 2.0× under XLM-R 0.3
Higher fertility is negatively correlated with Belebele reading-comprehension accuracy in the raw analysis, but the relationship becomes small and positive after controlling for resource level. Output Quality	mixed	Per-language reading-comprehension accuracy on the Belebele benchmark.	Reading fidelity high Study strength medium	n=13 raw r=−0.61, 95% CI [−0.86, −0.03]; partial r=0.25 0.18
The paper does not establish an independent causal effect of fertility on downstream reading-comprehension accuracy because resource level substantially explains the apparent raw relationship. Output Quality	null_result	Independent relationship between tokenizer fertility and reading-comprehension accuracy.	Reading fidelity high Study strength medium	n=13 partial r=0.25 after controlling for resource level 0.18
For identical content under cl100k_base, an API conversation costing one unit for an English user would cost roughly thirteen units for a Malayalam user when priced per token. Consumer Welfare	negative	Relative token-based API cost for equivalent content across languages.	Reading fidelity high Study strength medium	n=997 roughly 13 units versus 1 unit 0.18