0 cumulative citations
View corpus contextSwitching to byte-level distillation with an End-Of-Token conversion lets small models outperform token-distilled peers as compute increases, lifting asymptotic task accuracy by several percentage points while cutting logit storage and training-data needs.
Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models
Exact byte-level distillation using an End-Of-Token logit conversion yields byte-based 1B models that eventually surpass token-distilled 1B models in asymptotic downstream accuracy while being substantially more data- and storage-efficient.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Small models are made more capable through distillation from a larger one that shares their tokenization scheme. However, do distilled byte and token models behave similarly in terms of scaling trends as compute and data increases? To enable this comparison, we introduce two variants to efficiently convert token logits to Byte Logits: 1) approximate: Marginalize-It, and 2) exact: End-Of-Token. We then present the first large scale study of overtraining decoder-only dense transformer models varying two dimensions simultaneously: the tokenization scheme (Tokens, Bytes, Bytes w/ eot) and the training objective (Distillation vs. Cross-Entropy), sweeping layer-parameter-matched models with roughly 1 billion parameters up to 1 trillion bytes of data. Across eight benchmarks spanning three categories: Multiple Choice QA, Language Generation, and Machine Translation, we find that Token-1B models outperform byte models (End-Of-Token-1B and Bytes-1B) in the low-FLOP regime but eventually plateau; byte models start worse yet surpass Token-1B models with more compute, reaching a higher downstream task performance ceiling. Extrapolating the average top-1 error vs. validation BPB scaling laws predicts that, asymptotically, distilled End-Of-Token-1B outperforms distilled Token-1B by up to 4%. They are also far more data efficient, matching the performance of distilled Token-1B using only one-sixth of the training data. Moreover, by operating over a small vocabulary of 256 bytes instead of on the order of 100K tokens, they circumvent the need for top-k truncation during logit dumping, while also reducing logit storage costs to roughly one-fifth. Finally, our downstream performance scaling laws predict that our distilled End-Of-Token-1B models asymptotically surpass the Llama 3.2-1B, Gemma-3-1B-pt, and Gemma 2B models on averaged downstream tasks by up to 6.5%, 8.1%, and 2.1%, respectively.
Summary
Main Finding
Distilling 1B-ish layer-parameter transformer students to operate over bytes (≈256-vocab) rather than large token vocabularies yields a different scaling trajectory: tokenized students are stronger at low compute, but byte-based students—especially when trained with the exact End-Of-Token (EoT) logit-conversion—improve more with added compute and reach a higher asymptotic downstream-task ceiling. EoT-distilled byte models are substantially more data- and storage-efficient (match token-distilled performance with ≈1/6 the data and reduce dumped-logit storage to ≈1/5) and are predicted to outperform several open-weight 1B–2B baselines by multiple percent on averaged downstream benchmarks.
Key Points
- Two single-pass token→byte logit-conversion methods:
- Marginalize-It (approximate): marginalizes token probabilities across byte prefixes, re-normalizes; fast but approximate for later bytes.
- End-Of-Token (exact): append an
byte symbol and marginalize; preserves the teacher distribution exactly in one forward pass.
- Experimental sweep:
- Layer-parameter-matched decoder-only transformers (~1.28B layer params; Token-1B has ~1.81B total due to large vocab).
- Tokenization variants: Tokens (BPE), Bytes, Bytes w/
. - Objectives: supervised (cross-entropy) and distillation (teacher: Llama 3-8B).
- Data/compute range: many scales up to ~1 trillion bytes (overtraining regime), multiple learning rates swept.
- Benchmarks: 8 tasks across Multiple-Choice QA (ARC-Easy/Challenge, HellaSwag, PIQA), Language Generation (MBPP, Natural Questions), and Machine Translation (FLORES).
- Main empirical behaviors:
- Token-1B models perform best at low-FLOP regimes but plateau early.
- Byte models (Bytes-1B and End-Of-Token-1B) start worse but improve faster and surpass tokens with higher compute, achieving a higher asymptotic downstream performance.
- EoT-distilled models achieve the best asymptotic performance among studied variants.
- Marginalize-It, despite being approximate, works well in practice and is a strong baseline.
- Quantitative highlights (LR = 4e-3 fits to BPB vs FLOPs power law y = b x^a + c):
- Asymptotic BPB (c) values (lower is better): Token Supervised 0.9568; Token Distilled 0.9407; Bytes Supervised 0.8967; Marginalize-It Distilled 0.9016; Bytes w/
Supervised 0.8891; End-Of-Token Distilled 0.8983. - EoT-distilled is predicted to outperform Token-distilled by up to ≈4.0% on averaged downstream accuracy asymptotically.
- EoT-distilled matches distilled Token-1B using only ≈1/6 of the training data (data efficiency claim).
- Adding
yields ≈30.94% additional compute-per-unit-of-data (one every ~4.5 bytes) but improves asymptotic performance. - Logit-storage: byte models remove need for top-k truncation in offline distillation and cut logit storage to ~1/5 of tokenized equivalents.
- Asymptotic comparisons to open models: predicted gains vs Llama-3.2-1B, Gemma-3-1B-pt, Gemma 2B up to ≈6.5%, 8.1%, and 2.1% (averaged downstream tasks), respectively.
- Asymptotic BPB (c) values (lower is better): Token Supervised 0.9568; Token Distilled 0.9407; Bytes Supervised 0.8967; Marginalize-It Distilled 0.9016; Bytes w/
- Methodological artifact: some conclusions depend on extrapolating fitted power laws (BPB vs FLOPs, downstream error vs BPB); extrapolation uncertainty acknowledged.
Data & Methods
- Architecture and scale:
- Decoder-only transformers (Vaswani-style), layer-parameter-matched students ≈1.28B layers; Token-1B listed as 1.81B total params due to large vocab embedding.
- Vocab sizes: token BPE ~128k; byte vocab 256 (+4 special) = 260; byte w/
= 261.
- Logit-conversion innovations:
- Single-pass conversion from teacher token logits to student byte logits:
- Marginalize-It: aggregate token probabilities by byte-prefixes and re-normalize conditioned on observed prefix—exact for first byte, approximate for later bytes.
- End-Of-Token: append
symbol to token decoding and byte vocab; marginalize so residual continuation mass is captured exactly in —exact in one pass.
- Training data and teacher:
- Students trained on the Llama-2 training mixture (same corpus used across scenarios, designed so token and byte logits occupy comparable memory for equal raw-data).
- Distillation teacher: Llama 3-8B.
- Training procedure:
- AdamW optimizer; LR sweep {1e-3, 4e-3, 8e-3}; 10% warmup; both cross-entropy and distillation losses applied depending on experiment.
- Extensive overtraining across many data scales (data- and FLOP-level sweeps) up to ~1T bytes to fit scaling laws.
- Evaluation:
- Validation metric for pretraining: Bits-Per-Byte (BPB) on held-out validation set.
- Downstream: accuracy/metrics across the 8 benchmarks listed above.
- Fitted power-law relationships: BPB vs training FLOPs and downstream error vs BPB; used to extrapolate asymptotic behavior and to construct "Feather Plots" connecting iso-FLOP points.
Implications for AI Economics
- Deployment vs pretraining trade-offs:
- Byte-based students have much smaller vocabularies (≈256) and hence smaller final-model embedding/storage footprints and cheaper logit storage for offline distillation pipelines—reduces downstream storage and transfer costs (claimed ≈80% reduction in dumped-logits storage).
- Byte students are more data-efficient: achieving equivalent performance with substantially less pretraining data (reported ≈1/6), implying large potential savings in data-collection and compute cost for pretraining when data is the binding constraint.
- However, EoT-style byte training incurs extra compute per raw data unit (~30.94%) due to inserted
tokens; that increases pretraining FLOP cost per byte—but yields higher asymptotic performance. Practitioners must weigh the extra compute cost against data and storage savings and the desired performance target.
- Distillation pipeline economics:
- Single-pass token→byte conversion methods (especially EoT exact conversion) remove the need for multiple teacher forward passes per sequence, lowering teacher-inference costs during distillation compared to previously proposed multi-pass methods.
- Avoiding top-k truncation simplifies offline distillation and reduces the complexity/cost of logit dumping pipelines.
- Model choice as a function of compute budget:
- If constrained to low inference/serve compute budgets or low pretraining FLOPs, tokenized students give better returns early (lower FLOP regimes).
- For operators planning large pretraining compute budgets or long-term amortized deployments, byte-based distilled models have better upside: higher asymptotic performance and ongoing gains with more compute.
- Comparative advantage vs open-weight models:
- The predicted asymptotic gains over popular open-weight 1B/2B models suggest byte-distilled students could be a cost-effective way to outperform those baselines without dramatically increasing parameter counts—valuable for organizations with limited ability to train very large models.
- Practical economic takeaways:
- For practitioners: choose token students for fast, low-compute wins; choose byte (EoT) students if planning for heavy pretraining investments or when data and logit-storage costs dominate.
- For platform and tool builders: supporting single-pass token→byte conversion and EoT-style tokenization in distillation tooling can reduce both engineering and runtime costs.
- For policy/economic modeling: scaling-law extrapolations should be used cautiously; capital allocation decisions (compute vs data vs storage) should incorporate the observed trade-offs (compute per data unit, data-efficiency, storage savings).
Caveats & limitations - Extrapolations rely on fitted power laws and carry uncertainty outside observed regimes. - Models are layer-parameter-matched but Token-1B has more total parameters due to embedding sizes—this asymmetry complicates direct apples-to-apples interpretation. - Experiments use one teacher family (Llama 3-8B) and one pretraining mixture (Llama-2 mix); behavior could vary with different teachers, architectures, or data mixes. - Benchmarks are a set of 8 tasks; results may differ on other task suites (instruction following, RLHF, long-context tasks, etc.).
If you want, I can: - Produce a concise decision flow for practitioners (when to prefer Token vs Bytes vs EoT given a specific compute/data/storage budget). - Extract the fitted power-law coefficients and plotting-ready numbers for your own cost-modeling.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper reports a large, well-controlled set of ML experiments comparing token vs byte tokenizations and two distillation conversions across many data and FLOP budgets and multiple benchmarks; this provides strong empirical evidence for model-level performance and efficiency differences. However, claims about asymptotic superiority rely on extrapolated power-law fits beyond observed regimes, and results are conditional on architecture, teacher choice, training mixture, and implementation choices (e.g., top-k truncation), which limits causal generality.
Methods Rigorhigh — The authors run layer-parameter-matched controlled experiments, sweep learning rates, evaluate on eight diverse benchmarks, and fit power-law scaling laws with high R^2; they introduce both approximate and exact logit-conversion methods and report storage/compute accounting. Remaining concerns are typical ML-experimental ones: reliance on a single teacher model and pretraining mixture, potential sensitivity to hyperparameters and top-k settings, and extrapolation assumptions.
SampleLayer-parameter-matched decoder-only dense Transformer models (≈1.28B layer parameters; Token-1B reported as 1.81B due to vocab) trained on the Llama-2 pretraining mixture across many data scales up to ~1 trillion bytes; teacher model is Llama 3-8B for distillation; evaluated on eight downstream benchmarks spanning Multiple-Choice QA (ARC-Easy, ARC-Challenge, HellaSwag, PIQA), Language Generation (MBPP, Natural Questions), and Machine Translation (FLORES EN↔DE). Experiments include supervised cross-entropy and two distillation conversions (Marginalize-It approximate and End-Of-Token exact), with FLOPs and logit-storage accounting reported.
Themesproductivity adoption
GeneralizabilityResults reported for decoder-only dense Transformers at ~1B scale; may not transfer to substantially larger or different architectures (e.g., mixture-of-experts, encoder–decoder)., Distillation teacher is Llama 3-8B and training mixture is Llama-2; different teachers or data mixtures could change relative outcomes., Asymptotic claims rely on extrapolated power-law fits beyond observed compute/data regimes and may not hold at very different scales or distributions., Benchmarks are NLP-focused and largely English; cross-lingual or domain-specific tasks might behave differently., Implementation choices (top-k truncation for token logits, exact preprocessing, tokenizers) and hardware/optimization differences could affect reproducibility and deployment trade-offs.
Claims (10)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| The End-Of-Token method converts token logits to byte logits exactly in a single teacher-inference pass by adding an <eot> token and assigning residual probability mass to it. Other | positive | Accuracy of token-to-byte logit conversion |
Reading fidelity
high
Study strength
high
|
not reported
|
| The study evaluates six overtraining configurations combining three tokenization schemes with supervised or distilled training objectives, using approximately 1-billion-parameter decoder-only models and up to 1 trillion bytes of training data. Other | other | Scaling behavior as a function of tokenization, objective, compute, and data |
Reading fidelity
high
Study strength
high
|
n=6
|
| Token-1B models perform better than byte models in the low-compute regime, but byte models improve more steeply with additional compute and eventually surpass Token-1B models. Output Quality | mixed | Average downstream task performance as training compute increases |
Reading fidelity
high
Study strength
medium
|
n=8
|
| At asymptotically high compute, End-Of-Token-1B models have lower validation BPB than Bytes-1B models, which in turn have lower validation BPB than Token-1B models. Output Quality | positive | Asymptotic validation bits per byte |
Reading fidelity
high
Study strength
medium
|
End-Of-Token BPB 0.8983 vs. Bytes BPB 0.9016 vs. Token BPB 0.9407 for distilled models
|
| Distillation improves validation BPB relative to supervised training for both the approximate Marginalize-It byte model and the exact End-Of-Token byte model. Output Quality | positive | Validation bits per byte |
Reading fidelity
high
Study strength
medium
|
BPB improvement of 0.0051 for Marginalize-It and 0.0092 for End-Of-Token relative to corresponding supervised models
|
| Extrapolated downstream-performance scaling laws predict that distilled End-Of-Token-1B models asymptotically outperform distilled Token-1B models by up to 4% on averaged downstream performance. Output Quality | positive | Average top-1 accuracy across downstream tasks |
Reading fidelity
high
Study strength
low
|
n=6
up to 4% improvement
|
| Distilled End-Of-Token-1B models are predicted to match the performance of distilled Token-1B models using only one-sixth of the training data, given sufficient training. Training Effectiveness | positive | Training data required to reach a given downstream performance level |
Reading fidelity
high
Study strength
low
|
one-sixth of the training data
|
| Using a byte vocabulary of approximately 256 values instead of approximately 100,000 tokens eliminates the need for top-k truncation during offline logit dumping and reduces logit storage costs to roughly one-fifth. Organizational Efficiency | positive | Logit storage cost and need for top-k truncation |
Reading fidelity
high
Study strength
medium
|
roughly one-fifth of token-model logit storage costs
|
| Extrapolated scaling laws predict that distilled End-Of-Token-1B models asymptotically outperform Llama 3.2-1B, Gemma-3-1B-pt, and Gemma 2B on averaged downstream tasks by up to 6.5%, 8.1%, and 2.1%, respectively. Output Quality | positive | Average downstream task accuracy |
Reading fidelity
high
Study strength
low
|
n=6
up to 6.5%, 8.1%, and 2.1% improvement
|
| The End-Of-Token method incurs approximately 30.94% more compute per unit of underlying data than the Bytes method because it inserts an <eot> token approximately every 4.5 bytes. Organizational Efficiency | negative | Compute required per unit of training data |
Reading fidelity
high
Study strength
high
|
≈30.94% additional compute per token
|