A new Compute–Data scaling law shows that derived tokens (repeats or paraphrases) become less valuable as models and data availability grow, so spending extra compute instead of curating fresh data delivers diminishing returns and Chinchilla’s compute-only allocation is often suboptimal.

Main Finding

The paper introduces Compute-Data (CD) scaling laws that extend classical Chinchilla-style scaling to the regime where fresh pretraining data is limited. Key to the framework is a token-effectiveness function η(strat; N, TPP, r) (TPP = D/N, r = D′/D) that maps derived tokens (from repetition, paraphrasing, distillation, etc.) to their fresh-token equivalents. CD-scaling shows that the benefit of substituting compute for fresh data diminishes with model size and with data availability, imposes a finite saturation ceiling on effective data, and partitions pretraining into three regimes—compute-bound, data-bound, and model-bound—yielding a joint compute–data Pareto frontier that generally supersedes Chinchilla’s compute-only allocation.

Key Points

• CD-scaling law (compact form): L(N, D, D′) = E + A / N^α + B / (D + η · D′)^β, where η ∈ [0,1] is the strategy-dependent token effectiveness.
• Parametric form for η used in the paper: η = (R / r) · (1 − exp(−r / R)), with r = D′/D and R the saturation ceiling (R = lim_{r→∞} η·r).
• Functional form for the saturation ceiling: R(D, N) = K · (D/N)^ρ · N^σ = K · TPP^ρ · N^σ, so R (and therefore η) depends jointly on model size N and tokens-per-parameter TPP.
• Empirical regime & fitted range:
  • Models: N ∈ {14M, …, 600M}
  • Fresh data D: 30M → 30B tokens (nested subsets)
  • Derived tokens D′: 30M → 120B tokens
  • Two expansion strategies studied: multi-epoch repetition and paraphrasing (SmolLM2-1.7B-Instruct used to produce paraphrases).
• Main empirical findings:
  • η (and thus effective contribution η·D′) decays with r and TPP and with N; η·D′ saturates to a finite R* as r→∞.
  • Larger models saturate sooner and at lower R* (i.e., derive less fresh-equivalent value per derived token).
  • Paraphrasing is relatively more valuable for small models and small/sub-Chinchilla budgets; repetition wins for larger models and higher TPP.
  • Classical Chinchilla (1-epoch on fresh data) is recovered as D′→0; the data-optimal limit is obtained by substituting D(1+R*) for D.
• Fitted Chinchilla (1-epoch) parameters (point estimates, Table 1):
  • E ≈ 1.35, A ≈ 205, B ≈ 16597, α ≈ 0.283, β ≈ 0.435 (95% CIs reported in paper)
• Strategy-specific fitted parameters (log K, ρ, σ, reported in Table 1) indicate different saturation behavior for repetition vs paraphrase; fit RMSE on log loss was small (≈0.02–0.04).
• Practical rules:
  • The oft-cited “4 epochs” heuristic holds only for medium-scale models (~3B) near Chinchilla-optimal TPP; recommended extra epochs decrease as N and TPP grow.
  • For N ≲ ~600M and sub-Chinchilla TPP, paraphrasing can outperform repetition; for very large models (extrapolated N ≥ ~7B) and large D′ budgets, paraphrasing provides little benefit.

Data & Methods

• Architecture & training:
  • OLMo3 architecture, sequence length 4096, batch size 512, AdamW, cosine LR decay.
  • Hyperparameter sweep for LR ∈ {1e-4, 3e-4, 1e-3, 3e-3} and weight decay ∈ {0.1, 0.2, 0.4, 0.8, 1.6} (selected at key sizes then locally tuned).
  • Validation metric: held-out Dolma-3 documents.
• Data:
  • Fresh tokens sampled from Dolma-3 150B corpus; smaller D are nested subsets of larger D.
  • Paraphrases generated with SmolLM2-1.7B-Instruct (up to 16 paraphrases per document; 4-style mixture).
  • Paraphrase generation FLOPs were not counted as training compute (treated as asynchronous preprocessing).
• Experimental sweep:
  • Model sizes 14M → 600M; D 30M → 30B; D′ 30M → 120B; both repetition and paraphrase strategies applied.
• Fitting procedure:
  • First fit the 1-epoch Chinchilla law (E, A, B, α, β).
  • For each run with D′>0, solve for empirical ηemp that reproduces observed loss given 1-epoch constants.
  • Parametrize η via R*(N, TPP) and the exponential form above; jointly fit all parameters using Huber loss on log L.
  • Uncertainty quantified via bootstrap resampling (200 refits) to obtain 95% CIs.
• Validation & ablations:
  • Cross-scale validation: fitting on small models (14M, 30M) and predicting larger models; ablation showed the (D, N)-dependence of R* is important and the exponential form fits better than alternatives.
  • Reported RMSEs on held runs were small, indicating good fit quality within the experimental regime.

Implications for AI Economics

• Joint compute–data optimization: CD-scaling provides a principled way to allocate budget between buying/curating fresh tokens versus spending compute to produce derived tokens (repetition, paraphrase, distillation). Decisions should be made on the joint compute–data Pareto frontier implied by CD-scaling rather than by Chinchilla’s compute-only rule alone.
• Diminishing marginal returns to compute as data scarcity increases: When D is limited, extra compute spent creating/learning from derived tokens yields bounded gains (η·D′ saturates). Economically, this implies a cap on the return to additional compute for a fixed corpus—past a point, buying more compute is inefficient versus acquiring fresh data.
• Value of synthetic data depends on model scale and dataset abundance:
  • For smaller models and low TPP, synthetic augmentation (paraphrasing) can be cost-effective. Renting/using augmentation services or investing in paraphrase pipelines can be economically justified for small-to-medium-sized training targets.
  • For large models and/or when TPP is high, synthetic augmentation (and multi-epoch repetition) yields rapidly diminishing returns; investment should shift to fresh data acquisition (if possible) or model scaling adjustments.
• Pricing and markets for data-augmentation tools and curated datasets:
  • The presence of a finite saturation ceiling (R) suggests an upper bound on the market value of augmentation for given (N, D). Suppliers of synthetic-data services should focus on customer segments (model sizes and TPP regimes) where η and R are materially large.
• Cost accounting: the paper excludes paraphrase-generation compute from training compute. From an economic perspective, total-cost analyses must include the cost of generating synthetic data (compute and human curation) to accurately compare alternatives (buy fresh data vs generate paraphrases vs buy compute).
• Strategic implications for research and deployment:
  • For a given firm deciding whether to scale models or expand datasets, CD-scaling supplies a quantitative decision rule: determine whether you are compute-, data-, or model-bound and allocate capital accordingly.
  • The finite effective-data ceiling makes long single-corpus overtraining less attractive; firms should prefer strategies that increase fresh-data diversity (curation, acquisition, multi-source mixing) if operating near or beyond R*.
• Limitations and cautions for economic use:
  • Empirical fits are based on Dolma-3 and models up to 600M parameters; extrapolation to multi-billion-parameter models (where some policy/practice conclusions are stated) relies on model assumptions and/or external evidence. Economic decisions for very large models should treat extrapolations cautiously and incorporate the (nontrivial) costs of synthetic-data generation.
  • η depends on the expansion strategy and on the quality of the augmentation model (paraphrase model here was SmolLM2-1.7B-Instruct); different generation models or higher-quality synthetic data may shift R* and the cost-benefit balance.
  • The framework assumes fresh-corpus composition is fixed; mixing or changing the fresh corpus composition (domain diversification) can alter outcomes and may be a higher-leverage economic lever than derived-token generation.

Summary takeaway for AI economists / managers: Use CD-scaling to decide whether to (a) buy/curate more fresh tokens, (b) invest compute into model scaling, or (c) spend compute (and monetary cost) on generating derived tokens. The token-effectiveness η and its saturation R* quantify where each dollar of compute or data yields most reduction in loss; because η falls with model size and TPP, many practical settings will favor data acquisition or model resizing over just spending more compute on derived tokens.