1 cumulative citations
View corpus contextTraining language models longer can erode energy efficiency: for a 1.1B TinyLlama, increasing token counts from 500K to 2M produced inconsistent performance gains but a clear, statistically significant drop in energy-aware parameter efficiency, highlighting hidden cost trade-offs when scaling token budgets.
Revisiting Training Scale: An Empirical Study of Token Count, Power Consumption, and Parameter Efficiency
arxiv
quasi_experimental
medium evidence
7/10 relevance
Full text
usable extracted full text
Source
PDF
In controlled repeated-measures training of a 1.1B TinyLlama, increasing total training tokens produced inconsistent or diminishing conventional performance gains but caused a strictly monotonic decline in energy-aware parameter efficiency, with token count having a strong, significant effect.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Research in machine learning has questioned whether increases in training token counts reliably produce proportional performance gains in large language models. Building on prior work introducing an energy-aware parameter efficiency metric, this study empirically examines the effects of increasing training token counts under fixed hardware and training conditions. The significance of this work lies in the explicit integration of power consumption and execution duration, as reflected by the power sampling frequency, into token-scale analysis. This addresses a gap in prior studies emphasizing performance outcomes while underrepresenting computational and energy costs. Using a repeated-measures experimental design on a constant GPU instance with an identical model architecture, optimizer settings, and epoch counts, a 1.1-billion-parameter TinyLlama model was trained at three token counts (500K, 1M, and 2M). While conventional performance metrics exhibited inconsistent or diminishing returns across token scales, the inclusion of power consumption and execution duration revealed a strictly monotonic decline in training efficiency as token count increased. Repeated-measures ANOVA demonstrated a strong effect of token count on parameter efficiency, with all pairwise comparisons remaining significant following Bonferroni correction. These findings indicate that increases in training token counts may be energetically inefficient even when marginal performance improvements are observed, underscoring the importance of efficiency-aware evaluation in large language model training.
Summary
Main Finding
Increasing training token counts (500K → 1M → 2M) for a fixed 1.1B TinyLlama model on a single-A10G GPU produced only modest or inconsistent improvements in model performance (inverse perplexity) while producing a strictly monotonic decline in energy-aware parameter efficiency. Measured GPU power and runtime increased with token count, and repeated-measures ANOVA showed very large, statistically significant effects of token count on both parameter efficiency (F(2,98)=24,268.23, p<.001, η2g=.997) and RMS GPU power (F(2,98)=3,143.21, p<.001, η2g=.969). All pairwise comparisons between token conditions were significant after Bonferroni correction.
Key Points
- Experimental design: within-subject repeated-measures; 3 token conditions (500K, 1M, 2M); 50 independent trials per condition; total N=150 trials.
- Model & training: TinyLlama 1.1B (~1.1B parameters); AdamW, lr=1e-6, FP16 mixed precision, gradient clipping 1.0; 3 epochs, batch size 1; TinyStories corpus (JSONL), max seq length 64.
- Energy measurement: NVIDIA Management Library power samples collected every 60s (plus immediate startup sample); per-trial power aggregated using RMS(W).
- Efficiency metric (energy-aware parameter efficiency, PE):
- PE ∝ invPPL · TFLOPSmeasured · TTscale / (MSparams · TTtokens · RMS(W))
- invPPL = 1 / exp(Leval) (inverse perplexity as performance signal)
- TFLOPS used as a normalization constant; RMS(W) converts power into the denominator (energy cost).
- Results summary:
- Parameter efficiency declined monotonically with token count; ANOVA: F(2,98)=24,268.23, p<.001, η2g=.997; all pairwise t-tests significant (e.g., 500K vs 2M: t≈220.82).
- RMS GPU power increased monotonically; ANOVA: F(2,98)=3,143.21, p<.001, η2g=.969; all pairwise differences significant (e.g., 500K vs 2M: t≈−84.16).
- Inverse perplexity (performance) showed modest changes, indicating the efficiency collapse was driven primarily by higher energy/runtime rather than large performance gains.
- Assumptions/testing:
- PE values: normality and sphericity satisfied.
- Power values: non-normal for 1M and 2M; Mauchly’s test indicated violated sphericity (W=0.422, χ2(2)=41.43, p<.001), but Greenhouse–Geisser–corrected ANOVA remained significant.
Data & Methods
- Hardware/software: AWS SageMaker ml.g5.xlarge with a single NVIDIA A10G GPU; power samples via NVIDIA Management Library; experiments run under identical software and hyperparameter settings to isolate token count.
- Dataset & tokenization: TinyStories dataset; token counts targeted as true non-padding tokens (500K / 1M / 2M); fixed-length padding/truncation to 64 tokens; batch size 1; random sample order.
- Trial protocol: each trial reinitialized model and dataset; no curriculum learning, adaptive scheduling, or early stopping; three epochs per run; fallback and numerical-stability rules for evaluation (inverse perplexity computed from stable post-training loss; fallback to a valid training batch if necessary).
- Aggregation & metrics:
- RMS(W) computed over all power samples per trial.
- Total tokens processed = token target × epoch count (normalized to millions).
- PE combines inverse perplexity with TFLOPS normalization and divides by (params × tokens × RMS power).
- Statistical analysis: Python + Pingouin; one-factor repeated-measures ANOVAs on PE and RMS power; Shapiro–Wilk for normality; Mauchly’s test for sphericity; Bonferroni-corrected pairwise tests; generalized eta squared reported.
Implications for AI Economics
- Marginal cost vs. marginal performance: Under fixed model size and hardware, raising token budgets can produce diminishing or negligible performance gains while substantially increasing energy consumption and runtime—implying poor marginal returns per token for constrained deployments.
- Token budgeting and procurement: Organizations should treat token count as a budgeted, cost-bearing decision (not just a "free" way to improve models). Procurement/price models ought to account for per-token energy/runtime cost as well as model compute.
- Carbon and operational cost accounting: Measured per-trial RMS power allows more accurate energy and carbon accounting for training jobs. Energy-aware efficiency metrics (like PE per watt) should be reported alongside loss/perplexity in research and procurement cases to inform sustainability and TCO (total cost of ownership) assessments.
- Model scaling strategies: For constrained budgets or sustainability goals, it may be more cost-effective to reallocate resources (e.g., increase model size vs. increasing token count, apply data-centric methods, or use pruning/curation) rather than linearly increasing tokens.
- Pricing and marketplace effects: Cloud pricing and charging models that bill by GPU-hour should consider effective energy usage and efficiency; buyers may prefer metrics that translate tokens → energy → cost to compare strategies.
- Policy and transparency: Recommends standard practice—report raw energy/runtimes and energy-aware efficiency metrics in experimental papers and benchmarking to allow economically and environmentally informed decisions by smaller labs and policy makers.
- Research directions relevant to economics: Extend energy-aware efficiency measures to multi-GPU and distributed settings, larger models, and different accelerator types to inform cross-platform cost modeling and investment decisions (e.g., whether to invest in more efficient hardware or in dataset curation).
Limitations to consider when applying these findings: single model architecture (1.1B TinyLlama), single GPU class (A10G), small batch size and short sequences—results may not generalize quantitatively to larger models, different hardware, distributed training, or other datasets. Nonetheless, the qualitative lesson—token increases can be energetically inefficient even when performance gains appear—has direct relevance for economic and sustainability considerations in ML practice.
Assessment
Paper Typequasi_experimental
Evidence Strengthmedium — Strong internal control over hardware and training settings and an appropriate within-subjects statistical test support a causal link between token count and the reported efficiency metric; however, the experiment is limited to a single model size/architecture, one hardware instance, a narrow range of token counts, and the number of independent replication runs/seeds is not reported, limiting robustness and external validity.
Methods Rigormedium — Good use of a repeated-measures design, clear control of confounders (same GPU, hyperparameters), formal ANOVA and multiple-comparisons correction, and explicit measurement of power sampling; shortcomings include a small experimental scope (three token levels), unclear replication/seed strategy, limited reporting detail on measurement error and dataset composition, and lack of tests across other model scales, optimizers, or distributed settings.
SampleA 1.1-billion-parameter TinyLlama model trained on a constant GPU instance at three total token counts (500,000; 1,000,000; 2,000,000) with identical architecture, optimizer settings and epoch counts; power consumption and execution duration were sampled at a given frequency to compute an energy-aware parameter-efficiency metric; repeated-measures (within-instance) comparisons were performed (number of independent runs/seeds not specified).
Themesproductivity adoption
IdentificationWithin-instance repeated-measures experiment: the authors train the same 1.1B-parameter TinyLlama architecture on a single, constant GPU instance while holding optimizer, architecture, epoch counts, and other hyperparameters fixed and varying only the total training token count (500K, 1M, 2M); inference on causal effect uses repeated-measures ANOVA and pairwise tests (Bonferroni-corrected) on an energy-aware parameter-efficiency metric that integrates power draw and execution duration.
GeneralizabilitySingle model size and architecture (1.1B TinyLlama) — may not generalize to larger or different architectures, Single hardware/GPU instance — power and timing behavior may differ across GPUs, data-center infrastructure, and distributed training, Small and relatively low token counts compared to large-scale pretraining regimes — scaling behavior may change at much larger token budgets, Unclear dataset composition and limited reporting of random seeds/repeats — possible sensitivity to initialization or data sampling, Findings tied to the specific optimizer/settings used — other optimizers, batch sizes, or learning-rate schedules may alter efficiency gains
Claims (6)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| A 1.1-billion-parameter TinyLlama model was trained at three token counts (500K, 1M, and 2M) using a repeated-measures experimental design on a constant GPU instance with identical model architecture, optimizer settings, and epoch counts. Other | null_result | experimental training protocol (token counts and hardware/control conditions) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Conventional performance metrics exhibited inconsistent or diminishing returns across token scales (500K, 1M, 2M). Output Quality | mixed | conventional performance metrics (unspecified in excerpt) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| When power consumption and execution duration were included, training efficiency (energy-aware parameter efficiency) declined strictly monotonically as token count increased. Organizational Efficiency | negative | energy-aware parameter efficiency (training efficiency incorporating power consumption and execution duration) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| A repeated-measures ANOVA demonstrated a strong effect of token count on parameter efficiency, with all pairwise comparisons remaining significant following Bonferroni correction. Organizational Efficiency | negative | parameter efficiency |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Increases in training token counts may be energetically inefficient even when marginal performance improvements are observed, highlighting the importance of efficiency-aware evaluation in large language model training. Organizational Efficiency | negative | energetic efficiency of training vs. performance gains |
Reading fidelity
high
Study strength
speculative
|
not reported
|
| The study explicitly integrates power consumption and execution duration (as reflected by the power sampling frequency) into token-scale analysis, addressing a gap in prior studies that emphasized performance outcomes while underrepresenting computational and energy costs. Other | positive | inclusion of power sampling frequency and execution duration in token-scale analysis |
Reading fidelity
high
Study strength
low
|
not reported
|