TokenPowerBench lets operators quantify how inference choices change energy and operating costs: across Llama, Falcon, Qwen and Mistral models (1B–405B) it attributes joules per token to prefill vs decode and shows how batch size, context length, parallelism and quantization affect efficiency, all using software-based measurements rather than external meters.

Main Finding

TokenPowerBench is an open-source, lightweight benchmark and measurement pipeline that quantifies LLM inference energy use at token granularity. It couples phase-aligned telemetry (prefill vs decode) with component-level power sampling (GPU/CPU/DRAM/node) and a declarative experiment interface to show how model choice, inference engine, batching, context length, parallelism, and quantization drive joules-per-token and operating cost. Key empirical results: energy per token grows super-linearly with model size; Mixture-of-Experts (MoE) models can deliver large accuracy-per-joule gains (2–3× token energy savings over dense models of similar quality); and inference-engine and configuration choices (e.g., TensorRT-LLM/vLLM vs DeepSpeed/Transformers, batch size, quantization) materially change energy profiles.

Key Points

• Motivation: Inference (not training) accounts for the vast majority of lifecycle energy for deployed LLMs; yet prior benchmarks poorly support energy-normalized, LLM-specific, phase-aware measurements.
• What TokenPowerBench provides:
  • Declarative configuration: model pool, prompt datasets, inference-engine selector, hardware/parallelism/config sweeps.
  • Multi-level telemetry without mandatory external meters: NVML/DCGM for GPU, Intel RAPL for CPU/DRAM, IPMI/PDU for node-level, optional rack/facility inputs.
  • Phase-aligned sampling: tags each timestamped sample with prefill vs decode to attribute energy precisely.
  • Normalized metrics: Joules/token, Joules/response, energy-delay product, power imbalance, cost and CO2 conversions given price/carbon factors.
  • Support for single-node multi-GPU and multi-node distributed inference (Ray) and common engines: vLLM, TensorRT-LLM, DeepSpeed, Transformers.
• Representative empirical findings (H100 cluster experiments):
  • Energy per token increases faster than parameter count (super-linear scaling). Example: LLaMA-3 from 1B→70B increased energy per token by ~7.3× while parameters increased 70×.
  • MoE (Mixtral-8×7B) achieved similar quality to much larger dense models while consuming roughly the same energy as an 8B dense model (2–3× lower token energy than dense equivalents of similar emergent accuracy).
  • Engine effects: TensorRT-LLM and vLLM draw more prefill-stage power but can reduce energy per token by ~25–40% relative to Transformers due to runtime optimizations. DeepSpeed typically sits between extremes.
  • Spatial breakdown: GPUs usually account for >60% of total energy; fans contribute only a few percent.
  • Configuration effects: batch size, context length, parallelism strategy (TP vs PP), and quantization substantially change per-token energy and its stage distribution.
• Accessibility: TokenPowerBench intentionally avoids dependence on costly, bespoke meters, enabling reproducible energy benchmarking across institutional testbeds.

Data & Methods

• Hardware used for evaluation:
  • 8-node cluster, each node with 4× NVIDIA H100 (94 GB), 2× Intel Xeon Gold 6426Y (16 cores/32 threads), 512 GB RAM.
• Models benchmarked:
  • Wide coverage: LLaMA-3 family (1B, 3B, 8B, 70B, 405B), Mistral variants (7B, 24B, MoE mixes), Qwen (8B, 32B, 480B), Falcon (7B, 10B, 180B), and others (15+ models total).
• Datasets:
  • Alpaca (short chat-style prompts, ~52k prompts) and LongBench (longer contexts up to 10k tokens). Custom inputs supported.
• Inference engines and setup:
  • vLLM, TensorRT-LLM, DeepSpeed-Inference, Transformers. Distributed runs use Ray for multi-node model partitioning.
• Measurement stack:
  • GPU: NVML/DCGM; CPU/DRAM: Intel RAPL; node: IPMI/PDU where available. All streams timestamped and aligned.
  • Each power sample tagged with the active inference phase (prefill or decode). Energy computed by integrating component traces over phase intervals.
• Metrics:
  • Joules per token (overall and phase-specific), energy per response, energy per second, peak power, energy-delay product, plus cost/tCO2 conversions if electricity price and carbon intensity provided.
• Evaluation approach:
  • Parameter sweeps over batch size, context length, quantization levels, and parallelism strategies; cross-engine and cross-model comparisons with plotted summaries (example figures shown for prefill vs decode energy and context-length effects).

Implications for AI Economics

• Inference is the dominant ongoing OPEX driver for LLM services: systematic, phase-aware joules-per-token measurement is critical for accurate cost and carbon forecasting.
• Marginal cost and scalability:
  • Super-linear energy growth with model size implies that marginal cost per token rises quickly as models scale; firms must weigh quality gains against sharply increasing operating costs.
• Architectural and software choices matter economically:
  • MoE models (sparse activation) can materially reduce operating electricity cost and carbon per unit of model quality—valuable for cost-sensitive or sustainability-constrained deployments.
  • Investing in optimized inference engines (TensorRT-LLM, vLLM, DeepSpeed, quantization toolchains) and efficient runtime configurations can yield 25–40% energy savings per token or more—translating to substantial OPEX reductions at scale.
• Operational policy levers:
  • Batch sizing, request aggregation, and limiting unnecessary context length directly reduce joules/token and should be part of service-level design and pricing strategies.
  • Quantization (e.g., FP16→FP8 or integer formats) can lower memory footprint and compute energy; cost-benefit analysis should use measured joules/token and any quality/latency trade-offs.
• Procurement, pricing, and capacity planning:
  • TokenPowerBench enables institutions to translate hardware/software configurations into actionable cost-per-token estimates (and CO2 estimates), informing hardware procurement, instance pricing, and run-rate forecasts.
  • Facility- and rack-level inputs (when available) let operators include non-accelerator costs in unit economics (e.g., networking, cooling).
• Accessibility and reproducibility:
  • A no-special-meter design lowers the barrier for universities, labs, and cloud operators to produce comparable OPEX/carbon estimates—improving market transparency and enabling benchmarking-driven efficiency improvements.
• Recommendations for economists and operators:
  • Use phase-aware Joules/token in financial models rather than coarse FLOPS or peak-power proxies.
  • Include non-linear scaling of energy with model size in long-term cost projections.
  • Evaluate MoE and quantized models as potential avenues for better quality-per-dollar and quality-per-CO2.
  • Account for engine/runtime choice and expected request patterns (context length, batchability) in pricing and capacity decisions.

Limitations noted by authors and to consider when applying results: - Primary experiments on an H100 cluster; absolute numbers and relative rankings may vary across GPU generations, cloud instance types, and system designs. - Sensor-based telemetry (NVML, RAPL, IPMI) is convenient but can differ in precision from external high-accuracy meters; consider facility-level metering for absolute billing-grade accounting. - Real-world traffic heterogeneity (burstiness, mix of short/long prompts, concurrent workloads) requires workload-aware measurement for precise operational forecasts.

Overall, TokenPowerBench gives AI economists and operators a practical toolkit and empirical basis to convert LLM design and deployment choices into energy, cost, and carbon estimates—enabling more informed trade-offs between model capability and operating expense.