Training LLMs during renewable curtailment windows can sharply reduce operational emissions: a three-site prototype cut emissions to about 5–12% of single-site training while maintaining model quality, suggesting cheaper, cleaner pretraining is feasible if curtailment access and scheduling are available.

Main Finding

Training full-parameter LLMs opportunistically during renewable curtailment windows is operationally feasible: a reactive, elastic federated system can preserve model quality while shifting almost all energy use into low-carbon windows and cutting operational emissions to roughly 5–12% of single-site baselines.

Key Points

• Prototype system trains a 561M-parameter transformer (nanochat d20) on 12.8B tokens across geographically distributed clusters that are only active during regional curtailment windows.
• Curtailment signal: WattTime marginal operating emissions rate; a region is considered "curtailed" when marginal emissions < 100 gCO2/kWh.
• Execution modes:
  • 0 active sites: suspend training.
  • 1 active site: continue uninterrupted local training (no sync overhead).
  • ≥2 active sites: federated mode with periodic work-weighted FedAvg aggregation.
• Elastic membership and hysteresis: provisioning triggered after τ↑ (used in experiment τ↑ = 10 s); deprovisioning after τ↓ (used τ↓ = 10 min) to avoid oscillations from short signal fluctuations.
• Round sizing: wall-clock rounds (Δround = 600 s / 10 min in prototype) rather than fixed local step counts to handle heterogeneous join/leave timing.
• Implementation: Exalsius control plane for provisioning, Flower federated-learning framework with a custom Kubernetes operator, gRPC communication, Redis pub/sub for control.
• Aggregation and provisioning overhead: ~115 s non-training per aggregation round (≈60 s serialize/comm + ≈55 s DDP setup/teardown). At Δround = 600 s this gives about 80% compute utilization.
• Data management: pre-tokenized fixed-size shards, coordinator-maintained progress vector p, round-robin (progress-weighted) shard assignment; sites return updated model and shard progress.

Data & Methods

• Model and data:
  • Model: nanochat d20 (20-layer transformer, 561M parameters).
  • Training corpus: FineWebEdu-100B subset, 12.8B tokens.
• Infrastructure:
  • Three geographically distributed GPU clusters; each site: 4 × NVIDIA A100.
  • Regions selected for frequent curtailment: CA ISO Northern, SPP North Texas, South Australia, Germany (Germany had no curtailment during the trace; effective run used three sites).
• Curtailment signal & replay:
  • WattTime marginal emissions traces starting 2026-01-11 17:00 UTC.
  • Controlled replay via Vessim to reproduce site availability over time.
• Federated aggregation:
  • Work-weighted FedAvg: global θ = Σs∈A bs θs / Σk∈A bk, where bs = batches processed by site s in the round.
• Evaluation metrics reported:
  • Wall-clock runtime, best train perplexity, total energy consumption (kWh), fraction of energy consumed during curtailment, and operational emissions computed by multiplying site energy by WattTime marginal emissions rates.
• Key quantitative results (prototype run):
  • Runtime: centralized single-site 17.8 h; 2-site continuous FL 11.1 h; curtailment-aware 14.6 h.
  • Best train perplexity (reported in table): centralized 14.7; 2-site FL 15.2; curtailment-aware 14.6.
  • Energy consumption: centralized 36.0 kWh; 2-site FL 36.1 kWh; curtailment-aware 37.7 kWh (slight increase due to provisioning/wall-clock overhead).
  • Curtailment utilization: curtailment-aware run used 97% of its energy during curtailment windows.
  • Operational emissions: single-region runs 11.4–27.1 kgCO2; curtailment-aware run 1.38 kgCO2 (≈5–12% of single-region baselines).

Implications for AI Economics

• Direct operational cost and carbon benefits:
  • Shifting training into curtailment windows leverages otherwise-wasted, low-cost renewable energy; this can substantially reduce marginal emissions and electricity costs per kWh used for training.
  • Prototype shows large proportional emissions reductions (≈88–95%), though absolute energy consumption for the job did not materially decrease.
• Workload design and market opportunities:
  • AI workloads with flexible timing and large compute budgets (pretraining, periodic refreshes, large-scale generation tasks, some RL workloads) are prime candidates to monetize curtailment windows.
  • Providers can offer discounted, low-carbon compute products tied to curtailment availability; this creates a market differentiation for "green compute".
• Incentives and location decisions:
  • Economic incentives rise for colocating datacenters near regions with frequent curtailment (or building flexible capacity there) to capture cheap surplus renewables.
  • Conversely, substantial elastic demand during curtailment could reduce curtailment levels and affect local prices; the interaction with electricity markets needs careful modeling.
• Cost–benefit tradeoffs & operational constraints:
  • Provisioning and synchronization overheads (state transfer, optimizer warm-up, DDP setup) impose nontrivial fixed costs; the economic value of curtailment-aware scheduling depends on window length distribution, forecast accuracy, and provisioning latency.
  • Waiting for curtailment can increase wall-clock time; economic analyses must trade energy-price/carbon savings against delayed time-to-result (opportunity cost).
• Policy and market considerations:
  • If demand-side flexibility like this becomes widespread, grid operators and regulators may need to adapt market designs (e.g., time-varying tariffs, incentives for flexible loads, treatment of curtailed energy contracts).
  • Measurement/verification standards (e.g., what counts as "curtailed" or "marginally low-carbon") are important for carbon accounting and product claims.
• Future research & modelling needs for AI economics:
  • Macro-scale models quantifying how aggregated flexible AI demand affects curtailment rates, local prices, and generator dispatch.
  • Detailed cost models incorporating provisioning costs, bandwidth, staff/ops, model transfer, and potential revenue from selling flexibility into capacity or ancillary-service markets.
  • Optimization frameworks that jointly schedule compute across time/space given probabilistic curtailment forecasts, spot prices, and SLA constraints.
  • Assessment of scaling behavior: larger models (≥7B) incur higher synchronization overhead—economic feasibility may change with model size and communication costs.

Limitations to keep in mind - Prototype on a small (561M) model and three sites—results may not generalize to very large models or many more, geographically dispersed sites where synchronization and state transfer costs grow. - The experiment used selected regions with frequent curtailment (an optimistic case). Real-world benefits hinge on geography, seasonality, and forecast quality. - Curtailment utilization reduces marginal emissions but does not reduce total energy consumed for the training job; net system-level emissions impacts depend on wider grid responses to flexible demand.

Repository and artifacts - Prototype and scripts: https://github.com/exalsius/curtail-llm (authors’ implementation and replay setup).