Matrix-aware optimizers improve large-model training in many settings, but AdamW remains the robust baseline; optimizer choice must be evaluated compositionally across budgets (tokens, FLOPs, time, memory) because rankings flip with scale, batch size, tuning budget and system constraints.

Main Finding

Matrix-aware optimizers and richer temporal/representation strategies are a genuine algorithmic advance, but they do not provide a context-free replacement for AdamW. Optimizer design has become compositional: four largely independent axes (temporal estimation, update geometry, horizon management, and representation/systems) interact with scale, batch size, parameter class, memory, and the evaluation metric (tokens, FLOPs, wall time, memory). Practical optimizer choice therefore depends on the resource budget and the training protocol; a stricter, multi-budget evaluation protocol is required to make reproducible economic claims about training cost and speedups.

Key Points

• Four orthogonal axes organize modern optimizers:
  • Temporal estimation (how gradient history is aggregated; e.g., AdEMAMix, MARS, spike resets).
  • Update geometry (coordinate, matrix/whitening, spectral/norm-aware, or curvature-based choices; e.g., Adam, Shampoo, Muon).
  • Horizon management (learning-rate schedules, schedule-free methods, checkpoint merging, cooldowns).
  • Representation and systems (which statistics are stored, quantization/precision, sharding, communication).
• A useful decomposition of an optimizer is an optimizer-stack: Tt (temporal estimator) → Pt (geometry) → Qt (representation/serialization) → Rt (update, schedule, regularization). Different papers often change only one box, and many modern implementations mix ideas across boxes.
• AdamW remains the practical baseline because it is predictable, well-understood, easy to shard and quantize, and backed by extensive engineering recipes. New methods must beat AdamW’s total risk (statistical, implementation, and systems) — not merely outperform a loss curve in a narrow protocol.
• Matrix-aware methods (Shampoo, SOAP, SPlus-family, Muon, hybrids) improved training by exploiting row/column or spectral structure:
  • Whitening-type transforms (Shampoo, SOAP) balance covariance and reduce variance; their empirical value often comes from variance adaptation rather than “second-order” curvature estimation per se.
  • Spectral-norm / polar-factor approaches (Muon, Scion) impose norm-aware constraints, changing the trust-region geometry; these are not identical to curvature inversion.
  • Hybrids proliferate because different blocks (token embeddings, attention, MLP weights, heads) have different symmetries and require different treatments.
• Temporal estimators matter: multi-timescale EMAs (AdEMAMix), variance-reduced recursive estimators (MARS), agreement filters, and spike-aware resets (SPAM) materially change sample efficiency and robustness.
• Optimizer state is a statistical object with three design dimensions: support (which coordinates/blocks hold state), timescale (effective memory in tokens), and numerical representation (precision/quantizer). Changing batch size or unit-of-time without rescaling timescales changes the estimator.
• Memory, low-precision representations, and communication constraints materially alter optimizer rankings. A method that wins per token can lose in FLOPs, wall-clock, or when memory constraints force different batch/model choices.
• Evaluation needs multiple evidence tiers and budgets: scaled evidence (billion-parameter, multi-billion-token runs), controlled evidence (systematic sweeps/ablations at smaller scale), and emerging evidence (initial reports/preprints). Claims should specify which tier and which budget metric they address.
• Practical recommendations appearing in the survey: report EMA half-lives in tokens, report per-budget performance (tokens-to-target, FLOPs, wall time, memory), use fixed-memory evaluation frontiers, and respect parameter-class routing when benchmarking.

Data & Methods

• Scope: literature and component studies released Jan 2025–Aug 2026, with earlier precursors included where necessary. Emphasis on large-scale LM pretraining because it exposes trade-offs among convergence, memory, and systems.
• Evidence classification:
  • Scaled evidence: billion-parameter models or multi-billion-token horizons (valuable for systems-level claims).
  • Controlled evidence: smaller-scale but carefully ablated experiments designed to isolate components.
  • Emerging evidence: early reports or limited replications; useful but provisional.
• Theoretical framing:
  • Norm-based view (steepest descent under a chosen norm) clarifies methods like sign/Lion, Muon, and Scion that change update geometry without necessarily estimating curvature.
  • Curvature/natural-gradient view for Fisher/Hessian surrogates (K-FAC, Sophia).
  • Whitening/covariance perspective (Shampoo, SOAP) where transforms balance gradient covariances and act as variance-adaptation.
• Empirical methods referenced:
  • Controlled ablations and component studies (to separate temporal vs geometric effects).
  • Billion-parameter pretraining comparisons (to measure wall-clock, token-efficiency, and systems interactions).
  • Memory-ledgers and quantization studies: blockwise dynamic quantization, differential codebooks for first/second moments, state-staleness analysis.
  • Examples reported in the review: AdEMAMix (multi-timescale EMA; reported 1.3B model on 101B tokens matching an AdamW baseline trained on 197B tokens), MARS (variance-reduced recursive momentum), ADOPT (reordering normalization & accumulation), SPAM (spike-aware clipping + resets), Singularity-aware Adam (randomized directional probes).
• Metrics: tokens-to-target loss, gradient evaluations, FLOPs (including matrix decompositions), wall-clock (kernels, syncs, communications), memory (impact on batch/context/model size).
• Systems considerations: sharding, communication patterns, quantized state, master weights elimination, and how communication can itself be an optimizer operation (e.g., layerwise routing).

Implications for AI Economics

• Training cost is multi-dimensional: speedups claimed in tokens-to-target do not automatically translate to lower real-world costs (FLOPs, wall time, or cloud spend). Buyers and practitioners must demand multi-budget reporting before valuing optimizer-driven savings.
• Memory-efficient optimizers can change feasible model/batch/context choices, producing indirect economic effects larger than per-step gains. For infrastructure owners and cloud providers, optimizers that reduce memory can increase throughput (larger batches/longer contexts) and thus shift effective utilization and pricing dynamics.
• Engineering risk matters economically: AdamW’s predictability reduces operational and tuning costs. New optimizers that require heavy per-layer routing, tuning, or fragility under sharding/quantization impose hidden labor and time costs — reducing their net economic benefit unless they deliver robust, repeatable gains at scale.
• Hyperparameter transferability is an economic hidden cost. Methods that require extensive re-tuning per scale, batch size, or partition increase experiment budgets and slow iteration. Optimizers with more stable default behaviors lower experimentation costs.
• Quantization and low-precision optimizer states reduce memory and communication costs, enabling cheaper training (less DRAM, fewer interconnect costs). But optimizer choice affects later quantizability: some geometries are more tolerant to quantized states (e.g., Muon is unusually quantization-tolerant per the survey). This has direct cost implications for hardware selection and cloud instance choices.
• Market products and services:
  • Optimizer-as-a-service or optimizer-tuned offerings could be monetizable if they reliably reduce end-to-end training cost for customers (not just tokens).
  • Cloud providers and hardware vendors can gain differentiation by integrating memory-efficient, low-precision optimizer implementations that reduce effective resource usage.
• Investment and procurement decisions should require:
  • Multi-metric benchmark evidence (tokens, FLOPs, wall-clock, memory) at relevant scales.
  • Fixed-memory evaluation frontiers (to compare methods under realistic memory constraints).
  • Sensitivity analyses for batch-size, schedule, and parameter partitioning.
• Policy for organizations:
  • Adopt a compositional approach to optimizer adoption: route parameter classes to appropriate optimizer components (matrix methods for hidden dense matrices; AdamW for scalars/embeddings/heads as needed).
  • Treat improvements as system-level: software, quantization, tuning and hardware choices jointly determine realized cost savings.
  • Prioritize methods with reproducible scaled evidence and documented robustness under sharding and low-precision.
• Energy and carbon economics: any optimizer that reduces tokens-to-target or wall-clock time can reduce energy per trained model — but the net effect must account for any additional FLOPs or increased synchronization. Claims of reduced energy/carbon should therefore be tied to full-stack measurements.

Summary takeaway for economic decision-makers: evaluate optimizer innovations as system-and-budget policies, not as single-number algorithmic advances. Require multi-budget, scale-appropriate evidence and account for engineering/tuning risk and memory/quantization interactions when forecasting cost savings.