A decay-aware mixed-precision scheme slashes recurrent-state storage by about 69% and speeds up recurrent updates up to 2.01× while maintaining near-FP32 accuracy at 9.9 bits per state value, lowering end-to-end decoding cost (TPOT) by up to 10.9%.

Main Finding

DAMP (Decay-Aware Mixed-Precision recurrent-state quantization) is a post-training, static mixed-precision scheme for the persistent recurrent states used by GDN and KDA layers. By ranking key channels with a combined score of (quantization-error energy × decay-based persistence) and protecting the top channels at higher precision (FP16) while storing the rest in INT8 (with a Hadamard transform), DAMP achieves near-FP32 accuracy at 9.9 effective bits per state value, reduces recurrent-state storage by 69.1%, speeds the recurrent-state update kernel up to 2.01×, and lowers end-to-end time-per-output-token (TPOT) by up to 10.9% on evaluated models and benchmarks.

Key Points

• Problem and novelty

  • Recurrent states in GDN/KDA are persistent, repeatedly read/updated and therefore sensitive to quantization error feedback; prior work had not studied post-training quantization of these states.
  • DAMP is the first static, offline-calibrated mixed-precision layout tailored to this error-feedback recurrence.

• Empirical structure exploited

  • States show strong two-axis concentration: few key channels dominate magnitude; within-channel value-axis imbalance remains but is reduced via a per-channel Hadamard transform.
  • After Hadamard + INT8, quantization-error energy remains concentrated in a small subset of key channels.
  • Learned decay (per-key in KDA, per-head in GDN) exhibits a stable ordering of retention strength across prompts/tasks; this allows estimating persistence offline.

• Risk scoring and selection

  • For each key channel u, DAMP computes:
  • Eu = empirical average squared quantization-error energy (under the low-precision mapping).
  • aeff,u = geometric mean of per-step retention (exp(E[log a_t,u])).
  • Pu = capped cumulative persistence = 1 / max(1 − aeff,u^2, τ) (τ caps persistence).
  • Cu = Eu × Pu is the static risk proxy; protect the top-Khi channels per layer/head under a fixed budget.
  • Main operating point: protect Khi = 16 channels per head (FP16), remaining channels INT8+Hadamard → ≈9.9 bits/value.

• Implementation

  • Calibration is offline and one-time; DAMP records a permutation per layer/head that places protected channels contiguously (packed layout) so the run-time update is regular and fused (no per-token ranking).
  • Fused executor reconstructs INT8 channels, computes updates in FP32, re-quantizes and writes back.

• Quantization baselines and limits

  • Uniform low-precision formats (INT8, FP8) already degrade complex-reasoning accuracy; INT4 / NVFP4 nearly collapse accuracy.
  • DAMP recovers near-FP32 performance at much lower storage than uniform formats.

Data & Methods

• Models evaluated

  • Qwen3.6-35B-A3B (30 GDN + 10 full-attn layers).
  • Kimi-Linear-48B-A3B-Instruct (20 KDA + 7 MLA layers).

• Benchmarks and metrics

  • Math: AIME 2026 Parts I/II, HMMT Feb 2026, IMO-AnswerBench.
  • General reasoning: GPQA-Diamond, MMLU-Pro.
  • Code: LiveCodeBench-v6.
  • Metrics: task-specific accuracies (multiple generations per problem), recurrent-state storage, recurrent-update kernel latency, full-model TPOT.

• Quantization & calibration details

  • Low-precision integer storage: per-block asymmetric affine quantization (b = 8 for INT8, b = 4 for INT4).
  • Hadamard transform applied along the value dimension within each key channel before INT8 quantization to reduce within-channel range imbalance.
  • Calibration set: 32 unlabeled Pile documents (8 each from DM Mathematics, PubMed Abstracts, GitHub, StackExchange), truncated to 256 tokens, processed with teacher forcing.
  • Sampling: state sampled every 8 tokens → ~1,024 state samples per recurrent layer for calibration.
  • Allocation: per-layer, per-head scoring; same Khi across layers/heads for kernel shape stability. Main config: FP16 protected tier + INT8+Hadamard low tier, Khi = 16 → 9.875 bits (reported 9.9).

• Error propagation model used to motivate design

  • Reconstructed low-precision state follows Sq_t = Q( A_t Sq_{t−1} + β_t k_t v_t^⊤ ), leading to accumulated error ∆S_t = A_t ∆S_{t−1} + R_t. This shows injected quantization residuals are transformed and potentially retained by decay factors; hence combining injection energy and decay-based persistence is appropriate.

• Results (summary)

  • At ~9.9 bits/state-value, DAMP maintains accuracy close to FP32 across math, reasoning, and code benchmarks on both evaluated models.
  • Recurrent-state storage reduced by 69.1%; recurrent-update kernel accelerated up to 2.01×; full-model TPOT reduced up to 10.9%.

Implications for AI Economics

• Lower memory footprint per active request

  • 69% reduction in recurrent-state storage directly reduces GPU memory used per concurrent session. For high-concurrency deployments (e.g., batch inference, agentic multi-turn systems), this increases the number of simultaneous sessions per GPU or reduces required GPU instance counts, lowering provisioning and amortized hardware costs.

• Increased throughput and lower latency

  • Up to 2× speedup in the recurrent-state update kernel and up to 10.9% reduction in TPOT improve throughput and user-perceived latency. This translates to lower compute time per token and therefore reduced cloud compute billable time (or higher QPS on given hardware), improving cost-efficiency.

• Enables longer-context and agentic workloads at lower marginal cost

  • Recurrent-state compression reduces the dominant memory growth that otherwise scales with interaction length. This makes supporting longer contexts or extended multi-step agents more economical, lowering marginal cost per additional token/context length.

• Low operational and engineering cost

  • DAMP is post-training and uses a one-time offline calibration (no retraining). That reduces engineering and ML lifecycle costs compared to approaches requiring fine-tuning or model surgery. The static layout and fused kernel minimize per-request runtime complexity.

• Applicability constraints and risks that affect economics

  • Applicability is limited to models using GDN/KDA-style recurrent states (hybrid architectures), not standard softmax KV caches; benefits therefore depend on model architecture choices.
  • Calibration-domain mismatch risk: if deployment inputs differ drastically from calibration data, protected-channel selection might be suboptimal, causing accuracy regressions — a potential operational risk that could demand guardrails or periodic recalibration.
  • Implementation complexity: requires packed mixed-precision storage and a fused executor kernel. There is engineering cost to implement efficient kernels on target hardware (GPU, ML accelerators). Hardware lacking efficient mixed-precision gather/reconstruct paths could limit speedups.
  • Small residual accuracy risk: while average accuracy is near-FP32, edge cases on rare, critical tasks (high-stakes reasoning) could still see degradations; this may affect decisions in safety- or compliance-sensitive deployments.

• Broader market impact

  • If widely adopted, methods like DAMP lower the marginal cost of serving long-context/agentic workloads, potentially accelerating adoption of such services and making high-context agents commercially viable at lower price points.
  • Providers can trade some additional implementation complexity for significant infrastructure savings; for cloud providers or large-scale inference services, such techniques can materially reduce operational expenses (OPEX) and energy consumption per token.

Summary takeaway: DAMP provides a practical, low-engineering-cost way to substantially cut memory and runtime costs of recurrent-state hybrid models while preserving accuracy, shifting the economics in favor of deploying long-context and agentic LLM applications — provided calibration aligns with deployment data and engineering for fused mixed-precision kernels is handled.