A simple SVD-based factorization of Transformer keys cuts KV cache size by up to 75% with only ~2% perplexity cost and, at 7B scale, saves ~25 GB per user for 128K contexts—enabling roughly 60% more concurrent users; training-from-scratch at r=d/4 matches full-attention perplexity while reducing parameters and speeding training.

Main Finding

Factored keys — reducing key/query dimensionality for selection (dselect) while keeping value dimensionality full — lets practitioners shrink the KV cache of pretrained transformers with minimal quality loss. A simple truncated-SVD on each key projection (WK ≈ A·B) yields “thin” cached keys (A) and absorbs B⊤ into the ephemeral query projection (WQ B⊤), preserving attention scores exactly. Optional lightweight QK fine-tuning (3 epochs on <1% pretraining data) recovers most quality. Result: large reductions in per-user memory for long-context inference, measurable decode throughput gains, modest parameter and training-time savings for from‑scratch models, and straightforward composition with GQA/MLA and quantization for greater compression.

Key Points

• Conceptual split: attention does selection (Q·K⊤ → scalar weights) and value transfer (weights·V). Selection needs far fewer dimensions (O(log N)) than value transfer.
• Factored keys method:
  • Compute truncated SVD of pretrained WK: WK ≈ A (d×r) · B (r×d).
  • Use W′K = A as r-dimensional cached keys; set W′Q = WQ B⊤ so queries are r-dimensional at runtime.
  • No change to values, attention computation, or inference kernels.
  • One-time offline SVD per layer; no retraining required. Optional small QK fine-tune improves results.
• Compression asymmetry: keys are far more compressible than queries. K-only SVD degrades quality much less than Q-only.
• Empirical results (highlights):
  • GPT-2 (post-hoc SVD): K-only rank 384 (dmodel/2) → +2% PPL; rank 192 (dmodel/4) with 3-epoch QK fine-tune → +1.8% residual gap while saving 75% of key cache.
  • Mistral-7B: SVD + QK fine-tune at rank 256 (75% K cache saved) → residual ≈ +2% vs control; some math/reasoning tasks are more sensitive but recover with domain fine-tuning.
  • LLaMA-7B trained from scratch with dselect = dmodel/4: matches full-attention perplexity after 20B tokens while using 12% fewer parameters and training ~8% faster.
• Deployment/throughput:
  • For a 7B model at 128K context, factored keys (dselect=dmodel/4) save ~25.2 GB per user (37.5% total KV reduction). SVD-only (dselect=dmodel/2) saves ~16.8 GB (25%).
  • Measured decode throughput gains for Mistral-7B (context 4096): up to 1.44× speedup (batch size 32, r=256). Gains grow with batch size since inference is bandwidth-bound.
• Composition: factored keys compose with GQA and MLA; e.g., GQA-8 + thin keys → ~84.4% KV savings. Combined with quantization, authors report up to ~16× combined key compression potential.
• Limitations noted: reasoning-heavy math tasks (GSM8K) can be sensitive; domain-matched fine-tuning mitigates this.

Data & Methods

• Theoretical basis: Johnson–Lindenstrauss lemma to argue selection (ranking) needs O(log N) dims; independent lower bounds (Haris & Onak 2025) show Ω(log n) per-token is necessary.
• Core method: truncated SVD of each layer’s WK to rank r ≡ dselect; set W′K = UrΣr and W′Q = WQ Vr, so q′·k′⊤ = original q·k⊤ exactly.
• No change to V projections, FFNs, output heads, or attention computation.
• Experiments:
  • Small-scale controlled algorithmic/positional experiments supporting 1 dimension per head for positional selection and log2 N scaling for content selection.
  • Post-hoc compression: GPT-2 (124M), SVD-only and SVD+QK fine-tuning on WikiText-103 (3 epochs).
  • Large-scale training: LLaMA-7B variants trained from scratch (2B and 20B tokens) comparing full attention vs thin keys (dselect = dmodel/4).
  • Post-hoc at 7B scale: Mistral-7B SVD + QK fine-tuning (r = 512, 256, 128) evaluated on validation PPL and downstream tasks (Hellaswag, ARC-Challenge, WinoGrande, MMLU, GSM8K).
  • Decode throughput measured on H100 SXM; bandwidth roofline model used to predict speedups.
• Metrics: perplexity (PPL), downstream task accuracy/EM, decode tokens/sec, KV cache size (GB).
• Fine-tuning footprint: reported QK fine-tuning uses ~3 epochs, ~10M tokens (<1% of pretraining corpora in presented cases).

Implications for AI Economics

• Lower marginal cost of long-context inference:
  • Per-user KV cache reductions (e.g., ~25 GB saved per user for 7B at 128K) reduce memory footprint per active session, directly lowering required GPU memory resources and enabling higher concurrency on the same hardware.
  • Authors report enabling ~60% more concurrent users on identical hardware in representative config; this maps to proportional reductions in per-request infrastructure cost.
• Capital and operational expenditure (CapEx and OpEx) impact:
  • Fewer or smaller GPUs required for a given user load and context length reduces CapEx (hardware purchases) and OpEx (power, cooling).
  • Smaller model-weight footprint (thinner WQ/WK) and reduced KV transfer during decode lower memory bandwidth demand and energy per token, cutting runtime energy costs and potentially instance-hour billing.
• Faster time-to-result and throughput improvements:
  • Measured 1.1–1.44× throughput improvements (depending on batch) accelerate aggregate throughput, enabling better utilization and higher revenue per GPU-hour.
  • For latency-sensitive services, improved throughput at large batch sizes (e.g., multi-user batched serving) allows more efficient multiplexing.
• Reduced switching and deployment friction:
  • Zero-cost SVD (no retrain) offers a low-friction retrofit to existing, large pretrained models—firms can extend context length support and cut serving costs without costly pretraining re-runs.
  • This lowers barriers for incumbents to adopt long-context capabilities, reducing incentives to retrain or switch architectures solely for cache savings.
• Product and market implications:
  • Cheaper long-context inference enables new product features (multi-document synthesis, long personalized assistants) at lower unit costs, potentially increasing demand.
  • Lower marginal cost per session can shift pricing models—providers may offer longer-context tiers at competitive prices or adjust metering (e.g., cost per token vs cost per context).
  • Small providers or on-prem deployments may be more viable as memory requirements fall, increasing competition and reducing concentration.
• Composability multiplies value:
  • Combining factored keys with GQA/MLA and quantization multiplies KV reductions, so integrators can tune trade-offs across latency, quality, and cost. This flexibility has strategic value when optimizing SLAs and pricing.
• Risks, caveats, and follow-ups:
  • Quality-sensitive workloads (multi-step reasoning/math) can require domain fine-tuning; some business-critical tasks may need validation before compressing deployed models.
  • Economic gains depend on workload characteristics: user concurrency, context lengths, batching behavior, and GPU pricing. Savings are largest when KV cache dominates bandwidth (long contexts, large batches).
  • One-time operational cost: performing per-layer SVDs and optional fine-tuning is cheap relative to full retrain but involves engineering validation and CI to ensure no regressions.
  • Competitive dynamics: since this is a low-friction retrofit, widespread adoption could compress margins in the inference market as providers pass savings to customers or engage in price competition.
• Policy and societal considerations:
  • Lower inference costs and better resource efficiency reduce environmental footprint per token, aligning with sustainability goals.
  • Easier access to long-context LLMs could accelerate adoption in sensitive domains (health, law); firms must ensure robustness and monitor downstream harms.

Overall, factored keys present a practical, low-cost lever to reduce memory- and bandwidth-related inference costs for long-context transformers, with measurable throughput and concurrency benefits and modest quality trade-offs that are often recoverable with lightweight fine-tuning. For AI economics, the method lowers marginal serving costs, reduces friction to adopt long-context features, and enables new product and pricing configurations while increasing competitive pressure in the inference services market.