Tensor methods provide a unifying algebraic toolkit to compress, adapt, and interpret large language models, but parameter savings alone rarely produce end-to-end speedups — the authors propose ρgap to expose the algorithmic and hardware sources of the compression-realization gap.

Main Finding

This survey argues that tensor decompositions and tensor networks provide a unifying, underused algebraic language for structured high‑dimensional objects in Transformer LLMs (token representations, weights, activations, KV caches, adaptation updates). By organizing and comparing tensor methods across a seven‑stage lifecycle and at the component level (embeddings, attention, FFNs), the paper clarifies when parameter savings from tensorization can plausibly translate into memory, compute, or interpretability benefits — and when they do not. It also introduces the notion of a compression‑realization gap (ρgap) to make explicit why theoretical parameter/memory reductions often fail to produce system‑level speedups.

Key Points

• Scope: A broad survey connecting classical tensor algebra (CP, Tucker, Tensor‑Train/TTM, Kronecker, block term, contractions, mode‑n products) to LLM‑specific objects and problems.
• Two complementary organizations:
  • Lifecycle taxonomy (7 stages): tokenization, embeddings, pre‑training, adaptation (PEFT), compression, inference, interpretability.
  • Component view: which Transformer modules are tensorized — embeddings, attention (incl. KV cache), feed‑forward networks.
• Contributions:
  • Unified notation and theoretical foundations for tensor operations in Transformer contexts.
  • Comparative tables and protocol‑aware comparisons that record model scales, baselines, metrics, and evaluation differences.
  • Practical systems perspective: identifies algorithmic overhead vs hardware realization and formalizes the compression‑realization gap via ρgap.
  • Links tensor methods to adjacent techniques (quantization, pruning, PEFT, efficient attention) and probabilistic tensor networks.
• Concrete observations:
  • Many LLM objects are inherently multilinear (e.g., stacked attention projections, KV caches, activation streams), so matrix‑only views underexploit structure.
  • Tensor formats (CP, Tucker, TT/TTM) trade interpretability, uniqueness, parameter counts, and numerical stability differently — choices depend on component and lifecycle objective.
  • Reported gains in parameter count do not automatically yield memory bandwidth reductions or end‑to‑end latency improvements; empirical realization depends on implementation, GPU memory layout, and additional compute from decompression or contractions.
• Practical artifacts: uses LLaMA‑3‑8B as a running case study; provides a GitHub resource with pointers and code/examples.

Data & Methods

• Nature of the work: survey / synthesis (no new experimental datasets); aggregates and systematizes literature across four clusters: classical tensor algebra, tensorized NN work, LLM efficiency literature, and mechanistic interpretability.
• Methodological approach:
  • Develop unified tensor notation and tensor‑network diagram conventions tailored to Transformer objects.
  • Introduce and explain main tensor decompositions (CP, Tucker, TT, TTM, Kronecker) and core operations (outer product, contraction, mode‑n product).
  • Dual organizational lenses:
  • Component analyses detailing how specific Transformer modules can be tensorized and what tensor formats suit them.
  • Lifecycle analyses mapping tensor methods to deployment/training stages (tokenization → embeddings → pretraining → adaptation → compression → inference → interpretability).
  • Comparative synthesis: collates reported results while explicitly noting heterogeneity in model size, baselines, evaluation metrics, and protocols so readers can compare apples‑to‑apples where possible.
  • Systems‑level analysis: defines ρgap (compression‑realization gap) to separate algorithmic parameter/memory reductions from measured system speedup and to highlight sources of overhead.
• Case studies & tooling: discusses LLaMA‑3‑8B as an illustrative benchmark; surveys available software and practical implementations on the accompanying GitHub.

Implications for AI Economics

• Cost structure vs theory: Parameter reductions from tensorization do not automatically reduce real training or inference costs. Economic evaluations must measure system‑level outcomes (memory footprint, latency, throughput, energy) not just parameter counts. The ρgap concept is directly actionable for cost modeling and ROI estimates.
• Deployment / inference costs:
  • Long‑context KV caches are a dominant memory/IO cost for many applications; tensorized representations of KV caches can reduce memory but require careful assessment of runtime contraction costs and hardware support.
  • For latency‑sensitive services, algorithmic overhead from tensor decompression or additional contractions may negate theoretical memory savings; providers must benchmark end‑to‑end latency and throughput on target hardware.
• Training & fine‑tuning economics:
  • Tensorized PEFT/adapters can reduce storage and compute for multi‑task or multi‑client fine‑tuning (lowering per‑task marginal cost), but success depends on stability of training with reduced tensor ranks and compatibility with existing optimizer/acceleration stacks.
  • Pretraining savings from tensorizing large weight matrices could lower upfront capital expense, but practical savings require reductions in total GPU hours (not just parameter storage).
• Interpretability and regulation:
  • Tensor methods that lead to more compact, structured representations may aid mechanistic interpretability; that can lower compliance/auditing costs, improve explainability, and reduce risk premiums.
• Market and productization considerations:
  • Adoption hinges on software/hardware support: BLAS/cuBLAS, kernels for tensor contractions, and model serving stacks must efficiently handle tensor formats to realize economic benefits.
  • Heterogeneous returns across applications: batch‑oriented high‑throughput inference (e.g., search ranking) is more likely to translate compression into cost savings than ultra‑low‑latency interactive services unless ρgap is small.
• Recommendations for economic decision‑making:
  • Use ρgap or similar metrics to quantify the fraction of theoretical savings realized in system throughput/latency and incorporate that into TCO models.
  • Benchmark tensorized variants on representative workloads (including long‑context traces for KV cache effects) and on target hardware before committing to model redesign for cost reasons.
  • Consider combined approaches (tensorization + quantization + PEFT) and measure interactions — combined methods often yield better practical cost reductions than any single technique alone.
• Open economic risks:
  • Uncertainty in rank selection and model quality tradeoffs may lead to hidden redevelopment costs.
  • Lack of standardized evaluation protocols across the tensorization literature complicates cross‑study cost comparisons and procurement decisions.

Summary: The survey provides a practical framework for researchers and practitioners to evaluate tensor methods in LLMs. For AI economists and decision‑makers, its main takeaway is cautionary but actionable: tensor methods can reduce capital and operating costs, but realizing those savings requires system‑level benchmarking, attention to hardware/software ecosystem support, and explicit accounting for the compression‑realization gap (ρgap).