A probabilistic residual quantizer for multimodal embeddings lifts advertiser value by 1.5% in live tests; the method improves embedding discretization and has been fully deployed across a major short‑video recommendation platform.

Main Finding

RQ-GMM (Residual Quantized Gaussian Mixture Model) is a two-stage multimodal discretization method that combines residual quantization with Gaussian Mixture Models (GMMs). By modeling residual embeddings probabilistically and using soft assignments with residual quantization, RQ-GMM produces higher-quality discrete semantic IDs (better reconstruction RMSE, near-complete codebook utilization) that, when used as categorical features in CTR models, yield consistent offline and online gains. In a large-scale online A/B test on a short-video platform, RQ-GMM improved Advertiser Value by 1.502% over RQ-VAE and 3.600% over direct embedding concatenation; it is deployed in production serving hundreds of millions of users.

Key Points

• Motivation
  • Directly feeding pre-trained continuous modal embeddings (e.g., BERT/CLIP) into CTR models is suboptimal because pre-trained objectives differ from preference modeling and joint training dynamics mismatch with sparse ID features.
  • Two-stage discretization (map embeddings → semantic IDs → CTR model) decouples semantic extraction and preference learning, stabilizes training, and provides coarse-to-fine signals.
• Method (RQ-GMM)
  • Replace hard clustering (K-means / VQ) with GMM at each residual quantization level.
  • Model residuals r(l-1) as K-component Gaussian mixtures with diagonal covariances; compute posteriors γ and select k* = argmax γ for residual propagation and ID assignment.
  • Train per level with EM (E-step: soft posteriors; M-step: update mixing coefficients, means, variances). Use hard assignment for residual propagation to keep inference consistent.
  • Semantic ID is the sequence of selected component indices across L levels: ID(x) = [k1, ..., kL].
• Advantages over baselines
  • Captures cluster shape and uncertainty (covariances) → better modeling of complex multimodal distributions.
  • Soft assignment smooths boundary effects; mixing coefficients reflect data density, mitigating codebook collapse.
  • Improved utilization and reconstruction vs VQ-VAE, RQ-VAE, and RQ-KMeans.
• Deployment & integration
  • IDs stored in feature store; inference is batched offline so online latency is unchanged.
  • Semantic IDs used as categorical embeddings concatenated or combined with MLP-transformed continuous embedding in standard CTR models (FNN, IPNN, DeepFM-style architectures).

Data & Methods

• Datasets
  • Public offline: Amazon-Review categories — Appliances, Beauty, Automotive. Ratings binarized (1–3 → 0; 4–5 → 1); chronological 8:1:1 train/val/test splits.
  • Industrial online: proprietary short-video platform (hundreds of millions DAU) for 7-day A/B testing measuring Advertiser Value (AV).
• Baselines
  • None (no multimodal), direct embedding concatenation, VQ-VAE, RQ-VAE (residual quantized VAE), RQ-KMeans.
• Implementation details
  • Embeddings: BERT 768-d for offline experiments.
  • Discretization configurations: two-level residual structure with 128 codes per level for RQ-GMM and RQ-KMeans (VQ-VAE used 256 single-level; RQ-VAE two-level 128).
  • EM/K-means iterations up to 30; diagonal covariance for efficiency.
  • CTR models: FNN and IPNN backbones; embedding dimension for IDs = 16; CTR MLP hidden layers (128, 32, 8); binary cross-entropy loss.
• Metrics & results
  • Discretization: RMSE reconstruction and codebook utilization (proportion of codes used). RQ-GMM achieved the lowest RMSE and highest utilization across datasets.
  • CTR: AUC and LogLoss improvements across datasets and backbones; RQ-GMM consistently best.
  • Online: RQ-GMM improved Advertiser Value by 3.600% vs embedding concatenation, 1.502% vs RQ-VAE, and 0.613% vs RQ-KMeans (statistically significant).
• Complexity
  • Training per level: O(T · N · K · D) for T EM iterations; inference per sample: O(L · K · D); memory O(L · K · D) for means and variances. Diagonal covariances reduce overhead; RQ-GMM converges faster in practice.

Implications for AI Economics

• High-leverage small-percentage gains
  • A ~1.5% Advertiser Value lift at platform scale (hundreds of millions daily users) translates to large absolute revenue/ROI improvements. Small relative CTR/AV gains justify investment in better multimodal discretization methods.
• Cost vs benefit trade-offs
  • RQ-GMM increases offline training complexity (EM with covariance updates) but converges faster and is run offline. Serving costs remain low because IDs are precomputed and stored—no added online inference burden—so operational costs rise modestly while revenue increases.
• Efficiency and capacity utilization
  • Near-complete codebook utilization implies more efficient representation capacity (fewer wasted code entries), enabling more expressive discrete semantics without growing codebook size—reduces need for larger codebooks and associated storage/maintenance costs.
• Product & auction effects
  • More accurate CTR predictions change auction dynamics and estimated ad values; improved prediction precision can increase advertiser willingness to bid and/or platform pricing power, but may also require recalibration of bidding/outcome metrics.
• Deployability and operational considerations
  • Offline batch inference and feature-store integration make RQ-GMM practical to deploy in large production systems with low latency constraints—lower engineering risk than online joint-training alternatives.
  • Regular retraining of GMMs and periodic re-generation of IDs (to reflect content drift) is necessary; this maintenance cost should be accounted for in long-term economic evaluation.
• Generalizability & multiplier effects
  • The approach is applicable beyond short-video: other recommender systems and ad platforms with multimodal content can replicate gains. If broadly adopted, such discretization methods can systematically raise industry-level CTRs and ad market values.
• Risks & caveats
  • Economically meaningful improvements depend on stable online A/B results and continued distributional match between training embeddings and live content; codebook drift or modality shifts (new content types) may reduce gains if not managed.
  • Privacy/attribution: discretizing embeddings may alter privacy/attribution characteristics of features; teams should assess compliance and upstream embedding sensitivity.

Bottom line: RQ-GMM offers an economically attractive, operationally practical way to convert multimodal embeddings into high-quality categorical signals for CTR models, yielding meaningful revenue improvements at scale with modest operational overhead.