Using large language models to generate and clean training interactions raises e-commerce recommendation recall by 6.02% and gross merchandise value by 1.22%, mainly by improving long-tail item recommendations; the approach boosts I2I systems without changing model architectures.

Main Finding

LLM-I2I — an LLM-enhanced, data-centric pipeline that (1) fine-tunes an LLM to generate synthetic user→item interactions and (2) fine-tunes a second LLM to discriminate/filter those interactions — substantially improves item-to-item (I2I) recommendation quality, particularly for long-tail items. Across academic benchmarks and a large-scale deployment on AliExpress, LLM-I2I improved multiple I2I backbones and produced real-world gains (online: +6.02% recall number (RN) and +1.22% GMV).

Key Points

• Approach: A data-centric augmentation pipeline composed of two supervised-fine-tuned LLMs:
  • Generator: conditioned on user static info + recent behavior, trained to predict next item(s).
  • Discriminator: conditioned on user and candidate item features, predicts whether a user would interact with the candidate (Yes/No + confidence).
• Long-tail focus: the generator is trained with a long-tail aware loss (upweighting long-tail items; paper uses α=4.0 vs β=1.0) to encourage generation of rare items.
• Filtering is critical: discriminator confidence is used to select high-quality synthetic interactions; experiments show higher-confidence synthetic data improves downstream model performance, while excessive low-quality synthetic examples can hurt performance due to distribution drift.
• Practical recipe:
  • Input to generator: each user’s most recent ten interactions (to respect LLM input limits and emphasize recency).
  • Keep synthesized pairs only if discriminator outputs “Yes” with high confidence (they used threshold of 1.0 in the paper).
  • Merge filtered synthetic data with original interaction data and train existing I2I models (no change to the I2I model architecture required).
• Works across diverse I2I algorithms: neighborhood (BM25), matrix factorization (BPR), deep retrieval (YouTubeDNN), graph-based (Swing), and LLM-CF style methods — all showed improvements.
• Real-world deployment: billion-scale industrial dataset (AEDS) and production deployment on AliExpress with measurable business KPIs improved (RN and GMV).

Data & Methods

• Datasets:
  • ARD (Amazon Review Dataset) — three categories used (Beauty, Sports & Outdoors, Toys & Games). Long-tail = bottom 20% items by purchase frequency.
  • AEDS — proprietary AliExpress dataset: ~209M users, 68M items, 10B interactions; ~16M long-tail items; ~25% items clicked only once.
• Model training / implementation details:
  • Base LLM: Llama2-7B-Chat (for fair comparison with prior LLM-CF work).
  • Training: full-parameter supervised fine-tuning (SFT) of generator and discriminator on domain interaction data; batch size 16; max input length 1024; learning rate 5e-5; random seed 42.
  • Acceleration: DeepSpeed ZeRO-2 and FlashAttention used for training speed/memory.
• Objectives:
  • Generator: next-item prediction loss weighted higher for long-tail targets (wt+1 ui = α if target is long-tail else β).
  • Discriminator: binary classification loss trained on observed positives (real user interactions) and randomly sampled negatives; outputs label + confidence.
• Augmentation procedure:
  • For each user, use recent 10 items as prompt; generator predicts candidate item IDs.
  • Discriminator evaluates each candidate; keep those labeled positive with high confidence.
  • Combine filtered synthetic interactions with original data for I2I model training.
• Evaluation:
  • Offline metrics: Recall@K and similar retrieval metrics (shown to improve as discriminator confidence increases).
  • Online metrics (production): recall number (RN) and gross merchandise value (GMV) uplift reported from A/B test of deployed system.
• Empirical observations:
  • Increasing amount of synthetic data helps up to a point, then degrades performance (distribution shift).
  • Filtering by discriminator confidence materially improves downstream I2I results.
  • The method is particularly effective for long-tail items (reduces sparsity problems).

Implications for AI Economics

• Cost-effectiveness vs model-centric upgrades:
  • Data-centric approach preserves existing small/fast I2I models and online serving infrastructure (inverted index, top-K tables), avoiding the CPU/GPU and latency costs of deploying larger or deeper models online.
  • Offline costs: SFT of two LLMs and large-scale generation/discrimination still require compute, but these are offline batch operations and can be scheduled/optimized — trade-off is favorable when modest online model changes are costly.
• Revenue and market structure effects:
  • Uplifts to recall and GMV indicate direct revenue benefits; improved discovery for long-tail items can increase sales diversity and flatten concentration on head items, with implications for platform-side supplier competition, inventory allocation, and long-tail supplier welfare.
• Allocation of compute / return on investment:
  • Platforms should weigh offline LLM fine-tuning and synthetic-data-generation costs versus marginal GMV and conversion lift — the paper’s reported +1.22% GMV is an example point estimate for a large cross-border marketplace.
• Risks and governance:
  • Synthetic data can amplify biases or introduce artifacts — the discriminator mitigates but does not eliminate this risk; monitoring for fairness, category bias, supplier over- or under-promotion is needed.
  • Privacy and compliance: synthetic generation uses user behavior traces; ensure compliance with data-use policies and user consent frameworks.
  • Hallucination and semantic drift: LLMs may fabricate items or misrepresent item features; strong discriminator thresholds and sanity checks on item IDs/availability are important.
• Scalability and future directions:
  • Potential to use smaller/efficient LLMs or retrieval-augmented generation to reduce offline cost while preserving benefits.
  • Joint optimization: dynamically tune the amount of synthetic data per user segment and discriminator confidence threshold for cost-benefit maximization.
  • Broader economic impacts: better tail item recommendations can change supplier pricing/promotion strategies and platform commission structures; platforms may invest more in data-centric augmentation rather than costly model upgrades.
• Suggested operational evaluation:
  • Conduct a formal cost-benefit analysis (compute + engineering vs incremental GMV/RN), segment-level A/B tests (head vs tail), and audits for bias and supplier fairness before wide roll-out.