Fine‑tuned LLM labels nudged App Store search ranking to modestly higher conversions: a worldwide A/B test shows a +0.24% lift, driven mainly by improvements on rare queries where behavioral signals are weak.

Main Finding

Fine-tuning a relatively small in-house LLM (3B parameters) on human textual-relevance judgments and using it as an offline "LLM-as-a-Judge" to generate millions of additional pointwise relevance labels materially improves a production App Store ranker. Offline, the LLM-augmented model shifts the behavioral–textual relevance Pareto frontier outward (improving both objectives); online, a worldwide A/B test showed a statistically significant +0.24% conversion-rate lift, with the largest gains on tail queries.

Key Points

• Problem: human-generated textual relevance labels are scarce and expensive, while behavioral signals (clicks/downloads) are abundant but sparse for tail queries.
• Approach: fine-tune a 3B in-house LLM on existing human judgments; use it (few-shot prompts, string labels) to generate millions of additional pointwise textual relevance labels; mix these labels into multi-objective ranker training via scalarization / data-mixing.
• Model comparison (validation vs human labels):
  • Pretrained 3B: F1 ≈ 0.287
  • Pretrained 30B: F1 ≈ 0.382
  • Fine-tuned 3B: F1 ≈ 0.800 (large improvement — fine-tuning beats a much larger pretrained model)
• Offline ranker NDCG (prod → llm-augmented):
  • Textual NDCG@1: 0.867 → 0.868; @3: 0.803 → 0.805; @7: 0.760 → 0.761
  • Behavioral NDCG@1: 0.646 → 0.652; @3: 0.479 → 0.484; @7: 0.403 → 0.407
  • Authors interpret these consistent gains as a Pareto improvement.
• Online A/B (worldwide): +0.24% conversion-rate lift (statistically significant); improvement observed in 89% of storefronts; largest lift on low-frequency (tail) queries.
• Practical observation: fine-tuning a smaller, efficient model provided the best tradeoff of accuracy and operational cost.

Data & Methods

• Data:
  • Large-scale historical App Store search logs: millions of query–app candidate pairs (used for behavioral labels and for LLM inference).
  • Smaller human-annotated dataset of textual relevance judgments (split into train/validation for fine-tuning and evaluation).
• LLM label generation:
  • Fine-tune 3B on human judgments; experiment with pretrained 3B and pretrained 30B for comparison.
  • Prompt design: explored zero-shot and few-shot; few-shot with string labels worked best.
  • Inference: generate ordinal pointwise textual relevance labels (same rubric as human judges) for millions of query–app pairs across storefronts/languages.
  • Prompt strictness: instruct model to output label only, to align with judge format.
• Ranker training:
  • Multi-objective learning-to-rank optimizing textual and behavioral objectives.
  • Use scalarization/data-mixing: same query–app feature vectors can appear multiple times with different labels (behavioral vs textual).
  • Train multiple models by varying the behavioral/textual data-mix (e.g., 90–10, 70–30, 50–50) to explore the Pareto frontier.
• Evaluation:
  • Offline: NDCG@k for textual and behavioral relevance on held-out sets (textual NDCG uses human judgments disjoint from LLM training).
  • Online: worldwide A/B test measuring conversion rate (proportion of search sessions with ≥1 download); breakdown by query frequency to identify head vs tail effects.

Implications for AI Economics

• Cost-effectiveness and scaling:
  • Fine-tuning a small model can deliver higher task-specific accuracy than much larger pretrained models, lowering compute and operational costs per label and enabling scalable label generation.
  • Generating labels with an LLM is a force multiplier for human labor: fewer human annotations are needed to supervise and validate a much larger synthetic training set.
• Revenue and product impact:
  • Even modest relative lifts (+0.24% conversion) can be economically significant for mature, high-traffic marketplaces; gains concentrated in tail queries can unlock long-tail demand and incremental downloads that behavioral signals miss.
  • Improving long-tail relevance increases search efficiency and discovery, which can raise consumer surplus and platform revenue, and reduce churn from poor discovery.
• Market structure and competition:
  • Platforms that adopt LLM-augmented labeling may gain an advantage in discovery quality, especially for niche items/apps, potentially increasing platform competitiveness and lock-in effects.
• Labor and organizational effects:
  • The approach augments, rather than replaces, human judges (humans provide the rubric, training data and validation), shifting human work toward oversight, auditing, and edge-case handling.
• Risks, externalities and governance:
  • Reliance on LLM-generated labels requires robust validation and continual monitoring to avoid drift, systemic bias, or misalignment with the human rubric.
  • Potential for label miscalibration or amplifying biases present in training judgments—necessitates auditing, calibration, and periodic human re-annotation.
  • Regulatory/consumer-protection considerations: decisions that affect visibility and conversion should be explainable/auditable; synthetic labels must not degrade fairness or competitive neutrality.
• Research and investment priorities:
  • Investing in smaller fine-tuned models and label-generation pipelines can be a higher-return strategy than only scaling model size.
  • Monitoring infrastructure, periodic human validation, and economics-focused A/B designs are essential to measure ROI and limit negative externalities.
• Generalizability:
  • The LLM-as-a-Judge, data-mixing multi-objective training pipeline is transferable to other large-scale search and recommendation marketplaces where labeled semantic relevance is scarce and tail discovery matters.

Suggestions for practitioners (brief): - Run cost-benefit analyses comparing fine-tuning smaller models vs. using larger pretrained models. - Design continuous human-in-the-loop validation and bias audits for synthetic labels. - Prioritize tail-query evaluation and business-metric A/B tests to capture economic impact.