An end-to-end reinforcement-learning ad-text generator that optimizes for conversions and policy compliance boosts conversion-adjusted click-throughs in production tests, outperforming conventional two-stage pipelines.

Main Finding

RELATE is an end-to-end reinforcement-learning–enhanced LLM framework for advertising text generation that unifies generation and business-objective alignment. By integrating multi-dimensional rewards (quality, diversity, and conversion—CTCVR) directly into the generation policy and using a granularity-aware credit-assignment mechanism, RELATE outperforms multi-stage baselines on large-scale industrial data and yields a 9.19% relative uplift in CTCVR in online production under strict policy constraints.

Key Points

• Problem addressed
  • Industrial ad text pipelines typically separate candidate generation from downstream performance alignment, causing objective mismatch, low funnel efficiency, and suboptimal online performance.
  • Challenges include reward sparsity, delayed feedback, compliance constraints, and text fatigue (creative reuse → degraded long-term engagement).
• Core idea
  • Formulate ad text generation as a constrained optimization problem and solve it end-to-end with policy learning: directly maximize expected reward E[r(x,y)] where r jointly encodes conversion, quality/compliance, and diversity.
• Multi-dimensional reward design
  • Quality rewards: length, required format, semantic relevance, factual/correctness checks, and risk-control (filter/exclude exaggerated or non-compliant content).
  • Diversity reward: penalize high-frequency template-like n-grams (n∈[2,10]) via an attributable penalty over generated n-grams to mitigate mode collapse.
  • CTCVR reward: predicted conversion (CTCVR) estimated by a Shared-Bottom multi-task model over Qwen3-Embedding-0.6B representations (entire-space modeling for CTR & CTCVR).
• Credit assignment
  • Granularity-aware mechanism distinguishes token-level rewards (e.g., blacklisted words, token-triggered penalties) from sentence-level rewards (e.g., predicted conversion, overall relevance).
  • Token-level and sentence-level attributions are combined via a group computation to produce token-level advantages for more effective policy updates—improves training efficiency and convergence.
• Training/regularization
  • RELATE integrates policy learning with reference-model regularization (KL constraints visible in framework) to stabilize optimization and control drift from the pretrained generator.
• Empirical outcomes
  • Extensive offline experiments show consistent gains on relevance and performance proxies.
  • Online A/B deployment in production yields a statistically significant 9.19% relative increase in CTCVR compared to the production baseline.

Data & Methods

• Data
  • Large-scale industrial advertising logs and corpora (generation context: user queries, bidwords, landing page content; online feedback: clicks and conversions).
• Representations & prediction
  • Qwen3-Embedding-0.6B used to encode (query, ad metadata, ad text) into a shared semantic space.
  • Shared-Bottom multi-task model jointly predicts CTR and CTCVR (entire-space modeling).
• Reward engineering
  • Explicit reward components for quality (several sub-rewards), diversity (n-gram frequency penalty), and conversion (predicted CTCVR).
  • Diversity penalty formula (summary): r_diversity = − sum_{g in n-grams(y)} I(g ∈ G_high) * Δ_d, where G_high is the set of high-frequency n-grams and Δ_d is penalty strength.
• RL & optimization
  • End-to-end policy learning (policy-gradient style with advantage estimation) built on a GPRO-like architecture; group computation produces token-level advantages from mixed-granularity rewards.
  • KL-based regularization to a reference model (to keep outputs practical and stable).
• Evaluation
  • Offline metrics for relevance, quality, diversity; online A/B testing on production ad serving measured CTCVR (primary business metric).

Implications for AI Economics

• Better alignment of LLMs with monetizable outcomes
  • Directly optimizing for conversion (CTCVR) compresses the generation-to-optimization funnel and raises the ceiling of monetizable performance from creative generation—improves advertiser ROI and platform revenue per impression.
• Operational efficiency and cost
  • End-to-end policy reduces reliance on multi-stage pipelines (candidate generation → separate ranking/filtering), simplifying engineering and potentially lowering latency and operational costs.
• Market and auction effects
  • Higher average CTCVR from generated creatives can alter effective quality scores/expected values in ad auctions, shifting competitive dynamics and pricing for advertisers.
• Long-run value and creative fatigue
  • Built-in diversity rewards address creative fatigue, which supports more stable long-term yields from the same inventory and may reduce churn in user engagement metrics crucial to platform economics.
• Governance, compliance, and externalities
  • Integrating quality and risk-control rewards is essential to avoid harmful short-term revenue chasing (e.g., misleading claims). However, reward misspecification could induce perverse incentives (e.g., exploitative framing) that harm user welfare or attract regulatory scrutiny.
• Data and privacy considerations
  • The approach relies on large-scale click/conversion logs and representation learning; platforms must manage data governance and privacy (and potential biases in conversion signals) as these directly shape reward-driven optimization.
• Transferability to other revenue-generating NLP tasks
  • The unified reward + credit-assignment architecture generalizes to other contexts where language outputs map to downstream economic metrics (e.g., recommender copy, email subject lines, sales chat), suggesting broader applicability across digital monetization tasks.
• Measurement & evaluation
  • Emphasizes need for robust online experiments (A/B tests) because offline proxies of business value remain imperfect; careful experimental design and guarding against short-term exploitation are essential for reliable economic conclusions.