A production conversational ad agent that co-evolves its decision-maker and retrieval tools raised revenue per thousand impressions by 22% and expanded ads coverage by 74% in a live A/B test, while substantially improving offline relevance and slate diversity.

Main Finding

AdsWorldEngine is a production-oriented, agentic framework for conversational advertising that (1) uses an Opportunity Gate to decide when to show ads, an Orchestrator to generate intents and call ad tooling, and an Evaluator to score slates offline; (2) introduces an iterative co‑training loop in which an RL-trained Orchestrator produces high/low reward rollouts that are transformed into preference data to improve retrieval/relevance/ranking tools; and (3) introduces label‑grounded judgment modeling (thinking traces + reflection + cost‑sensitive GRPO) to train reliable judgment models for subjective production decisions. Offline, the system raises diversity by 60% and relevance by 80% vs production; online A/B tests show +22% RPM and +74% ads coverage.

Key Points

• System architecture
  • Opportunity Gate: production rules + learned trigger model that abstains or passes turns (prioritizes user experience; false positives penalized heavily).
  • Orchestrator (actor): resolves dialogue state, extracts constraints, generates 1–3 commercial intents, calls tools (retrieval, relevance, ranking, pricing, signals), and uses reflection to pick a final top‑3 slate.
  • Tool Set: retrieval, relevance, ranking, pricing, signal models that the Orchestrator invokes.
  • Evaluator: offline scorer for user/advertiser/publisher perspectives; main signals used are conversation→ad relevance (learned) and slate diversity (pairwise cosine similarity of ad embeddings).
• Iterative actor–tool coevolution
  • Two‑stage Orchestrator training: supervised fine tuning (SFT) to teach format, tool calls, and intermediate traces; agentic RL (GRPO) to align intermediate actions with final slate objectives (relevance, diversity).
  • After Orchestrator RL updates, high/low reward rollouts for the same conversation are compared to construct preference pairs (intent, a+, a−) for tools.
  • Tools are then updated (e.g., DPO or ranking losses) on that preference data. Repeat loop (Algorithm 1).
  • This makes the system self‑improving: the actor learns to use tools well, and the tools improve from rewarded actor behavior.
• Label grounded judgment modeling (for subjective/prod decisions)
  • Data pipeline: collect human labels under explicit production guidelines; for each labeled example, generate a thinking trace conditioned on the guideline and label; run a reflection check to remove traces that contradict the label or guideline.
  • Train judges with SFT on trace+label pairs, then apply GRPO with cost‑sensitive rewards for asymmetric cost tasks (e.g., penalize false positives more than false negatives).
  • Cost‑sensitive GRPO variant: center rewards by group mean but omit group standard deviation scaling so the absolute reward gap (H−L) is preserved in learning signals.
• Concrete reward example (ad triggering)
  • Reward matrix R(y, ŷ): when true y = Yes/No and prediction ŷ = Yes/No, the chosen matrix encodes stronger penalty for false positives (e.g., −2) than false negatives (e.g., −1).
• Empirical gains reported
  • Offline: diversity +60%, relevance +80% vs current production ad serving.
  • Online A/B: RPM +22%, ads coverage +74%.

Data & Methods

• Supervised trajectory generation for Orchestrator:
  • Each trajectory contains intermediate traces: state resolution, constraint extraction, intent generation, tool calls, inspection of candidates, reflection and final slate selection. These teach decomposed decision processes (important for follow‑ups and under‑specified turns).
• Orchestrator training
  • Stage 1 SFT on formatted trajectories to ensure valid tool calls and constraint preservation.
  • Stage 2 agentic RL using GRPO where complete trajectories (including tool outputs) are sampled and scored by Evaluator; GRPO rewards trajectories that yield better slates (so intermediate intent choices are optimized for downstream outcomes).
• Evaluator training and labels
  • Relevance and trigger judgments use label grounded thinking traces produced from human-labeled examples (guideline + generated trace) and reflection filtering to remove inconsistent traces.
  • SFT followed by GRPO (cost‑sensitive variant) trains the judgment models. Rewards incorporate correctness, format validity, and guideline consistency.
• Tool improvement loop (Algorithm 1)
  • 1) Train initial Orchestrator from SFT.
  • 2) Iteratively: sample rollouts with current tools, score slates with Evaluator, update Orchestrator (GRPO); generate rollouts, compare high vs low reward rollouts for same conversation to construct preference tuples (intent, a+, a−); update tools with those preferences (DPO/ranking losses).
• Objective metrics
  • Conversation→ad relevance (learned evaluator), slate diversity (penalize near duplicates by cosine similarity), production metrics (RPM, ads coverage) for online experiments.

Implications for AI Economics

• Marketplace efficiency and revenue
  • Better intent extraction + slate construction increased RPM by 22% in A/B tests, suggesting that improved conversational matching (not just raw bidding) can substantially increase platform monetization per impression. More accurate, context-aware ad serving can raise advertiser ROI and publisher revenues.
• Trade-offs: user experience vs short‑term revenue
  • Opportunity Gate and cost‑sensitive training explicitly encode asymmetric social costs (false positive ad exposures can erode trust more than missed ad opportunities). Economically, this reflects a platform optimizing for long‑run user engagement and lifetime value rather than only instant revenue — a key consideration in platform welfare models.
• Dynamics of supply, demand, and pricing
  • As retrieval/ranking tools evolve to surface different candidates, the effective set of impressions and click probabilities change. This can alter advertisers’ bidding strategies and equilibrium prices in auctions. Co‑evolving tooling may shift which advertisers win and at what prices, potentially changing auction revenue and advertiser surplus.
• Strategic behavior and externalities
  • More effective conversational ads could induce strategic behavior by advertisers (e.g., optimizing landing pages or creatives to match Orchestrator intent patterns). Platforms must monitor for manipulations of signals/tools. Externalities include ad‑fatigue, privacy concerns if more contextual signals are used, and potential amplification of certain advertisers.
• Measurement and policy implications
  • The label‑grounded approach highlights that production judgments are value‑laden (asymmetric costs). Economists studying platform policy should treat objective metrics and reward formulations as design levers that influence welfare trade‑offs (trust vs revenue). Regulatory scrutiny may focus on opacity of learned gates and auction impacts.
• Research directions for AI economics
  • Model the system as a multi‑agent game among Orchestrator (platform), advertisers, and users; study steady states when tools and advertisers co‑adapt.
  • Quantify long‑run effects: retention, conversion, advertiser ROI, and distributional effects across advertisers (who wins/loses as tooling shifts).
  • Welfare analysis comparing strict revenue optimization vs cost‑sensitive, trust‑preserving objectives; estimate social welfare changes from different reward matrices.
• Operational costs and barriers
  • Running an LLM-driven Orchestrator plus iterative tool retraining has compute and measurement costs; smaller platforms must weigh benefits vs engineering expense. Privacy and permitted context signals constrain available information, altering the achievable gains and economic outcomes.