A simulated-persona evaluator turns synthetic click intent into a smooth, reliable signal for LLM-native ads, outperforming zero-shot judges and generalising to unseen products; its continuous output also enables incentive-compatible pricing and best-of-N selection without historical click logs.

Main Finding

The paper builds a reproducible, continuous, differentiable estimator of click-through intent (CTI) for advertisements embedded in LLM-generated answers by (1) creating synthetic CTI supervision via persona-based agent simulation grounded in Big Five priors and decomposed ad features, (2) distilling that supervision into a parameter-efficient shared‑bottleneck neural evaluator (frozen Qwen3-4B + LoRA) trained with an ordinal Earth Mover’s Distance loss, and (3) using the resulting smooth intent signal to derive incentive‑compatible pricing (a Truthfulness‑in‑Expectation payment rule) and Best-of‑N allocation procedures. The evaluator outperforms zero‑shot LLM judges on relevance sensitivity, generalises semantically (including to fictional products), and provides the differentiable signal needed for both truthful auctions and gradient‑based ad generation.

Key Points

• Problem: LLM-native ads lack behavioral click logs and human CTI labels are highly uncalibratable; zero-shot LLM judges conflate fluency with intent.
• Label construction: persona-agent framework:
  • Stage 1: objective scoring of six ad features (relevance, copy quality, novelty, credibility, urgency, warmth) on 1–5 scale (K=3 calls averaged).
  • Stage 2: sample K_personas = 30 from Big Five distributions; modulate the objective features with signed trait–feature weights, combine via tanh and a relevance gate to produce per-persona CTI; final label is persona mean.
  • Design choices ensure reproducibility, bounded personality effects, and relevance dominance (irrelevant ads cannot score high).
• Evaluator architecture:
  • Frozen Qwen3-4B backbone adapted with rank-16 LoRA; pooled representation → shared bottleneck → four dimension-specific heads (relevance, coherence, ad-effectiveness, CTI).
  • CTI head trained on agent labels; Q1–Q3 trained on NaiAD VC‑PPI human labels.
  • Loss: squared Earth Mover’s Distance (EMD) to respect ordinal geometry and produce smooth expected scores.
  • Shared bottleneck chosen after ablations (was the only stable, effective design); flow‑matching auxiliary head gave no benefit.
• Main held-out performance (n = 5,837):
  • Pearson correlations: Q1 relevance 0.85, Q2 coherence 0.73, Q3 ad-effectiveness 0.55, Q4 CTI 0.82.
  • σpred/σlabel ratios indicate realistic prediction variance (CTI ≈ 0.76).
• Behavioural (sign-certain) perturbation tests:
  • Cross-category swap: evaluator lowers CTI in 79% of cases (vs. 33% for same-backbone zero-shot judge; Sonnet 4.6 = 67%, GPT-5.5 = 60%).
  • Graded content degradation: evaluator yields smooth, monotone dose–response across five corruption levels; zero-shot judges collapse to floors or act like rigid classifiers.
  • Keyword stuffing and input ablation tests confirm robustness to gaming and correct fusion of query+ad.
  • Quintile calibration: mean true label rises monotonically across predicted quintiles.
  • Fictional products: for 103 fabricated categories disjoint from training, evaluator correctly rates relevant placement higher in all 103 cases (semantic generalisation with zero token overlap).
• Human validation and Best-of‑N:
  • Pairwise agreement: evaluator agrees with majority human preference in 86% of pairwise judgements across five annotators, with agreement increasing with evaluator confidence.
  • Best‑of‑N selection: under high-confidence conditions, evaluator’s selection agrees with human majority 96% of the time.
• Pricing / mechanism contribution:
  • Using the continuous CTI estimate, the authors derive the unique payment rule under which truthful bidding is optimal (Truthfulness‑in‑Expectation), demonstrate it for Best‑of‑k allocation, and extend the approach to non‑monotone allocations.
• Practical properties: deterministic, differentiable, computationally efficient at inference (single forward pass), suitable as a training objective for ad generation.

Data & Methods

• Dataset: NaiAD benchmark of 58,999 advertisement‑embedded responses; zero-overlap train/validation with n = 5,837 held-out items for evaluation.
• Supervision:
  • Q1–Q3: NaiAD’s variance‑calibrated prediction-powered inference (VC‑PPI) human labels.
  • Q4 (CTI): agent-grounded synthetic labels from persona simulation (K_personas = 30; six continuous features; λ = 0.5 personality strength; relevance gating).
• Model:
  • Frozen Qwen3-4B backbone with rank‑16 LoRA adapters; pooled output → shared bottleneck → four heads.
  • Trained with DeepSpeed ZeRO-2, mixed precision, cosine LR schedule; separate learning rates for LoRA and heads (full hyperparameters in supplement).
• Loss & output:
  • Squared Earth Mover’s Distance (EMD) loss per head to capture ordinal structure; outputs smooth expected scores on 1–5 scale.
• Experimental tests:
  • Held-out fit (Pearson/Spearman/RMSE/σ ratios).
  • Six sign-certain perturbation tests (cross-category swap, graded degradation, keyword stuffing, quintile calibration, input ablation, fictional-product generalisation).
  • Baselines: zero-shot Qwen3.6-35B-A3B, Claude Sonnet 4.6, GPT-5.5.
  • Human pairwise studies for Best‑of‑N.

Implications for AI Economics

• Enables principled pricing in LLM-native ad markets:
  • Provides the continuous, monotone(izable) CTI mapping from generated response to expected click probability required by classical mechanism design (e.g., Myerson‑style truthfulness proofs).
  • The differentiable CTI signal permits deriving a unique payment rule that makes truthful bidding optimal in expectation (Truthfulness‑in‑Expectation), and supports Best‑of‑N allocation mechanisms and extensions to non‑monotone allocation settings.
• Closes the loop between generative optimization and marketplace design:
  • Because the CTI estimator is differentiable and cheap to run, it can be used as a training objective or reward for gradient-based ad generation (enabling advertiser-side optimization of phrasing/content under market incentives).
  • It supports large candidate scoring at scale for Best‑of‑N selection, improving matching between ad copy and user queries.
• Market design and welfare considerations:
  • A reliable, continuous intent estimator reduces informational asymmetries in cold‑start regimes (no historical logs), potentially improving allocation efficiency and advertiser surplus.
  • The persona-grounded construction may help designers evaluate heterogeneous consumer responsiveness and price discrimination, but this requires cautious translation to real populations.
• Risks and open limitations (important for economic deployment):
  • External validity: agent‑generated labels are synthetic; behavioral ground-truth click data may differ (calibration to live logs will be necessary once available).
  • Distributional assumptions: persona priors (Big Five distributions) and trait–feature weightings are literature‑grounded but remain modeling choices; systematic bias or mis-specification could distort pricing or targeting.
  • Strategic risk: once deployed, advertisers may adapt copy strategically to exploit evaluator weaknesses (necessitating continual evaluator monitoring and adversarial robustness tests).
  • Ethical/privacy concerns: persona targeting and auctionic pricing raise normative issues (microtargeting, fairness, disclosure) that require governance beyond technical design.
  • Dependence on backbone and training regime: results reported for a specific frozen-backbone + LoRA setup; portability and sensitivity to model family should be evaluated.
• Overall: the work provides a practical, testable pathway to operationalize CTI in log‑poor environments, enabling incentive‑compatible marketplaces and gradient‑based ad generation, while flagging the need for live-data calibration, robustness monitoring, and policy safeguards before large‑scale economic deployment.