A model-based offline RL system raises per-user net profit in incentivized advertising by about 8% versus a strong baseline, using a world model plus conservative Q-regularization and an independent counterfactual scorer to safely select policies before deployment. The result, validated on large-scale ByteDance logs and live A/B tests, demonstrates a practical route to automate incentive allocation but is narrowly scoped to short-session RTB-driven ad products.

Main Finding

The paper formulates incentive allocation in incentivized advertising as a new cost-sensitive sequential decision problem and proposes an Offline Model-Based Reinforcement Learning (Offline‑MBRL) framework that (i) learns a structured world model of user response and ad revenue from logs, (ii) trains conservative policies on a mix of real and model‑generated transitions, and (iii) uses an independent counterfactual evaluation scorer (CES) for safe offline policy selection. On large-scale industrial data and online A/B tests, their MB‑IQL variant increased per-user net profit substantially (MB‑IQL vs TD3+BC: +7.96%); switching to plain IQL (removing the model‑based conservative extension) decreased net profit (−6.56%). Results were statistically significant (p < 0.0001).

Key Points

• Problem setting

  • Incentivized advertising: platform promises a bonus before observing downstream RTB revenue; bonus paid only on ad completion.
  • Decisions create exposures (or not), so incentives both generate opportunities and incur delayed costs.
  • User behavior exhibits short-horizon carryover within bounded sessions (Lmax ≈ 6–20 attempts).
  • Optimization must trade off expected RTB revenue vs incentive cost and manage short-window dynamics and user expectations.

• MDP formulation

  • Each incentive attempt is a transition (s_t, a_t, r_t, s_{t+1}). Episode ends if no exposure or session cap reached.
  • Observables per attempt: z_t (exposure), y_t (completion), I_t (RTB revenue).
  • Reward: r_t = I_t − λ y_t a_t (cost paid only on completion).
  • Real-time half-life discounting is used to handle uneven inter-attempt timing.

• Offline‑MBRL architecture

  • World Model (WM): structured heads predicting
  • exposure probability p̂_z(s,a,ε),
  • completion probability conditional on exposure p̂_{y|z}(s,a,ε),
  • conditional revenue mean m̂_I(s,a,ε),
  • dynamics model predicting mutable real-time state subvector s'_*.
  • ε = request-level context (available in logs) is used by WM/CES heads but not by the policy (policy uses s only).
  • WM rollouts: short one‑step (or short) rollouts from logged states to generate synthetic transitions.

• Conservative policy optimization with mixed data

  • Train critic on mixed distribution d_f = f·D_off + (1−f)·D_wm (real + WM synthetic transitions).
  • Add a conservative regularizer (in spirit of CQL) that penalizes high Q-values on WM-proposed state–action candidates (ρ_wm) while anchoring to logged pairs to avoid broad suppression.
  • Implemented with TD3+BC and IQL backbones; MB extensions (MB‑TD3+BC, MB‑IQL) keep backbone objectives and add WM samples + conservative penalty.
  • Actor uses behavior cloning anchoring (L_DDPG+BC) to stay close to logged behavior while improving under the conservative Q.

• Counterfactual Evaluation Scorer (CES)

  • Independent model trained on held‑out users (not the WM) to predict p̂z, p̂{y|z}, m̂_I for counterfactual actions.
  • For each held-out user, replace logged a with policy action a_π and aggregate predicted revenue and cost to per‑user estimates:
  • estimated revenue/u = sum_t p̂_z(x_π)·m̂_I(x_π)
  • estimated cost/u = sum_t p̂z(x_π)·p̂{y|z}(x_π)·a_π
  • estimated net/u = revenue/u − λ_eval·cost/u
  • CES provides reusable, stable offline ranking for pre-launch policy selection.

• Empirical results & ablations

  • Validated on large-scale industrial logs and online A/B tests.
  • MB‑IQL outperformed TD3+BC in net profit per user by 7.96%; plain IQL (without MB conservative extension) performed worse (−6.56%).
  • CES rankings were consistent with online A/B outcomes, enabling safer rollout.
  • Ablations show benefits of WM augmentation and conservative regularization; also demonstrate viability on a public benchmark.

Data & Methods

• Data

  • Large-scale industrial app logs from an incentivized advertising product.
  • Each episode contains up to Lmax (≈6–20) consecutive attempts within a bounded window; logs include request-level context ε, user features (s), exposure/completion flags, RTB billing records, and timestamps.

• Modeling & training

  • WM uses MMoE-style subnetworks with three reward heads and a dynamics head; exposure head trained on all requests, completion and revenue heads trained conditional on exposures.
  • Rollouts: sample z and y from WM heads, set I to m̂_I when z=1 (else 0), sample inter-attempt elapsed times near empirical medians to compute discounts.
  • Policy learning mixes real and WM transitions; uses a temperature log-mean-exp conservative term and proposal densities (uniform and actor‑based perturbed proposals) for ρ_wm.
  • Backbones: TD3+BC and IQL (with deterministic extraction via DDPG+BC).
  • Key hyperparameters: mixing factor f between real/model data, conservative strength β, temperature T, policy‑BC weight α, and cost multiplier λ (training vs evaluation λ_eval).

• Evaluation

  • Offline: CES on held-out users to compute per-user net profit estimates for candidate policies.
  • Online: A/B tests deploying selected policies; reported statistically significant improvements (p<0.0001).
  • Ablation studies and tests on a public benchmark to validate generality.

Implications for AI Economics

• Practical monetization: The work demonstrates a deployable pipeline to optimize platform net profit in settings where actions (incentives) create monetizable opportunities with delayed payoff—common in many two-sided platforms. Modeling the creation of opportunities (not just auction-side bidding) is crucial for correct economic optimization.
• Risk‑controlled policy deployment: Combining model-based augmentation with conservative value regularization and an independent CES enables aggressive offline policy learning while controlling the financial and UX risks of online exploration—important for businesses where experimentation is costly.
• Short‑horizon behavioral externalities: The paper highlights that incentives have within-session carryover effects on user expectations/engagement; optimizers should internalize these short-run dynamics rather than treat interactions as i.i.d. This has implications for coupon/subsidy design, promotional calendars, and short-term demand shaping.
• Methodological blueprint: Integrating causal‑style counterfactual scoring (CES), structured world models that separate exposure/completion/revenue, and conservative offline RL offers a replicable approach for other economic decision problems with sparse logged actions and delayed returns.
• Limitations & considerations for policy makers/economists
  • Horizon: The model targets short-window dynamics; long-term LTV, retention, or strategic user adaptation beyond the session window are not captured and could alter welfare/profit outcomes.
  • Model dependence: WM accuracy and CES calibration critically affect policy safety; model misspecification can still bias optimization despite conservative penalties.
  • Fairness & strategic responses: Incentive policies can change user expectations and cross-product behavior—platforms should monitor distributional effects and potential gaming by users.
  • Generalizability: While validated in production and a public benchmark, domain adaptation (different ad markets, user bases, or regulatory regimes) requires careful re‑training and CES re-calibration.

Overall, the paper offers a practical, empirically validated method for profit‑aware incentive allocation in settings where incentives themselves generate monetizable opportunities, advancing both applied RL methodology and economic decision‑making for platform markets.