A two-stage LLM bidding system that pairs few-shot plan generation with a precision optimizer lifts advertisers' cumulative value under budget constraints in offline tests, outperforming prior RL and heuristic methods; real-world deployment effects remain to be validated.

Main Finding

DARA (Dual-phase Adaptive Reasoning and Allocation) is a two-stage LLM-based framework for few-shot budget allocation in online advertising that combines LLM in‑context reasoning with RL fine‑tuning (GRPO‑Adaptive). By (1) decomposing the task into a Few‑shot Reasoner that generates initial budget plans and a Fine‑grained Optimizer that refines plans from feedback, and (2) fine‑tuning LLMs with an adaptive reference KL‑regularized RL algorithm, DARA achieves better cumulative advertiser value under budget constraints than prior baselines in both real-world and synthetic environments.

Key Points

• Problem framing

  • Goal: allocate fixed total budget B across T time periods to maximize total return (equivalently ROI), subject to ∑t b_t = B and b_t ≥ 0.
  • Assumption: per‑period value functions v_t(b_t) are differentiable, strictly increasing and concave → optimality implies equal marginal ROI across periods. Thus objective can be approximated by minimizing variance of marginal ROI across periods.
  • Practical regime: few-shot / cold‑start personalized advertisers with limited historical episodes.

• Architectural idea (DARA)

  • Dual‑phase decomposition:
    • Few‑shot Reasoner: uses LLM in‑context prompting on a small set of historical episodes H to produce an initial allocation vector. Emphasizes generalization from few examples.
    • Fine‑grained Optimizer: receives the initial plan and marginal ROI feedback and performs local, numerically sensitive adjustments; operates with a sliding window of recent episodes.
  • Separation aligns model capabilities: LLMs’ in‑context strengths for strategy + an optimizer tuned for numeric sensitivity and feedback adaptation.

• RL fine‑tuning: GRPO‑Adaptive

  • Builds on group relative proximal optimization (GRPO) style KL regularization for stable LLM tuning.
  • Key innovation: periodically update the reference policy (the KL anchor) during training instead of keeping it static; this preserves and improves structured reasoning and numeric precision that otherwise deteriorate.
  • Objective: maximize expected discounted cumulative reward under budget constraints (see paper Eq. 9).

• Environments & training data

  • Real‑world environment: built from enterprise ad data (marginal ROI curves observed across time).
  • Synthetic environment: controllable MROI functions (polynomial or exponential) that are monotone decreasing and clipped at zero, enabling dynamics randomization / sim‑to‑real style variation to improve robustness.
  • Few‑shot prompt design: structured prompt including task objective, few-shot data H (episodes of allocations and observed marginal ROIs), trial records, and explicit output format.

• Empirical findings

  • DARA outperforms RL baselines (e.g., Q‑MCKP, hierarchical RL / HiBid, ABPlanner) and single‑stage LLM prompting on cumulative return under budget constraints.
  • Ablations show single LLM prompting hits a performance ceiling; the dual‑agent decomposition plus GRPO‑Adaptive is necessary to achieve both generalization and numerical precision.
  • Simulation environments provide diverse scenarios to improve robustness to varying advertiser dynamics.

Data & Methods

• Data

  • Real-world: enterprise advertising logs used to construct per‑period marginal ROI (MROI) curves and historical episodes.
  • Synthetic: parameterized functional MROI models (polynomial/exponential) producing monotone decreasing marginal ROI over budget; used to generate many diverse allocation scenarios.

• Problem input / few-shot dataset

  • H = {(b^(i), m^(i))}_{i=1..n} where b^(i) ∈ R^T is an allocation episode and m^(i) = MROI^(i) ∈ R^T are marginal ROI samples observed for that episode.
  • Task: given H and budget B, produce b (length T) with ∑ b_t = B minimizing variance of MROI(b).

• LLM prompting

  • Structured prompt template: task objective, few‑shot examples, prior trial records, required output format (allocation vector + reasoning).
  • Few‑shot Reasoner uses this prompt to produce an initial proportional allocation.

• Dual‑agent execution (Algorithmic workflow)

  • Step 1: Few‑shot Reasoner generates b^(1). Observe r^(1) = MROI feedback.
  • Step 2: Initialize sliding window W of recent (w−1) records plus (b^(1), r^(1)). Repeatedly call Fine‑grained Optimizer with encoded W to produce refined allocations b^(s), observe feedback, and update W.
  • Terminate after specified episodes or convergence.

• RL fine‑tuning (GRPO‑Adaptive)

  • Policy π_θ(b | prompt) parameterized by an LLM producing allocation tokens.
  • Reward R(b) measures cumulative return (and enforces budget constraint).
  • GRPO‑A uses KL regularization against a reference policy π_{θ0}, but periodically updates π_{θ0} to a more recent checkpoint to avoid stale anchoring.
  • This dynamic reference reduces policy drift while allowing meaningful improvements in numeric/symbolic precision.

• Evaluation metrics

  • Primary: cumulative advertiser return (or ROI) achieved under the budget constraint.
  • Secondary: variance of marginal ROI across periods (proxy for optimality condition), robustness across environment variations, sample efficiency in few‑shot settings.

Implications for AI Economics

• Practical personalization with low data: DARA demonstrates a viable pathway to personalize budget allocation for advertisers with very limited historical data by leveraging LLM in‑context learning combined with targeted RL fine‑tuning. This can reduce the amount of live exploration advertisers must endure to get good policies.
• Hybrid LLM + RL is valuable for numeric economic decisions: LLMs bring fast adaptation and interpretability (reasoning traces), while RL fine‑tuning (with adaptive reference anchoring) injects the numerical precision and feedback sensitivity needed for economic optimization tasks. This hybridization is likely useful across other economic control problems with scarce data and structural constraints.
• Simulation + dynamics randomization as economic policy tool: constructing realistic synthetic environments and randomizing dynamics can lower sample requirements and increase policy robustness when live experimentation is costly or risky—important for market mechanisms where poor early policies can have financial or platform‑level side effects.
• Mechanism & market effects to consider
  • Deployment of high‑quality automated budget allocators can change auction dynamics (e.g., bid shading, competition intensity), potentially altering equilibrium outcomes. Platforms and advertisers should monitor aggregate effects.
  • Incentives and strategic behavior: if many advertisers adopt adaptive LLM‑based allocators, equilibrium bidding strategies could shift; mechanism designers may need to revisit reserve pricing, pacing, or auction formats.
• Research directions for AI economics
  • Multi‑agent and market equilibrium analysis: extend DARA to competitive multi‑advertiser settings to study systemic effects.
  • Incentive alignment & regulatory concerns: ensure allocators do not exploit undesirable dynamics (e.g., discriminatory targeting, collusion).
  • Integration with mechanism design: design auction mechanisms robust to adaptive LLM agents and that preserve welfare/fairness goals.
  • Live A/B and long‑horizon evaluation: evaluate economic externalities, learning dynamics, and platform revenue impacts in production deployments.

Summary takeaway: DARA shows that structurally decomposing few‑shot economic decision problems and combining LLM in‑context capabilities with an RL fine‑tuning approach that adaptively anchors policy updates yields a practical, robust path to numerically precise, few‑shot budget allocation—raising both opportunities and important system‑level questions for online advertising markets and AI economics more broadly.