A wide 'near-optimal' range of SFT–RL annotation splits delivers near-peak LLM performance and expands with model size, and regions discovered on small proxy models reliably transfer to larger models—cutting the need for costly large-scale allocation searches.

Main Finding

The paper reframes SFT–RL post-training allocation as a near-optimality problem and demonstrates that the set of allocations achieving within a small relative tolerance (ε, e.g., 2–10%) of peak performance is typically wide, grows with model scale, and transfers reliably from small proxy models to larger target models. Practically, this means inexpensive small-model experiments can identify a transferable near-optimal region of SFT vs. RL annotation allocation, avoiding costly exhaustive searches at large scale. The result holds across tasks, model families, off-policy (DPO) and on-policy (GRPO) RL methods, and under heterogeneous annotation costs.

Key Points

• Near-optimal region concept: Instead of seeking the single argmax allocation r, define Rε = {r : P(r) ≥ (1−ε)P}. Small ε (2–10%) already admits many allocations.
• Empirical regularities:
  • Even for tight tolerances, the near-optimal region often covers a large fraction (e.g., 55–75% at ε=10%) of possible allocations.
  • Region width generally increases with model size N (more flexibility in allocation for larger models).
  • Near-optimal regions discovered on small proxy models transfer well to larger targets; transfer improves with ε.
• Robustness across settings:
  • Tasks: math (GSM8K), instruction following (IFEval), summarization (ROUGE-L), helpfulness (reward-model based).
  • Model families: Llama 3, Qwen 2.5, Qwen 3 (scales up to 14B reported).
  • RL methods: off-policy DPO and on-policy GRPO produce qualitatively similar patterns.
• Annotation-cost asymmetry:
  • When SFT examples are more expensive than RL/preference examples (cost ratio ρ = cSFT/cDPO ∈ {1,2,5,10}), qualitative scaling and transfer patterns persist.
  • As SFT becomes relatively more expensive, the near-optimal region shifts (and typically widens), favoring allocations that reduce SFT sample counts in dollar-budget terms.
• Practical recommendation: run small-proxy experiments to estimate Rε (with a modest ε, e.g., 5%); choose any allocation within Rε for large-scale post-training to save annotation search cost.

Data & Methods

• Problem setup:
  • Sequential two-stage post-training: SFT (rB samples) then RL ((1−r)B samples), performance metric P(N,r,B).
  • Near-optimal region Rε defined relative to peak P*; grid-restricted estimates used in experiments.
• Models and tasks:
  • Families: Llama 3, Qwen 2.5, Qwen 3; scales up to 14B parameters.
  • Tasks/datasets: GSM8K (math), Tülu3 persona/instruction (instruction following), Reddit TL;DR / comparison (summarization), HelpSteer datasets (helpfulness).
  • Evaluations: GSM8K accuracy, IFEval accuracy, ROUGE-L F1, reward-model scores.
• Training:
  • SFT via next-token prediction on demonstration pairs.
  • RL-stage: off-policy DPO (preference pairs) and on-policy GRPO (group-relative PPO-style) depending on experiment.
  • LoRA used for compute efficiency; training hyperparameters in appendix.
• Allocation grid and budgets:
  • Allocation grid G = {0.00, 0.25, 0.50, 0.75, 1.00} (some checks with denser grid reported).
  • Budgets measured in annotated samples; analysis focused on stable regime B ≥ 5k (exclude noisy pre-convergence).
• Estimators and transfer metrics:
  • Range width w_rngε = max(Rε) − min(Rε) and count width (fraction of grid points).
  • Transfer from proxy Ns to target Nt: Tε(Ns→Nt,B) = |Rε(Ns) ∩ Rε(Nt)| / |Rε(Ns)|.
• Cost asymmetry experiments:
  • Monetary budget B with unit costs cDPO = $0.001 and cSFT scaled via ρ ∈ {1,2,5,10}.
  • Convert dollar allocation to sample counts: nSFT = rB/cSFT, nDPO = (1−r)B/cDPO.

Implications for AI Economics

• Lower exploration (annotation) cost via proxy experiments:
  • Organizations can reduce expensive large-scale allocation sweeps by using small-model proxy runs to find a transferable Rε, saving annotation dollars and time.
  • The approach converts an expensive grid search at target scale into a cheap proxy identification + single large-scale run within the identified region.
• Budgeting and procurement:
  • Because the near-optimal region is wide, procurement decisions (how many SFT vs. RL annotations to buy) are less brittle—buyers can accept any allocation inside Rε without large performance loss, simplifying contract/spec design.
  • When SFT annotations cost more, optimal allocations shift predictably toward relatively more RL/preference data in dollar terms. Procurement strategies should therefore price and purchase annotations with these trade-offs in mind.
• Market structure for annotation services:
  • Demand-side implications: preference-data providers (cheaper RL labels) may see relatively higher demand when SFT is expensive, while high-quality SFT demonstrations remain important but purchased more conservatively.
  • Suppliers can target offerings: e.g., sell bundles calibrated to position buyers within likely Rε bands rather than aiming for a single “optimal” mix.
• Risk management and robustness:
  • Using near-optimal regions lowers the risk of misallocating annotation budget due to sensitivity to task or method—important for firms deploying costly post-training pipelines.
  • However, tolerance choice ε is an explicit trade-off between acceptable performance loss and further reduction in search cost; economic decision-making should set ε based on value of incremental performance.
• Limitations and caveats for economic application:
  • The empirical grid used is coarse; extreme or highly nonstandard tasks, very small budgets (B < 5k), different RL objectives, reward design failures, or very large models beyond evaluated scales may behave differently.
  • Infrastructure and compute costs were treated as negligible versus annotation—this holds under current annotations-as-dominant-cost assumptions but may shift with cheaper annotation pipelines or different training regimes.
  • Transferability is probabilistic, not guaranteed; buyers should validate the proxy-to-target transfer on a small holdout run when stakes are high.
• Operational recommendation:
  • For applied budgeting, run a small-proxy sweep (cheap models) to estimate Rε for a chosen ε (commonly 5%), then allocate large-scale annotation purchases to any allocation in Rε, adjusting for SFT vs RL unit prices using the cost-ratio mapping.

Limitations to watch: coarse allocation grid, B ≥ 5k stability assumption, LoRA and specific RL algorithms used, and extrapolation beyond tested model scales/tasks should be validated case-by-case.