A new solvable model links training to sampling returns: increased training steepens the pass@k success curve until an intrinsic difficulty tail caps gains; the result yields a simple compute-allocation rule that prescribes training before saturation and more inference attempts after.

Main Finding

Training reduces the “hard tail” of instance-level errors so that pass@k inference scaling steepens with more training until it saturates at an irreducible, problem-intrinsic exponent. Concretely, in a solvable last-layer linear fine-tuning model with instance-heterogeneous noise (the LID model), pass@k failure follows a two-term power-law mixture Linf(k; N) ≈ P k−β + PN(N) k−γ(N), with γ(N) = Θ(1/Lgen(N)). The observed inference exponent βeff(N) = min{β, γ(N)} therefore grows with training sample size N and plateaus at the intrinsic difficulty index β set by the small-τ tail of the instance-difficulty distribution. This yields a simple compute-allocation prescription: train until βeff(N) ≈ β, then prioritize inference attempts.

Key Points

• Latent Instance Difficulty (LID) model: each input x has a latent precision τx (difficulty) drawn from a heavy-tailed prior (Gamma with tail index β/2) that controls the target noise variance; targets are sampled once per train input and freshly at inference.
• Training regime: last-layer linear ridge/OLS on fixed features; standard high-dimensional results apply. Lgen(N) shows the usual regimes:
  • Overparameterized (N < d): Lgen ∝ N−α (spectrum-controlled).
  • Underparameterized / variance-limited (N ≫ d): Lgen ∝ d/N (1/N tail).
• Inference scaling (pass@k):
  • Bias-free asymptotic: if model bias is negligible, pass@k failures decay as k−β with β set by the small-τ tail of the difficulty prior.
  • Finite-N correction: when model bias BN(x) is non-negligible, a second power-law term appears with exponent γ(N) = Θ(1/Var[BN(x)]) = Θ(1/Lgen(N)). Thus Linf mixes the intrinsic tail (β) and a training-dependent tail (γ(N)).
  • Effective exponent: over practical k-windows, βeff(N) = min{β, γ(N)} and so βeff increases with N (steepening pass@k) until it saturates at β.
• Testable predictions:
  • A crossover (N, k) surface separating bias-dominated (finite-N) and intrinsic-tail regimes.
  • A monotone, saturating βeff(N) curve.
  • Even after βeff saturates, prefactor improvements with N remain (i.e., continued gains from training but not steeper slopes).
• Compute-allocation implication: with a fixed compute budget split between obtaining more training samples and performing more inference trials, optimal allocation shifts toward training while βeff(N) is below β, then toward inference once βeff ≈ β. The paper derives analytic approximations for the optimal split under simple cost models.

Data & Methods

• Analytic model:
  • Features x ∈ R^d drawn with covariance eigenvalues decaying as σ2_j ∝ j−(1+α).
  • Latent precision τx ∼ Gamma(shape = β/2, rate = 1) (heavy near-zero tail → hard instances).
  • Targets Yx ∼ N(x⊤θ, σ2_η / τx); training observes a single y per x; inference draws k fresh y’s per test x and uses a perfect verifier with tolerance δ.
  • Learner fits last-layer linear head by ridge/OLS; evaluate generalization Lgen(N) = E[(x⊤ˆθ − x⊤θ*)2].
• Mathematical techniques:
  • High-dimensional ridge/OLS asymptotics to get Lgen(N) in the Nd regimes (spectral exponent α enters the under-sampled tail).
  • Tauberian theorems and Laplace–Stieltjes transforms to analyze the small-τ tail contribution to pass@k and derive the k−β asymptote.
  • Asymptotic expansions of Gaussian CDFs for finite-N bias corrections, leading to the PN k−γ(N) term and the scaling γ(N) ∝ 1/Lgen(N).
• Empirical validation:
  • Controlled simulations of the LID linear model confirm the predicted Lgen tails, the k−β asymptote in the bias-free limit, the two-tail mixture, and a βeff(N) curve that rises and plateaus.
  • Two real-data proxies:
    • CIFAR-10H: uses human label variance to proxy instance heterogeneity; observed pass@k behavior is consistent with the LID predictions.
    • Teacher–student distillation on GSM8K (math problems): a teacher produces stochastic targets and a student is trained; behavior of βeff(N) with training matches theoretical trends.
  • Practical fitting: βeff(N) empirically fitted by β − Δ/(1 + cβ Nν) in examples.

Limitations & modelling assumptions - Linear last-layer model with frozen features (analytic tractability) — nonlinear end-to-end models may change constants and crossover scales. - Independence assumed between τx and x; correlation would modify prefactors and crossover loci but not the tail exponents driven by small-τ behavior. - Perfect verifier and small-tolerance (δ) expansions simplify derivations; real verifiers and larger tolerances affect constants. - Analysis relies on heavy-tail behavior of difficulty prior; the Gamma choice is for convenience — exponent-level conclusions hold for other tails with the same index. - The training-scaling result assumes finite average target variance (β > 2) for classical 1/N regime; inference-exponent statements (β) depend only on the near-zero τ tail and are meaningful even when β ≤ 2.

Implications for AI Economics

• When to buy more data (training) vs. spend inference compute:
  • Early-to-mid training regime (βeff(N) < β): additional training both lowers average error and steepens pass@k scaling (increases γ(N)), so buying more data/training is relatively more valuable.
  • After βeff(N) ≈ β: slope gains saturate — further training yields only prefactor improvements (smaller absolute failure rate but no steeper k-scaling). At this point, allocating compute to more inference attempts (larger k) can be more cost-effective.
• Cost-effectiveness & pricing:
  • The model provides a quantitative rule: invest in training until the inferred γ(N) (≈ 1/Lgen(N)) reaches the intrinsic β; after that, marginal return on data (in terms of improving pass@k slope) declines.
  • This informs pricing/valuation of labeling or dataset acquisition: willingness to pay for extra training data should account for whether it meaningfully raises βeff towards β.
• Procurement and benchmarking:
  • Benchmarks relying on pass@k should report not only absolute pass@k but how βeff changes with training/data—otherwise comparisons can conflate models at different points on the βeff(N) curve.
  • When evaluating inference-cost strategies (e.g., sampling, reranking), consider where the model lies relative to the crossover surface: inference compute yields bigger marginal gains when βeff already near β.
• Operations and deployment:
  • For applications with substantial irreducible instance stochasticity (small β), expect a hard wall: even with unlimited training, pass@k scaling cannot surpass k−β. Planning for infrastructure and cost should reflect that diminishing slope limit.
  • For tasks with heavy human-label variance (e.g., ambiguous labels), investing in reducing label noise (better labels, consensus labelling that reduces small-τ mass) can shift β upward and materially improve inference scaling.
• Policy & investment prioritization:
  • Strategic investments (data collection, annotation quality) that change the intrinsic difficulty tail (β) have longer-term value than increasing training size alone once βeff has saturated.
  • Conversely, if intrinsic β is small (very heavy tail), practitioners may prefer investing in smarter inference strategies (sampling, search, external verifiers) rather than huge additional training.

Overall, the LID model gives a compact, testable framework linking training progress to inference-time scaling and yields clear, operational guidance on how to allocate compute and data budgets depending on whether observed pass@k slopes are training-limited or intrinsically limited.