Model performance frontiers map predictably to pre-training compute: at 10^24 FLOPs estimated top-tier accuracies are about 83% on IFEval and 54% on advanced math, and four of six task boundaries remain temporally stable while math reasoning continues to improve; a new sampling scheme recovers these frontiers using only 5–20% of evaluation effort.

Main Finding

Prescriptive scaling—estimating high conditional quantiles of post-trained benchmark accuracy as a function of log pre-training FLOPs—yields a simple, reliable, and practically useful compute→performance map. A monotone, saturating sigmoid in log-FLOPs approximates the attainable upper envelope of post-trained model performance (τ = 0.98) across many tasks, with good out-of-distribution calibration for most benchmarks. However, some tasks (notably math reasoning) show a moving boundary over time, indicating evolving capability ceilings.

Key Points

• Objective: estimate capability boundaries qτ(z) ≈ Qτ(Y | Z = z) where z = log10(pre-training FLOPs) and τ = 0.98, giving a conservative, decision-oriented mapping from pretraining compute to attainable post-trained accuracy.
• Data:
  • ≈7k model checkpoints aggregated (≈5k from Open LLM Leaderboard + ≈2–2.4k newly evaluated open-weight models, "PROTEUS-2K"), covering releases from mid-2022 through March 2026.
  • Six benchmarks: IFEval, BBH, MATH Lvl 5, GPQA, MUSR, MMLU-PRO.
  • Chronological splits P1–P4 (mid-2022 → Mar 2025) used for temporal validation.
• Main quantitative examples (estimated 0.98-quantile sigmoid boundary at 1024 FLOPs):
  • IFEval: 0.828
  • BBH: 0.700
  • MATH Lvl 5: 0.539
  • GPQA: 0.424
  • MUSR: 0.535
  • MMLU-PRO: 0.563
• Estimator performance:
  • Sigmoid boundary (monotone, saturating) matches more flexible I-spline in in-distribution pinball loss and attains best OOD calibration.
  • OOD calibration error: sigmoid ≈ 2.21% vs compute-agnostic constant baseline ≈ 3.60% (average over tasks and rolling splits).
• Temporal stability:
  • For 4/6 tasks (BBH, GPQA, MMLU-PRO, MUSR) the sigmoid boundary transfers to next-period models with coverage error within ±2% (stable predictability).
  • Math reasoning (MATH Lvl 5) shows consistent upward drift in the boundary, especially at high compute.
• Pretrain vs post-train:
  • Pretrained base models often lie below the post-trained capability boundary; gap is task dependent (larger for reasoning/instruction tasks).
  • Post-trained capability boundary is much more monotone in compute than raw pretrained accuracies.
• Robustness checks:
  • Contamination analysis on frontier models found no clear evidence of benchmark inflation (e.g., AIME-2025) attributable to contamination.
  • PCA of multi-benchmark space: top 3 PCs explain ≈95% variance; compute progress concentrated mainly on the first PC.
• Efficient monitoring:
  • Introduced a balanced I-optimal sampling design that recovers near-full-data capability frontiers using ≈20% of the parameter-count–weighted evaluation budget on average (as low as ≈5% on some tasks), enabling practical continuous monitoring.

Data & Methods

• Data sources:
  • Open LLM Leaderboard v1 & v2 (~5k evaluated checkpoints).
  • Public leaderboards for some frontier models.
  • PROTEUS-2K: newly evaluated ≈2.4k open-weight models (April 2024–Mar 2026) evaluated with the same pipeline.
• Modeling target:
  • qτ(z) = conditional τ-quantile of observed post-trained accuracy given z = log10(pretraining FLOPs), with τ = 0.98 (high upper tail).
• Parameterization:
  • Default estimator: sigmoid form qsigτ(z; θ) = y0 + L · σ(a + β z), with β ≥ 0, y0 ∈ [0,1], L ∈ [0, 1−y0], σ the logistic function. Monotone and saturating by design.
  • Comparators: constant baseline, binwise constant, I-spline (more flexible monotone family).
• Loss / estimation:
  • Smoothed pinball loss (quantile regression): ℓτ(u) = (1/κ) log(1 + e^{κ u}) + (τ − 1) u, with κ = 50; L(θ) includes ℓτ over observations + λΩ(θ), λ = 1e-3.
• Evaluation metrics:
  • Pinball loss (quantile accuracy) in-distribution (ID) and out-of-distribution (OOD, train on Pt, test on Pt+1).
  • Coverage error: empirical fraction of observations below predicted boundary per compute bin minus τ (signed).
• Temporal validation:
  • Rolling chronological train/test fits (Pt → Pt+1) to measure boundary drift and OOD calibration.
• Sampling / efficiency:
  • Balanced I‑optimal design: chooses a cost-aware subset of models (parameter-count–weighted cost) to evaluate so the fitted boundary approximates the full-data boundary with far fewer evaluations.
• Releases:
  • Proteus-2k dataset and codebase for replication / monitoring (paper indicates dataset/code/blog release).

Implications for AI Economics

• Quantifying returns to pre-training compute under contemporary post-training practice:
  • Prescriptive scaling gives a tractable way to translate a pre-training FLOPs budget into a conservative estimate of attainable downstream accuracy (upper-tail). That enables more informed cost–benefit calculations for scaling investments (how much extra compute to allocate to reach a target accuracy with high probability).
• Investment and procurement decisions:
  • Organizations can use capability boundaries to decide whether to invest in larger pretraining runs, buy a larger base model, or instead invest in post-training pipelines (instruction tuning, RLHF) depending on task-specific pretrain→post-train gaps. For knowledge tasks, pretraining may suffice; for reasoning/instruction tasks, post-training can provide large gains.
• Forecasting, project planning, and budgeting:
  • For tasks with stable boundaries (4/6 tasks here), compute-driven forecasts are relatively reliable — useful for planning model roadmaps, procurement lead times, and expected capability delivered per dollar of compute.
  • For tasks with moving boundaries (math), boundaries must be treated as time-varying: R&D forecasts should incorporate monitoring and rapid re-estimation to avoid underbudgeting or overestimating returns to compute.
• Risk management and monitoring:
  • The quantile-boundary framework is naturally conservative (upper-tail of observed ecosystem) and can serve as an operational monitoring tool: systematic under-coverage in new model releases signals the arrival of a qualitatively new recipe or family that shifts attainable capabilities, which has implications for regulatory surveillance, procurement risk, and competitive dynamics.
  • Efficient sampling design reduces monitoring costs, making continuous market surveillance (for capability surprises) economically feasible.
• Pricing and market structure:
  • Because progress concentrates along a low-dimensional axis strongly tied to compute (first PC), markets that price compute or large base models can reasonably benchmark expected capability gains—until recipe innovations decouple performance from pretraining FLOPs.
• Limitations and caution for economic use:
  • Estimates are observational and conditional on the observed ecosystem of architectures, training recipes, and post-training practices; they can be exceeded if a new family or recipe systematically outperforms at fixed compute.
  • The boundaries should be refreshed frequently; for tasks with temporal drift (math), relying on stale boundaries can produce incorrect investment signals.
  • The method provides an empirical envelope (what has been attainable) rather than a causal theory of returns to compute; use it for conservative planning and monitoring rather than definitive causal attribution.

Overall, the paper provides a practical, empirically grounded toolkit (sigmoid capability boundaries, temporal validation, efficient sampling) for turning pretraining compute budgets into actionable expectations about attainable post-trained performance—information directly useful for R&D budgeting, procurement, market forecasting, and risk monitoring in AI economics.