A decision-focused benchmark shows upgrading fine-tuned LLM specialists is context-dependent: freezing often suffices for intent classification over multiple releases, but text-to-SQL specialists can lose most of their edge in a single upgrade, and naive adapter copying fails as pretraining distance grows while label-free refresh can restore parity at lower cost.

Main Finding

UpgradeBench measures the real-world decision problem organizations face when a new base-model release arrives: keep a deployed specialist, port its adapter, refresh it via teacher relabeling, or retrain from scratch. The paper shows that (1) the value of upgrading is highly task- and episode-dependent (durability/half-life ranges from under one release to >14 months), (2) naive adapter copying (weight-space porting) is governed by weight continuity (continued pretraining distance), not merely architectural shape or family, and (3) a simple, decision-aware migration policy (using measured costs and validation) recovers nearly full quality at a small fraction of retraining compute/label cost (mean regret 0.37 percentage points at ~1/3 the retrain budget). The authors release a longitudinal benchmark, evaluation harness, per-example predictions, and cost logs.

Key Points

• Decision focus: defines and measures the upgrade tax — the quality-and-cost gap between the chosen migration strategy and the best attainable one.
• Empirical scope: four consecutive Qwen releases (plus one continuation), one controlled OLMo lineage, six tasks from three classes (intent classification, text-to-SQL/numerical-program generation, function-calling), two model scales; 98 trainings and 193 evaluation cells.
• Quantities and definitions introduced:
  • Specialization gain G(b,t) = specialist minus base score.
  • Retention Rm for a migration method m (normalized relative to retrain).
  • Upgrade tax Tm (shortfall times reference gain) plus measured costs (GPU-minutes, energy, labels).
  • Durability horizons: H50 (half-value) and H0 (crossover to zero advantage).
  • Recovery cost Cm95: labeled/GPU budget to reach 95% of attainable gain.
• Heterogeneous durability:
  • Intent classification specialists often retain performance: freezing retained 99–101% of attainable gain across nine hops.
  • Text-to-SQL specialists can obsolesce quickly: freezing forfeited up to 59% of attainable gain on a single hop and often decayed to zero within a few generations.
  • Measured half-life ~2–3 generations at 7–8B; one crossing observed at 1.5–1.8B.
• Portability depends on continuity distance:
  • On OLMo controlled edges, retention fell from 0.88–0.99 after a 46B-token continuation to essentially zero after a 2.9T-token continuation.
  • Copying between independent pretraining runs (even shape-compatible) often fails; copying between continued-pretraining descendants succeeds.
  • Annealing and model-soup merges added little extra harm beyond continuation distance.
• Refresh (input-retaining teacher relabeling) often recovers retraining parity without new gold labels across tested cases, though not always cheaper in compute.
• Decision policy: replaying a prespecified decision policy over 33 measured upgrade episodes (with disjoint validation folds) achieved mean quality regret 0.37pp with zero behavioral regressions at ~1/3 of the always-retrain compute/label budget.
• Predictive probe: a cheap representational probe (CKA over 256 prompts) ranks portability across base pairs with Spearman ρ ≈ 0.74 over eight pairs.

Data & Methods

• Lineage and compute budget:
  • Qwen family: four generations spanning Feb 2024–Apr 2025 with architectural churn; one hop (2→2.5) is shape-compatible, others not.
  • Small-scale parallel track (1.5–1.8B → 1.5B → 1.5B → 1.7B).
  • OLMo lineage used for controlled ablation where training relationships and token distances are documented.
  • Full study: 98 fine-tuning runs, 193 evaluation cells; run end-to-end on one RTX 4090.
• Task suite (six tasks):
  • Discriminative: Banking77 (intent, 77-way), CLINC150 (151-way incl. OOS).
  • Structured generation: Spider (text-to-SQL), FinQA (numerical-program generation).
  • Agentic/function-calling: xLAM-FC and glaive-FC.
• Strategies evaluated:
  • FREEZE (keep old specialist),
  • ADOPT-0 / ADOPT-K (prompt-only adoption, zero-shot and 5-shot),
  • COPY / PORTDIFF (paste LoRA adapter or diff-vector),
  • REFRESH-D (teacher relabeling using retained inputs; produce new LoRA on target base),
  • RETRAIN (full QLoRA on target base; reference denominator),
  • RETRAIN-N (label-budget ladder: N ∈ {256,1024,4096}),
  • prompt-port / prompt-retune (pilot prompt-side comparison).
• Protocol and metrics:
  • Primary numeric outputs: per-task quality (with bootstrap 95% CIs), retention Rm, upgrade tax shortfall Lm, recovery cost Cm95 (labels and GPU-minutes), H50 and H0 intervals.
  • Cost accounting: wall-clock, peak VRAM, and energy (board power sampled at 10s intervals).
  • Validation and decision replay: decisions gated and scored on disjoint halves of each test set; policy replay over 33 measured upgrade episodes.
• OLMo controlled experiments:
  • Five OLMo edges between shape-identical checkpoints with verified relationships: a fresh-pretraining pair and four continued-pretraining targets at increasing token distance (46B up to ~2.9T tokens), plus anneal and soup stages to separate continuation distance from release operations.

Implications for AI Economics

• Upgrade tax is a measurable depreciation term for specialist fleets: organizations should treat adapters and specialists as assets with varying half-lives rather than costless durable goods.
• Migration policy matters: a data-driven decision policy can achieve near-reference quality with substantially lower retraining cost (illustrated: mean regret 0.37pp at ~1/3 retrain compute/label budget). This implies material savings for high-frequency upgrade cadences.
• Portability is a distance-budgeted asset: weight-space transfer methods will succeed only if the new base lies within a contiguous pretraining neighborhood of the old base. Firms relying on naive adapter-copying across independently trained releases risk large behavioral regressions and quality loss; they should validate portability (e.g., with cheap representational probes) before deploying copied adapters.
• Task-specific planning: upgrade cadence and migration strategy should be task-differentiated. For data-bound tasks (label taxonomies, call formats, intent classification), freezing may be low-cost and durable; for base-bound reasoning tasks (complex SQL/program generation), retraining or teacher-refresh may be required quickly.
• Refresh as a practical tool: input-retaining teacher relabeling can achieve retrain parity without collecting new labels, offering a pragmatic middle ground when retained inputs are available — but organizers must weigh compute costs, since relabeling can still be expensive.
• Operational tooling suggestions:
  • Track per-specialist durability metrics and computed H50/H0 to inform policy.
  • Maintain a small validation set and lightweight probes (e.g., CKA on 256 prompts) to predict portability before attempting weight-space transfer.
  • Account for training-side energy and compute when comparing strategies; measured cost vectors (labels, GPU-minutes, energy) should enter procurement and SLA decisions.
• Broader economic consequence: as base-model capability density continues to improve rapidly, treating fine-tuned specialists as depreciable capital assets and optimizing upgrade policies yields recurring savings across organizations that operate fleets of specialists. UpgradeBench provides the common empirical foundation to price that depreciation and evaluate transfer methods in a temporally realistic setting.

References and code/harness/predictions are released by the authors (paper includes per-example predictions, cost logs, split manifests, and evaluation harness).