0 cumulative citations
View corpus contextLanguage models have become dramatically cheaper and more capable: between 2020 and mid-2026 token prices fell roughly 60× while agentic coding solve odds rose about 5.8× per year, making recent flagship performance affordable at budget-tier cost. The frontier in 2026 is task-fragmented—no single model dominates—yet simple two-model routing and inference-time sampling recover most of the practical gains.
From BERT to Frontier Agents: Eight Years of Language-Model Progress, the Collapse of the Capability-Cost Curve, and the Rise of Task-Targeted Models
Using reproducible benchmark reconstructions, pricing series, and held-out experiments, the paper shows rapid capability growth on agentic coding tasks (~5.8× annual growth in solve odds from Oct 2024–Jul 2026), a ~60× decline in input-token prices since 2020, task-level model specialization on the 2026 frontier, and modest extra gains from inference-time sampling and two-model routing.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Between October 2018 and July 2026 AI models progressed from simple systems like BERT to massive agents that solve complex math and write software. The ability to resolve real coding issues improved by nearly six times per year since late 2024. During this time costs dropped sharply with OpenAIs budget model GPT 5 point 6 Luna matching flagship capabilities for just one to six dollars per million tokens beating older versions at a fraction of the price. Top performance is now split across specialized models as Claude Opus 5 leads in frontend coding Claude Fable 5 excels at repository level coding and GPT 5 point 6 Sol dominates terminal tasks. In a grade school math test using the Qwen 2 point 5 model basic methods solved 58 of 100 problems while advanced sampling solved up to 79. A confidence ranking tool correctly identified 47 right answers in its top 50 choices proving highly useful for sorting tasks with all research materials made fully public.
Summary
Main Finding
Between 2018 and mid‑2026 language models advanced from BERT‑class encoders to trillion‑parameter, agentic systems that routinely solve real software engineering tasks and other long‑horizon professional problems. Measured on public benchmarks and API prices, capability and cost efficiency have both improved rapidly: the odds of resolving a real GitHub issue grew ≈5.8× per year (Oct 2024–Jul 2026), while input‑token prices fell ≈60× from GPT‑3 (2020) to GPT‑5.6 Luna (2026). The consequence is (1) a collapsed capability–cost curve where recent budget tiers match last‑quarter flagships on many agentic/professional tasks, and (2) a fragmented task frontier that favors task‑targeted models and simple routing between them. Inference‑time techniques (sampling + aggregation, verifier, confidence triage, routing) yield meaningful practical gains without retraining.
Key Points
- Fast agentic progress
- SWE‑bench Verified (real GitHub issues) fit: logit(p) trend implies ~5.8× growth in odds per year (R² = 0.78, n = 14) from Oct 2024 to Jul 2026. Independent harnesses put the mid‑2026 frontier at ~96–97%.
- Harder variants (SWE‑bench Pro, Frontier‑Bench) remain far from saturation and are becoming the new frontier.
- Capability–cost collapse
- API input token price fell from ~$60/M (GPT‑3, 2020) to ~$1/M (GPT‑5.6 Luna, 2026) — ~60× decline (roughly 2× per year).
- GPT‑5.6 Luna ($1/$6 input/output per M tokens) matches or beats GPT‑5.5 flagship ($5/$30) on many agentic/professional benchmarks at roughly 1/5 the token price.
- Model‑family Pareto structure is now bimodal: a volume (cheap) tier and a premium tier; the mid tier (e.g., Terra) is economically dominated.
- Fragmented frontier and routing
- No single model leads all tasks in July 2026:
- Frontend coding / human preference: Claude Opus 5.
- Repository‑level coding: Claude Fable 5.
- Agentic terminal / long‑horizon professional work: GPT‑5.6 Sol.
- Novel reasoning / IMO / ARC‑AGI‑3: Claude Opus 5 (large gains on some benchmarks).
- A two‑model router (GPT‑5.6 Sol + Claude Fable 5) captures the full per‑benchmark gain of a six‑model oracle on a 14‑benchmark suite; a cheapest‑within‑one‑point router matches oracle score at ~95% of cost.
- No single model leads all tasks in July 2026:
- Inference‑time gains (locked dev/eval study with Qwen2.5)
- Frozen configuration: Qwen2.5‑1.5B‑Instruct, chat template with 4 worked examples, k=4 samples, temp=0.4.
- Results on 100 held‑out grade‑school math items:
- Greedy: 58/100
- Vote‑at‑4 (plurality): 62/100
- Prespecified greedy verifier: 62/100 (paired exact McNemar p = 0.481 vs greedy)
- Any‑sample oracle (best of 4): 79/100
- Post‑hoc confidence model (7 reference‑free features, L2 logistic regression, 5‑fold CV):
- ROC AUC = 0.833, Brier = 0.147 (vs prevalence baseline Brier = 0.236).
- Highest‑confidence 50 predictions contain 47 correct answers → useful triage signal.
- Reproducibility
- Repository includes timeline, frontier matrix, pricing series, locked splits, full traces, and scripts for reproducing all figures/analyses. Vendor vs independent provenance is noted throughout.
Data & Methods
- Data assets (all versioned in the repo):
- D1 Timeline: 27 (model, benchmark, score, date, source) rows, 2018–2026 (vendor‑reported and independent where available).
- D2 2026 frontier matrix: per‑model scores and API prices for July 2026 frontier (GPT‑5.6 family, Claude Fable/Opus, Kimi K3, GPT‑5.5).
- D3 Pricing: input/output prices per million tokens (2020–2026) from vendor docs.
- D4 Live evaluation: locked GSM8K‑derived split (16 dev IDs, 100 eval IDs) with full JSONL traces and exact seeds.
- Analyses
- Trend fitting: logit(p) (solve odds) regressed on release date for SWE‑bench Verified to report annual multiplicative growth of odds.
- Pareto analysis: used Artificial Analysis all‑effort dataset across GPT‑5.6 family to compute cost–capability frontier and identify dominated settings.
- Routing oracle: normalized per‑benchmark to best model; compared best single model, per‑benchmark oracle, and restricted routers (two‑model and cheapest‑within‑one‑point).
- Live experiment: locked configuration selection on dev (prompt/template, temp, model size), then single frozen run on 100 eval items; recorded greedy and sampled completions, verifier outcome, any‑sample oracle.
- Exploratory confidence model: 7 features (plurality share, normalized answer entropy, distinct sampled answers, agreement signals, sample length mean/std), standardized per fold, L2 logistic regression, 5‑fold stratified CV (seeded). Reported ROC AUC, Brier score, coverage/precision tradeoffs.
- Caveats
- Mid‑2026 comparisons mix vendor‑reported and independent measurements; the paper flags provenance and favors independent harnesses where available.
- Some large gains (e.g., ARC‑AGI‑3) may reflect genre‑adjacent training and do not always transfer perfectly to held‑out puzzles.
Implications for AI Economics
- Rapid democratization of capability per dollar
- Because token prices and capability both improved rapidly, a given budget now buys orders‑of‑magnitude more practical capability than in 2023–2024. This compresses time between flagship capability and cheap availability, accelerating adoption.
- Commercial product strategy: task specialization + routers
- The frontier is task‑fragmented. Vendors and platform builders should expect demand for task‑targeted models and for inexpensive routing/meta‑services that dispatch to the best model per task. Two‑model routers can recover near‑oracle gains, so simple routing products can be highly effective.
- Pricing and product tiers
- The mid tier within model families is economically squeezed (dominated Pareto settings); business models will bifurcate into high‑volume low‑price tiers and high‑margin premium tiers. Vendors may redesign tiers or introduce more explicit task‑specialized offerings.
- Procurement and evaluation
- Buyers should rely on independent, task‑relevant benchmarks (not just vendor launch tables) and plan for fast obsolescence of benchmark headroom. Contracting that ties payments to single benchmark progress may misprice rapid frontier shifts.
- Inference engineering matters
- Practitioners can extract meaningful improvements with no retraining by using sampling + aggregation, simple verifiers, and confidence triage. Any‑sample oracles show ceilings; confidence models can enable selective spending (triage) and reduce downstream human verification costs.
- Labor and automation effects
- Rapid gains on repository‑level coding and agentic workflows suggest accelerated automation potential for software engineering and professional tasks measured by agentic benchmarks (SWE‑bench, Agents’ Last Exam, Terminal‑Bench). The pace observed implies substantial near‑term displacement/augmentation effects in these task markets.
- Measurement & policy
- Benchmark turnover accelerates: new, harder benchmarks will be needed rapidly to measure frontier progress. Policymakers and standards bodies should emphasize independent measurement infrastructure and transparent provenance to avoid misled procurement/policy decisions based on vendor claims.
- Risk and transfer considerations
- Large, task‑targeted gains sometimes fail to transfer fully to out‑of‑distribution problems; buyers should validate on representative held‑out tasks. The paper’s reproducible traces highlight the practical importance of independent, auditable evaluation for credible economic decisions.
If you want, I can: - extract the numerical tables (timeline, pricing, SWE‑bench series) into CSVs, - produce a short decision checklist for procurement teams based on these findings, - or summarize the reproducibility artifacts and how to run the live eval locally.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper compiles and reproduces a broad set of public benchmark measurements, pricing data, and a locked held-out experiment with full traces, giving strong descriptive evidence about capability and cost trends. However, several key comparisons rely on vendor-reported scores (acknowledged by the author), the trend fits use a relatively small series (n≈14) spanning heterogeneous provenance, and causal claims about why capabilities changed (architecture, training, inference changes) are not identified with quasi-experimental methods.
Methods Rigormedium — Methods are transparent and reproducible: datasets and code are versioned, the author uses locked development/evaluation splits, reports exact counts and intervals, and applies reasonable statistics (logit trend fits, Wilson intervals, McNemar tests, cross-validated confidence models). Weaknesses include mixing vendor- and independent-reported benchmarks (potential reporting bias), modest sample sizes for some live experiments (100 evaluation items), and at least one exploratory analysis specified after seeing v1 outcomes (the exploratory confidence model), which the author notes.
SampleThree primary datasets: (D1) a 27-row timeline of (model, benchmark, score, date, source) spanning 2018–2026 compiled from papers and leaderboard mirrors; (D2) a July 2026 frontier matrix of per-model benchmark scores and API prices for GPT-5.6 family, Claude Fable/Opus 5, Kimi K3, GPT-5.5, and Opus 4.8 (mix of vendor and independent measurements); (D3) pricing series for representative API models 2020–2026; (D4) live evaluation using GSM8K with a frozen 16-item development split and a single sampled 100-item evaluation split (Qwen2.5-1.5B-Instruct configuration), plus Artificial Analysis model×effort settings (15 settings) used for Pareto analysis and routing comparisons.
Themesproductivity innovation
GeneralizabilityMany mid-2026 comparisons mix vendor-reported and independent harness scores, so some gains may reflect reporting differences rather than model superiority., Benchmark selection focuses on agentic, coding, and reasoning suites (SWE-bench, Agents’ Last Exam, ARC-AGI-3, etc.); results may not generalize to other task families (e.g., multilingual, low-resource, or niche domains)., Live evaluation is limited (100 GSM8K items) and uses a grade-school math task and small open models—findings about self-consistency/confidence may not scale to flagship, multimodal, or tool-using models., The analysis is a July 2026 snapshot; the rapid churn of models/pricing means conclusions may lose applicability quickly., Routing and Pareto results depend on vendor-provided cost and effort tables for specific model families and may not hold for differently configured deployments.
Claims (16)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| The odds of resolving a SWE-bench Verified GitHub issue increased by approximately 5.8 times per year between October 2024 and July 2026. Developer Productivity | positive | Odds of resolving a real GitHub software issue |
Reading fidelity
high
Study strength
medium
|
n=14
5.8× annual growth in solve odds
|
| MMLU performance increased from 43.9% for GPT-3 to 86.4% for GPT-4 and 91.8% for o1, exceeding the paper's estimated human-expert level of 89.8%. Output Quality | positive | MMLU benchmark accuracy |
Reading fidelity
high
Study strength
medium
|
n=3
43.9% to 91.8%
|
| MMLU had lost discriminative power by 2024, with four strong models clustered within four percentage points. Output Quality | null_result | Spread of model scores on MMLU |
Reading fidelity
high
Study strength
medium
|
n=4
within 4 points
|
| API input-token prices declined approximately 60-fold from GPT-3 in 2020 to GPT-5.6 Luna in 2026. Organizational Efficiency | positive | Input-token API price |
Reading fidelity
high
Study strength
medium
|
∼60× decline, from $60 to $1 per million tokens
|
| GPT-5.6 Luna outperformed GPT-5.5 on 7 of 10 agentic and professional-work benchmarks in the vendor's comparison table. Organizational Efficiency | positive | Benchmark performance on agentic and professional-work tasks |
Reading fidelity
high
Study strength
low
|
n=10
7 wins out of 10 benchmarks
|
| GPT-5.6 Luna achieved 92.9% on the reconstructed SWE-bench Verified evaluation, compared with 74.9% for GPT-5's August 2025 vendor-reported score. Developer Productivity | positive | SWE-bench Verified issue-resolution accuracy |
Reading fidelity
high
Study strength
medium
|
92.9% vs. 74.9%
|
| The paper reports that flagship-class capability from 2–12 months earlier can now be obtained at approximately 20% of the earlier flagship's token price. Organizational Efficiency | positive | Cost of obtaining comparable language-model capability |
Reading fidelity
high
Study strength
low
|
∼20% of the flagship token price
|
| Every Terra configuration in the GPT-5.6 family was dominated by another Luna or Sol configuration in the paper's Pareto analysis. Organizational Efficiency | negative | Cost-capability Pareto efficiency of model configurations |
Reading fidelity
high
Study strength
medium
|
n=15
6 of 15 settings off the frontier
|
| Frontier-model leadership was fragmented by task: Opus 5 led frontend coding, Fable 5 led repository-level coding, and GPT-5.6 Sol led agentic terminal work. Task Allocation | mixed | Relative model performance across task categories |
Reading fidelity
high
Study strength
medium
|
Fable 5 led SWE-bench Pro at 80.0% versus Sol at 64.6%; Sol led Terminal-Bench 2.1 at 88.8%
|
| Claude Opus 5 scored 30.2% on ARC-AGI-3, nearly four times the previous record of 7.8% reported for GPT-5.6 Sol. Output Quality | positive | ARC-AGI-3 puzzle-solving accuracy |
Reading fidelity
high
Study strength
medium
|
30.2% vs. 7.8%; nearly 4×
|
| A two-model router using GPT-5.6 Sol and Claude Fable 5 achieved the full per-benchmark oracle gain on the 14-benchmark comparison suite. Organizational Efficiency | positive | Normalized aggregate benchmark performance from model routing |
Reading fidelity
high
Study strength
medium
|
n=14
Best single model 97.6% versus oracle and two-model router 100%; +2.4 points
|
| On the frozen 100-item GSM8K evaluation, Qwen2.5-1.5B achieved 58/100 with greedy decoding and 62/100 with four-sample plurality voting or the prespecified verifier. Output Quality | positive | Exact-answer accuracy on grade-school mathematics problems |
Reading fidelity
high
Study strength
high
|
n=100
58/100 greedy; 62/100 vote-at-four and verifier
|
| Four-sample plurality voting did not produce a statistically significant improvement over greedy decoding on the 100-item evaluation. Output Quality | null_result | Paired exact-answer accuracy difference between vote-at-four and greedy decoding |
Reading fidelity
high
Study strength
high
|
n=100
62/100 vs. 58/100; paired exact McNemar p = 0.481
|
| The any-sample oracle solved 79 of the 100 evaluation problems, indicating that some errors from individual samples could be overcome by a perfect selector. Output Quality | positive | Maximum exact-answer accuracy when any correct sample is selected |
Reading fidelity
high
Study strength
high
|
n=100
79/100
|
| The exploratory confidence model ranked vote-at-four outputs with an out-of-fold ROC AUC of 0.833 and a Brier score of 0.147, compared with 0.236 for a prevalence baseline. Decision Quality | positive | Confidence-model discrimination and probability calibration for correctness |
Reading fidelity
high
Study strength
low
|
n=100
ROC AUC 0.833; Brier score 0.147 versus 0.236 baseline
|
| Among the 50 highest-confidence predictions from the exploratory confidence model, 47 were correct. Decision Quality | positive | Accuracy among the top-confidence half of predictions |
Reading fidelity
high
Study strength
low
|
n=50
47/50 correct
|