0 cumulative citations
View corpus contextExtended 'thinking' in LLMs pays off mainly on sequential inference tasks — math and code problems see the largest accuracy-per-token gains — but benefits shrink or reverse as reasoning effort rises, and on‑premises deployment can materially alter the cost calculus.
The Reasoning Tax: Token Economics of LLM Reasoning Across Task Types and Deployment Contexts
The paper introduces the Token Economy Score to measure marginal accuracy gain per generated-token multiplier and shows that reasoning tokens are most cost-effective for sequential inference tasks, exhibit diminishing returns (and sometimes negative returns) at high effort levels, and that deployment context (cloud vs on‑prem) substantially changes the dollar economics.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Accuracy-only benchmarking of reasoning-capable large language models misses a central deployment question: when do extended thinking tokens earn their cost? We introduce the Token Economy Score (TES), a marginal benchmarking metric that measures the accuracy gain of a reasoning model over a non-reasoning baseline, normalized by the generated-token multiplier. We define paired and approximated TES variants for model families with reasoning toggles and frontier models without direct non-reasoning counterparts. We then conduct an empirical benchmarking analysis across 151 model-benchmark evaluation runs on seven benchmarks spanning mathematics, code generation, science reasoning, instruction following, expert knowledge, knowledge recall, and research-level physics. The analysis examines three deployment-facing dimensions: which task structures yield positive marginal reasoning efficiency, how increasing reasoning effort changes TES within model families, and how deployment context changes economic viability. Results show that task structure predicts reasoning efficiency better than nominal difficulty: sequential inferencechain tasks such as AIME 2025 and LiveCodeBench show high TES, while knowledge-recall tasks such as MMLU-Pro show low TES despite their difficulty. We also find systematic diminishing returns at higher reasoning effort levels, including cases where additional thinking reduces accuracy. Finally, Reasoning Cost Share (RCS) shows that inference spend is often dominated by internal thinking, while Deployment Cost Multiplier (DCM) shows how on-premises deployment can change the economics of otherwise costly reasoning workloads. These findings support a benchmarking-driven model-selection rule: enable reasoning selectively by task type, effort level, and deployment context rather than treating it as a universally beneficial mode.
Summary
Main Finding
The paper introduces the Token Economy Score (TES), a marginal metric that quantifies whether the accuracy gains from enabling or increasing a model's internal "thinking" tokens justify the extra generated-token cost. Using TES (and two deployment descriptors, Reasoning Cost Share and Deployment Cost Multiplier), the authors benchmark 151 model–benchmark runs across 7 benchmarks and 27 model configurations and find that: (1) task structure predicts marginal reasoning efficiency better than nominal difficulty (sequential inference tasks yield the highest TES; recall-heavy and saturated tasks yield low or negative TES); (2) raising reasoning effort shows strong diminishing returns and can even reduce accuracy; and (3) deployment context (cloud vs on‑prem) meaningfully changes the dollar economics of reasoning workloads.
Key Points
- Token Economy Score (TES)
- TES = (Acc(Mr) − Acc(Mb)) / (GenTok(Mr) / GenTok(Mb))
- Numerator: accuracy gain in percentage points over a non-reasoning baseline.
- Denominator: multiplicative increase in generated tokens (reasoning + output).
- Interpretation: TES > 1 = efficient (accuracy gain > token multiplier); 0 < TES ≤ 1 = marginal; TES ≤ 0 = wasteful or harmful.
- TES variants
- TES-Δ: paired comparison within a model family when a reasoning toggle exists.
- TES-A: approximated baseline using the best available non-reasoning model when a paired baseline is unavailable (used for Gemini-family models).
- Reasoning Cost Share (RCS)
- Fraction of total inference spend attributable to the thinking chain (reasoning tokens).
- Observed to be often large — reasoning can dominate inference spend.
- Deployment Cost Multiplier (DCM)
- Ratio of total inference cost (cloud / on‑prem) for a workload.
- Illustrates how on‑premise deployment can make otherwise costly reasoning workloads affordable under sufficient utilization.
- Empirical results overview
- Dataset: 151 eval runs across 7 benchmarks (IFBench, MMLU‑Pro, GPQA Diamond, AIME 2025, LiveCodeBench, HLE, CritPt), 27 model configs, 8 families (GPT, Claude, DeepSeek, Qwen, Gemini, Grok, GLM, Gemma).
- Sequential inference tasks (AIME 2025, LiveCodeBench, IFBench) show the strongest TES (often > 1).
- Knowledge‑recall tasks (MMLU‑Pro, GPQA Diamond) show low TES even when difficult.
- Frontier-domain tasks (HLE, CritPt) require interpreting TES alongside absolute accuracy (low absolute accuracy makes TES less actionable).
- Within families, increasing reasoning effort generally reduces TES and sometimes reduces accuracy (overthinking).
- Practical rule: enable reasoning selectively by task structure, tune effort level, and consider deployment context—do not treat reasoning as universally beneficial.
Data & Methods
- Formal metrics
- Generated tokens = ReasoningTokens + OutputTokens. Input tokens excluded.
- TES formula as above; RCS and DCM defined and computed separately.
- Experimental scope
- 151 model–benchmark evaluation runs covering 7 benchmarks chosen to span task structures (instruction following, knowledge recall, sequential inference, frontier reasoning) and difficulty ranges.
- 27 model configurations across 8 model families; effort-level diversity included where models expose multiple reasoning settings (e.g., medium, high, xhigh, max).
- Data sources
- Cloud-hosted model token counts and accuracies: Artificial Analysis.
- Self-run evaluations (on‑prem) for selected open-weight models to validate token/accuracy and to measure throughput for on‑prem costs (models listed: Qwen variants, Gemma 4 variants, etc.).
- Price data: published API rates as of late May 2026 (used for RCS and DCM).
- On‑prem cost methodology
- Hardware: 8×NVIDIA B300 system.
- Total system cost (5‑year amortized CAPEX + OPEX) = $1.013M → amortized running cost $0.00633/sec.
- Per-token on‑prem cost = amortized cost per second / measured tokens-per-second throughput.
- Assumes sufficient utilization to amortize costs.
- Validity limits and caveats
- TES uses mean generated tokens; means can be skewed by outliers (very long chains on a few problems).
- One anomalous IFBench pair (Grok non-reasoning variant producing extremely large output) excluded from aggregates.
- TES-A conflates reasoning with cross‑family capability differences (limitation noted for Gemini).
- TES ignores input tokens (by design) and does not monetize accuracy gains — the TES = 1 breakeven is a practitioner-centered convention; users can rescale for task-specific value-per-accuracy-point.
- Pricing-dependent metrics (RCS, DCM) reflect prices at collection time and deployment‑assumption sensitivity (e.g., utilization).
Implications for AI Economics
- Procurement and model selection
- Use TES (paired when possible) as a decision metric: only enable reasoning when TES meets your value threshold (e.g., TES > 1 under default interpretation, or adjust to your dollars-per-accuracy-point).
- Prefer reasoning for sequential inference tasks (program synthesis, multi-step math, constrained multi-step instruction following); avoid enabling reasoning by default for recall-heavy or near‑saturated tasks.
- Runtime and product design
- Implement adaptive toggles: per-task or per‑user routing to reasoning-enabled versus non-reasoning models and dynamic effort-level tuning to avoid diminishing returns and overthinking.
- Monitor RCS: high RCS implies that the majority of inference spend is on internal thinking chains — budget and safety controls should account for that (abort thresholds, chain-length caps, cached subanswers).
- Deployment economics
- Evaluate DCM for high-reasoning workloads: on‑prem inference can substantially reduce per-token dollars at scale (DCM > 1), making expensive reasoning modes economically viable if utilization is high; for low or bursty workloads cloud may still be preferable.
- Consider throughput and latency: the paper’s on‑prem cost model assumes full utilization and amortization; adjust for real-world load profiles when comparing cloud vs on‑prem.
- Research and benchmarking recommendations
- Benchmarkers should report TES (paired where possible), RCS, and DCM alongside raw accuracy and token counts to make cost-aware model comparisons.
- Future work: headroom-normalized TES (to adjust for ceiling effects), per-problem token distributions (to handle outliers), incorporate latency and compute-cost-per-token (beyond token counts), and develop utility-weighted TES that converts accuracy points into dollars or user-value for direct procurement decisions.
- Risk and safety
- Agentic or multi-step systems that rely on long internal chains should expect high RCS and hence potentially large variable inference costs — both cost controls and semantic checks are needed to avoid runaway spend or hallucination-driven costs.
Summary recommendation: adopt TES (paired where possible) as a marginal, deployment-aware efficiency metric; enable reasoning selectively by task structure and tune effort levels conservatively, and evaluate on‑prem vs cloud economics using DCM under realistic utilization assumptions.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper presents a large, systematic benchmarking exercise (151 model-benchmark runs across 27 model configurations and seven benchmarks) and introduces a clear marginal metric (TES). It includes paired comparisons where available and on‑premises measurements for selected open models, which strengthens empirical claims. However, the evidence is observational and descriptive rather than experimental: some baselines are approximated (TES‑A), token counts are means that can be driven by outliers, pricing and model behavior are time-dependent, and some data come from third-party sources (Artificial Analysis), limiting causal interpretation and long-run robustness.
Methods Rigormedium — The authors define metrics precisely (TES, RCS, DCM), pre-specify benchmark and model-selection criteria, and publish code/data. They validate third-party figures with self-run evaluations for several models and measure on‑prem throughput using a transparent TCO approach. Limitations include reliance on mean token counts (sensitive to outliers), approximated baselines for some frontier models (Gemini), potential inconsistencies across evaluation harnesses, and pricing/throughput assumptions (e.g., full utilization) that affect economic conclusions.
Sample151 model-benchmark evaluation runs covering 27 distinct model configurations from eight families (GPT, Claude, DeepSeek, Qwen, Gemini, Grok, GLM, Gemma) evaluated on seven benchmarks spanning instruction following (IFBench), knowledge recall (MMLU-Pro, GPQA Diamond), sequential inference (AIME 2025, LiveCodeBench), and frontier-domain reasoning (HLE, CritPt). Token counts and accuracy were primarily sourced from Artificial Analysis, with self-run evaluations on an 8xNVIDIA B300 system for selected open-weight Qwen/Gemma models and supplementary accuracy for some Claude variants from MathArena; pricing reflects API rates as of late May 2026.
Themesproductivity adoption
GeneralizabilityResults reflect the chosen seven benchmarks and 27 model configurations and may not generalize to other tasks or new model versions., TES-A comparisons conflate model capability and reasoning toggle effects for frontier models lacking paired non-reasoning variants (e.g., Gemini)., Mean generated-token counts can be skewed by outliers; per-problem variance may change practical conclusions for particular workloads., On-prem cost estimates assume high utilization and a specific 8xB300 hardware TCO; lower utilization or different hardware materially change DCM., Pricing and model behavior are time-sensitive; API prices, model weights, and reasoning implementations may change rapidly.
Claims (9)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| The paper introduces the Token Economy Score (TES), which measures the accuracy gain of a reasoning-enabled model over a non-reasoning baseline normalized by the generated-token multiplier. Organizational Efficiency | positive | Marginal accuracy efficiency of reasoning relative to token expenditure |
Reading fidelity
high
Study strength
medium
|
TES = accuracy gain in percentage points divided by generated-token multiplier
|
| Across 151 model-benchmark evaluation runs covering seven benchmarks, task structure predicts reasoning efficiency better than nominal task difficulty. Task Allocation | positive | Reasoning efficiency as measured by TES across task structures |
Reading fidelity
high
Study strength
medium
|
n=151
|
| Sequential inference tasks produce the strongest reasoning returns: AIME 2025 has the highest mean TES, while LiveCodeBench and IFBench exceed the TES greater-than-one threshold on average. Task Allocation | positive | Token Economy Score for reasoning-enabled models |
Reading fidelity
high
Study strength
medium
|
n=151
TES > 1 on average for LiveCodeBench and IFBench
|
| Knowledge-recall benchmarks such as MMLU-Pro and GPQA Diamond show weak reasoning efficiency despite being difficult benchmarks. Task Allocation | negative | Marginal reasoning efficiency on knowledge-recall tasks |
Reading fidelity
high
Study strength
medium
|
n=151
|
| Increasing reasoning effort beyond a moderate level produces diminishing marginal TES across the model families analyzed, even when accuracy improves. Organizational Efficiency | negative | Marginal Token Economy Score as reasoning effort increases |
Reading fidelity
high
Study strength
medium
|
n=7
|
| For GPT-5.2 on LiveCodeBench, increasing effort from Medium to Xhigh reduced accuracy by 0.5 percentage points and produced a negative marginal TES of -0.153. Output Quality | negative | Benchmark accuracy and marginal token efficiency |
Reading fidelity
high
Study strength
medium
|
-0.5 pp accuracy change; marginal TES = -0.153
|
| For DeepSeek V4 Pro on GPQA Diamond, increasing effort from High to Max reduced accuracy by 1.7 percentage points and resulted in a negative marginal TES of -0.621. Output Quality | negative | Benchmark accuracy and marginal token efficiency |
Reading fidelity
high
Study strength
medium
|
-1.7 pp accuracy change; marginal TES = -0.621
|
| Inference spending is often dominated by internal reasoning tokens rather than final-answer tokens, as measured by the Reasoning Cost Share (RCS). Organizational Efficiency | negative | Share of total inference cost attributable to reasoning tokens |
Reading fidelity
high
Study strength
medium
|
n=151
|
| Deployment context can materially change the cost economics of reasoning workloads; the paper uses the Deployment Cost Multiplier (DCM) to compare cloud and on-premises inference costs. Organizational Efficiency | mixed | Total inference cost under cloud versus on-premises deployment |
Reading fidelity
high
Study strength
medium
|
n=4
|