Scaling gains do not automatically tell you where to spend: many published comparisons count successes that a deployed system would never deliver, rely on oracle information, or leave out important costs. Emadi proposes a capability surface and a concise 'resource envelope' to record budgets, mechanisms and information so teams can judge which allocation claims a study actually supports.

Main Finding

Scaling-study results about higher scores under larger budgets do not by themselves justify reallocating resources in deployment. To support allocation decisions, a study must report (a) how success was counted, (b) what information the tested system had access to, and (c) what costs were included. Emphasizing these three reporting dimensions, the paper proposes a formal framework (capability surface, feasible set, attainable frontier) and a practical reporting device (the resource envelope) to make allocation claims actionable and comparable.

Key Points

• Three recurring mismatches that undermine direct allocation claims:
  • Success counted before an answer is chosen: many papers report coverage (any candidate correct among many) rather than the accuracy of the single answer returned to users (returned-answer accuracy).
  • Information unavailable at deployment: experiments sometimes use answer keys, oracle feedback, or test-set tuning that a deployed system would not have.
  • Costs left out: studies often measure FLOPs for generation only and omit verifier/index construction, training, latency, or lifecycle update costs.
• Formal framework to read scaling results:
  • Quality as a function Q_k = g_k(b, s; I), where b = budgets, s = mechanisms (architecture, retrieval, verifier, stopping rule), I = information available.
  • Feasible set F_k(I): (b, s) pairs that can be run given available information/hardware.
  • Attainable frontier Q*_k(B; I) = sup{g_k(b, s; I) : (b, s) ∈ F_k(I), b ≤ B} — the best achievable under budget B (distinct from performance of any single tested procedure).
  • Lifecycle cost accounting: C_life = C_dev + C_update(N) + sum_t E[C_run,t], highlighting development vs run-time trade-offs and the importance of the accounting boundary.
• Types of claims a scaling result may support (Figure 1):
  • (i) Conditional performance: how the tested procedure responds to resource.
  • (ii) Mechanism: explanations for why the gain occurs.
  • (iii) Tested allocation: comparison among the specific alternatives actually evaluated (same accounting, information, selector, etc.).
  • (iv) Frontier: a claim about the best use of resources requires searching or proving optimality across the feasible set.
• Practical recommendation: adopt a resource envelope (a structured record attached to reported scores) that lists: task, development & run-time resources, information access, mechanisms (selector/checker/stopping), accounting boundary, deployment volume assumptions, and selection/aggregation rules.
• The paper provides worked analytical examples showing that changes in metric (coverage vs returned-answer accuracy), selection rule, deployment volume (N), or stopping policy can reverse allocation conclusions.
• Limits: the framework does not propose a universal scaling law nor infer general intelligence from benchmark gains; it focuses narrowly on evidence needed to choose how to spend a given budget.

Data & Methods

• Type: Critical integrative review (not a systematic review or meta-analysis). Search closed 9 September 2026.
• Sources: arXiv, conference proceedings, OpenReview, ACL Anthology, publisher records. Starting point: foundational scaling and evaluation papers; followed citations in both directions; targeted searches on scaling laws, training–inference split, test-time computation, retrieval/datastore scaling, verification/repeated sampling, agent coordination, metareasoning, and resource-aware evaluation. Narrower searches: distillation, quantization, speculative decoding, serving.
• Inclusion criteria: studies that varied a resource or mechanism, analyzed a computational/information constraint, or stated evaluation principles relevant to allocation decisions. Methods papers included where mechanisms needed explaining; compression/serving methods included only for their effect on what a budget can buy. Application surveys without transferable allocation arguments were excluded.
• Evidence handling: peer-reviewed work anchors the synthesis; preprints marked and treated as provisional. For each cited study the author recorded what was varied, what was measured, and what was held fixed or omitted. No formal bias instrument or registered protocol — the output is a chosen, annotated sample rather than a pooled statistical estimate.
• Additional methods: analytical worked examples, formulation of capability surface and attainable frontier, and one applied example filling a resource envelope for a published comparison.

Implications for AI Economics

• For procurement and cost–benefit analysis:
  • Require end-to-end cost accounting (development + run-time + updates) when comparing architectures or operational strategies. Partial FLOP-based comparisons are insufficient for deployment budgeting.
  • Evaluate and price systems on returned-answer accuracy (or other user-facing metrics) rather than coverage or internal oracle-assisted scores.
  • Consider deployment volume (N) explicitly: amortization of development costs (e.g., building an index or verifier) matters and can change the preferred allocation.
• For modeling investment choices:
  • The distinction between development expenditure (e.g., larger model, more pretraining) and run-time expenditure (more calls, retrieval, verifiers, repeated sampling) must be explicit; different workloads and N shift optimal allocations.
  • Efficiency methods (distillation, quantization, speculative decoding, caching) change what a budget buys and therefore should be included in capability-frontier assessments rather than treated as orthogonal.
• For experimental design and reporting standards:
  • Adopt the resource envelope as a reporting standard in AI economics and empirical ML to make allocation-relevant claims comparable and actionable.
  • Report selection/aggregation rules, stopping policies, accounting boundaries, and any use of test-set or oracle information explicitly.
• For regulation and policy:
  • Regulators and procurement bodies should require evidence about the feasible frontier (or extensive search among feasible allocations) before inferring that one resource allocation is superior.
  • Claims about “scaling makes X more useful” should be supported by tests that use only deployment-available information and account for all costs that affect the decision-maker.
• For research priorities:
  • More experiments are needed on transferability of allocation rules across tasks, operating conditions, and volumes; on how selection/aggregation machinery can be reliably trained without leaking test information; and on methods to approximate or bound the attainable frontier in practice.
• For economic modeling of AI adoption:
  • Models of firm-level investment should incorporate (i) selection/value of run-time meta-actions (retrieval, verification), (ii) information constraints (what feedback is available), and (iii) amortization of fixed development costs over deployment volume — all of which can change comparative statics of optimal investment in model scale vs inference effort.

Limitations noted by the paper: not a universal scaling-law proposal, not a statistical meta-analysis, and conclusions depend on the scope of surveyed literature (search closed Sept 9, 2026).