Explicitly asking Sonnet 5 to 'think harder' increased average inference bills by roughly $0.01 per call but did not produce a clear accuracy gain on the tested math problems, suggesting some buyers may pay more for an explicit effort term without measurable benefit under the studied contract and date.

Main Finding

A buyer-facing API "contract" term for reasoning effort changes delivered cost even when the documented default effort is the same: running Claude Sonnet 5 with an explicit high-effort request cost on average $0.01031 more per call than the identical model with the effort field omitted (95% bootstrap interval +$0.00204 to +$0.01974). There was no statistically detected accuracy difference (point estimate +1.33 percentage points, 95% interval −2.67 to +4.67), and the registered point estimate of cost per correct answer was higher under explicit high effort ($0.08665 vs $0.07662).

Key Points

• Unit of purchase is a dated contract (requested+served model, effort term or omission, prompt, output rail, price schedule, run date), not just a model name.
• Registered, paired within-model contrast: Claude Sonnet 5 with explicit high effort vs Sonnet 5 with effort omitted (both otherwise identical).
• Main numeric results (Sonnet, 30 AIME items, 5 calls/item per cell, item-clustered inference):
  • Mean cost difference = +$0.01031 per call (explicit-high minus omitted), 95% bootstrap CI [+0.00204, +0.01974].
  • Accuracy difference = +0.0133 (i.e., +1.33 pp), 95% CI [−0.0267, +0.0467] — not statistically detected; CI allows up to ≈4.7 pp gain.
  • Cost per correct (registered point estimates): $0.08665 (explicit high) vs $0.07662 (omitted).
• Repeated-call accounting: every paid attempt assigned one terminal outcome (correct, wrong, no_answer_rail, no_answer_other, provider_failure) and retained in cost and outcome totals rather than being treated as missing.
• Dated contract census and preregistered structural probes showed that omission semantics (what happens when the effort field is omitted) vary across models and even within a provider; some omissions were verifiably implemented, others remained documentation-grade (indeterminate in raw response structure).
• Robust preregistration and governance: frozen prompt, parser, terminal taxonomy, request registry, spending caps, analysis pipeline; item-clustered bootstrap (10,000 draws, seed fixed) was used for uncertainty; raw provider responses preserved and re-extracted during analysis.

Data & Methods

• Task: full 30-item AIME 2026 public cohort (closed-form integer answers 0–999). Parser required a final explicit "Answer: N" claim; final claim used even if earlier text contained valid-looking claims.
• Primary design: paired contrast on the same 30 items:
  • Claude Sonnet 5, explicit high effort — 30 items × 5 calls = 150 calls.
  • Claude Sonnet 5, effort omitted — 30 × 5 = 150 calls.
• Additional single-pass reference cells (same dated session):
  • GPT-5.6 Terra, effort omitted (30 × 1).
  • Claude Fable 5, effort omitted (30 × 1).
  • Two small non-contemporaneous GPT-5.4-mini bridge cells (descriptive).
• Each call was stateless, non-streaming, single request (no conversation history). For paired Sonnet runs, the high-effort call was queued before the omitted call for each item.
• Cost accounting: reconstructed from provider-reported usage under the registered dated price schedules (input/cached-input/output accounting, including billed reasoning hidden in output), reconciled to console balance changes at cent granularity.
• Outcome taxonomy: every paid attempt classified into exactly one terminal category under a frozen precedence rule; wrong/no-answer outcomes retained in totals.
• Statistical inference: item is independent unit; for repeated cells, calls averaged within item, then items weighted equally. Paired contrasts computed within items. Item-clustered percentile bootstrap (10,000 draws) provided 95% intervals. Single-pass cells used exact Clopper–Pearson for proportions.
• Governance: all analytical artifacts, code, registry, and raw responses pinned/frozen before outcome inspection; analyses run against immutably-stored artifacts.

Implications for AI Economics

• Contracts matter: buyers do not simply choose a model; they purchase a specific, dated contract that can materially change cost outcomes even when documentation claims the same default. Economists and engineers modeling deployment costs should treat effort-setting omissions as distinct service alternatives.
• Stochastic and model-specific thinking costs should be incorporated into cost-aware deployment and routing frameworks (e.g., FrugalGPT-style decision rules). A router or optimizer must treat different effort request states (including omission) as candidate services with their own cost/quality distributions.
• Pricing and transparency: providers’ documented defaults and raw-response semantics can diverge across models and over time. Markets would benefit from clearer, machine-readable contract metadata (e.g., whether omission maps to adaptive thinking, fixed token budgets, or an off-switch) and explicit accounting for hidden "thinking" tokens in price displays.
• Cost-per-correct tradeoffs: an explicit effort control can increase average cost and cost-per-correct without a detectable accuracy improvement in this setting. Buyers valuing cost-efficiency (e.g., high-volume query pipelines) may prefer omission if it yields similar accuracy with lower expected cost; buyers willing to pay for small accuracy gains should weigh the observed confidence interval (cannot rule out up to ≈4.7 pp gain here).
• Evaluation and procurement: evaluations that drop wrong or no-answer paid attempts from analyses understate true delivered cost and buyer experience. Procurement and benchmarking protocols should preserve purchased unsuccessful outcomes in both cost and outcome accounting.
• Versioning and reproducibility: results are tightly bounded to the model snapshot, prompt, task, and date. Economic analyses and policies should account for temporal contract heterogeneity (dated contracts) rather than treating a model name as static.
• Regulatory/market-design angle: because omission semantics vary across models and providers, regulatory disclosure standards that require explicit, testable metadata about reasoning-token behavior and omission semantics would reduce buyer uncertainty and enable fairer price comparisons.

Caveats - Task- and model-specific: study focuses on Sonnet 5 (and a few reference cells) on AIME 2026; findings do not automatically generalize to other tasks, models, or dates. - Sample size and power: 30 independent items limit precision; the accuracy CI admits nontrivial possible gains, so absence of a detected accuracy improvement is not proof of no benefit. - Queue ordering: within-item the high-effort call was queued before the omitted call; the design choice is documented but should be considered when interpreting conditional behavior.