Short, corpus-derived 'skills' capture most of test-time reasoning’s value: distilled prompts restore 55%–100%+ of reasoning-mode gains across agentic benchmarks while slashing output tokens by 2.7–6×, offering a cheap one-time amortization of recurring reasoning costs.

Main Finding

A small, one-time corpus distillation step — using a coding-capable LLM to compile a 40–130 line natural-language “skill” from 35–50 existing agent rollouts — recovers most or all of the accuracy gains that test-time reasoning (chain-of-thought / private reasoning tokens) provides on multi-step agentic tasks, while emitting far fewer output tokens and incurring only a small one-time production cost. For GPT-5.4-mini across four benchmarks, distilled skills recover 55%–100%+ of the think/no-think gap, beat the reasoning mode on two benchmarks, and reduce per-episode output tokens by roughly 2.9–4.5× (zero reasoning tokens).

Key Points

• Problem targeted: reasoning modes improve multi-step agent performance but impose a recurring per-episode token (and cost) premium because they re-derive domain procedures on every episode.
• Method (Passive Skill Distillation): take an existing corpus of ordinary rollouts (no extra environment runs), run a coding agent (Claude Sonnet 5 / Claude Code) to analyze failures and successes, and produce a compact rule-style skill (40–130 lines of markdown) that is appended to the non-reasoning model’s system prompt.
• Cost: one-time distillation cost ≈ $1.28–$2.44 per domain (as run here).
• Quantitative highlights (GPT-5.4-mini, held-out tasks):
  • ALFWorld: think 71.3% (3,723 tok) → no-think+skill 78.7% (832 tok) — skill exceeds think while emitting ~4.5× fewer tokens.
  • SpreadsheetBench-Verified: think 61.3% (3,291 tok) → no-think+skill 56.0% (831 tok) — recovers substantial gap with ~4.0× fewer tokens.
  • τ2-telecom: think 45.0% (2,143 tok) → no-think+skill 33.3% (597 tok) — recovers ~55% of gap, large token savings.
  • τ2-retail: think 35.0% (1,615 tok) → no-think+skill 40.8% (565 tok) — skill outperforms think with ~2.9× fewer tokens.
• Distillation-source ablation: skills distilled from non-reasoning trajectories alone are competitive with those distilled from paired think/no-think corpora. Domain-dependent effects observed (e.g., retail benefits from paired traces; SpreadsheetBench favored no-think-only).
• Comparison to prompt optimization: distilled skills outperformed GEPA on τ2 domains while costing ~4.1× less to produce in this evaluation.
• Interpretation (search lens): reasoning = deep per-episode search (paid repeatedly); distillation = wide cross-episode search (paid once). Distillation captures episode-invariant procedural knowledge; residual gaps indicate where per-instance deep search remains necessary (e.g., telecom, spreadsheets).

Data & Methods

• Benchmarks:
  • ALFWorld (text-based embodied household tasks), held-out 50 tasks.
  • SSB-Verified (verified subset of SpreadsheetBench), held-out 50 tasks.
  • τ2-bench telecom (50 train / 40 test) and τ2-bench retail (35 train / 40 test) — conversational customer-service agents with tools and simulated users.
• Corpus sizes for distillation: 35–50 training tasks per domain, using ordinary evaluation rollouts already available (no additional environment interaction).
• Distiller: Claude Code (Claude Sonnet 5) — an LLM with filesystem and code-execution, given the corpus and a fixed instruction; it computes statistics (failure frequencies, action n-grams, loops, win/loss contrasts) and compiles concrete, traceable rules in markdown.
• Skill form and deployment: 40–130 lines of rule-like markdown appended verbatim to the non-reasoning model’s system prompt (cacheable prefix). No model weights or runtime search changed.
• Models evaluated: GPT-5.4-mini (reasoning_effort ∈ {none, medium}) and Qwen3.6-27B (enable_thinking ∈ {false, true}). Each result averaged over 3 evaluation seeds.
• Costs reported: distillation one-time production cost roughly $1.28–$2.44 per domain (depending on domain); GEPA benchmark cost reported for comparison ($13.02 vs. $2.44 example in telecom).
• Ablations and robustness: evaluated distillation source (no-think only vs paired with think), compared to automated prompt optimization (GEPA), and reported per-seed consistency; distillation-run variance was not measured (skills distilled once per domain/model).

Implications for AI Economics

• Amortization of reasoning costs: Distilling domain-level procedural knowledge converts a recurring per-episode variable cost (reasoning token generation) into a one-time fixed cost (coding-agent distillation + prompt injection). This can produce large per-episode savings in token output and thus runtime cost when many episodes are expected.
• Cheap production, fast ROI: Production costs observed here are on the order of a few dollars per domain. The break-even point is extremely modest: once cumulative token-cost savings exceed the one-time distillation cost, remaining episodes are net savings. (General formula: break-even episodes ≈ one-time_distill_cost / (per-episode_token_savings × token_price).)
• Operational advantages:
  • Marginal deployment cost is negligible: skills are a cacheable prompt prefix; no extra runtime inference beyond the non-reasoning model.
  • No need for reasoning-mode access in the amortization loop if using non-reasoning-only rollouts, which reduces production costs and simplifies compliance/operational constraints.
  • Easier to apply where rollouts/logs already exist (logging pipelines, replay storage).
• When deep (per-instance) reasoning is still necessary:
  • Domains with long instance-specific dependency chains or ad hoc logic (telecom, complex spreadsheets) still benefit from test-time deep search; distillation reduces the base rate but does not fully substitute.
  • Recommended hybrid strategy: use distilled skills to eliminate repetitive, episode-invariant costs, and reserve test-time reasoning only for instances that need expensive per-instance search.
• Comparison to alternatives:
  • Distillation can be cheaper and faster than active prompt-optimization loops that require many scored rollouts (the paper shows better scores at lower production cost vs GEPA on τ2).
  • Distillation requires no model retraining or weight updates — attractive when model updates are expensive or restricted.
• Risks and limits for economic deployment:
  • Skills are model- and domain-specific; cross-model transferability untested → might require per-model per-domain distillations, increasing fixed costs in multi-model setups.
  • Distillation variance was not exhaustively measured; some domains showed sensitivity (e.g., Qwen retail regression), suggesting occasional re-distillation or QA may be needed.
  • Where instance-specific reasoning is essential, distillation reduces but does not eliminate runtime reasoning costs.
• Practical recommendation for product teams:
  • If you have many expected episodes in a stable domain, run a one-pass corpus distillation over existing logs: expected token (and latency/cost) savings generally repay the small production cost quickly.
  • Combine skills with selective triggering of reasoning mode for ambiguous or complex instances to optimize total cost-performance.

Limitations noted by the authors: only two models and four domains tested; skills distilled once per model-domain (no distillation variance study); skills are model-specific and cross-model transfer was not evaluated.