Selective loading of agent skills can cut prompt token overhead by tens of percent to over 70% on large, multi-turn tasks without detectable accuracy loss; but for small or repeatedly required skills the retrieval scaffolding and caching behavior erase the gains, and results depend on provider-specific caching and prompt scaffolding.

Main Finding

Conditional loading of skill “blocks” (selectively supplying parts of an instruction document instead of pre-loading the whole skill) can substantially reduce model input tokens and therefore deployment cost, but there is no universal best method. Gains are largest when skills are large and contain substantial material that is not needed on every turn; when skills are small or their content is needed almost every turn, loading overhead or repeated scaffolding can erase savings. Hybrid (short stubs + fetch full block on demand) is the best content-parity mechanism in single-turn, small-to-medium settings; Skill Block (core + tool to fetch blocks) and hybrid perform best for large, compressible, multi-turn skills. Static pruning (permanently removing rarely used blocks) can beat conditional loading if reduced coverage is acceptable.

Key Points

• Four content-preserving loading mechanisms compared (same source content available to each):
  • full: render the entire skill every prompt (baseline).
  • Skill Block: small core always-on; model calls a tool to fetch optional blocks.
  • reference: compact catalog; model selects a reference file (no tool).
  • hybrid: short stubs of every block first; fetch full block only if stub is insufficient.
• Caching-correct accounting: multi-turn provider logs include cache-read tokens. Paper reports:
  • new input = input_tokens − cache_read
  • effective input = new input + 0.1 × cache_read (d = 0.1)
  • raw input (d = 1) used for single-turn; effective input used for multi-turn.
• Practical trade-off: unused optional token mass must exceed tool/selection round-trip plus repeated footprint to justify conditional loading. Expressed roughly as: net saving ≈ sum_i (1 − p_i) s_i − ((r − 1)(F + c + X) + rT) where s_i = block size, p_i = load probability, r = rounds, T = tool-schema tokens, F/X = fixed/ task context.
• Outcome detection: paired outcome tests detected no quality differences in primary comparisons, but tests are not equivalence or non-inferiority proofs; routing errors can be partially mitigated by execution feedback in some benchmarks.

Key empirical deltas (gpt-5.5 primary endpoint; input shown with relative change vs full): - SearchQA (n=1,400, single-turn, ≈2K-skill): hybrid −27.4% raw input; Skill Block +48.4% (Skill Block was aggressively encouraged to load). Reference +10.4%. - SpreadsheetBench (n=276 token-complete cases, single-turn, large skill ≈8K tokens): hybrid −39.8%; Skill Block −35.6%; reference −31.7%. Static original8 (six blocks removed; non-content-parity) −55.5%. - ALFWorld (n=42, multi-turn, small procedures ≈1K): Skill Block −12.6% effective input; hybrid −3.2%; reference +24.2%. Because procedures are small and used repeatedly, output/round-trip effects reduce net gains. - ScienceWorld (n=84, multi-turn, large skill ≈6K): Skill Block −62.5% effective input; hybrid −52.8%; reference −23.0%. - SynthProc (n=40, synthetic multi-turn, ≈9.7K): Skill Block −73.0%; hybrid −66.6%; reference −14.8%. - Reruns with a different configured endpoint (gpt-5.6) preserved the large-skill pattern, subject to same-provider and configuration limitations.

Limitations highlighted by authors: - Experiments run on a production provider endpoint (primary logged as gpt-5.5 Copilot configuration); provider system prompt and decoding settings are not public. - Model identity in historical logs not always independently confirmed; same-provider testing limits external generalizability. - Static original8 is non-content-parity and exploratory. - Outcome tests are paired and exploratory; non-significant differences do not prove equivalence.

Data & Methods

• Skill representation: partition skill into named blocks (description, default flag, dependencies, priority, tags). Rendering all blocks reproduces monolithic skill (content-preserving).
• Benchmarks:
  • SearchQA (single-turn QA, n=1,400, small skill)
  • SpreadsheetBench (single-turn spreadsheet tasks with execution feedback, n≈276 token-complete)
  • ALFWorld (multi-turn embodied/text environment, n=42, small procedures)
  • ScienceWorld (multi-turn scientific simulator, n=84, many procedures)
  • SynthProc (synthetic multi-turn probe, n=40, controlled gate of operations)
• Primary token metrics:
  • Single-turn: raw input tokens (provider input_tokens).
  • Multi-turn: median effective input (new input + 0.1 × cache_read).
  • Also report total_λ = base_input + λ × output_tokens for λ ∈ {1,4,8} to approximate generated-token sensitivity.
• Protocol:
  • Paired within-item comparisons across mechanisms on same endpoint/config.
  • Token telemetry validated (new = input − cache_read).
  • Paired bootstrap 95% CIs for token intervals; paired McNemar tests for outcomes (Holm adjustments applied).
  • Some historical runs and additional pilot runs (gpt-5.6) used to check transferability.
• Special design choices:
  • Hybrid and reference implement progressive disclosure; Skill Block uses a tool schema that can require extra model-provider round trips.
  • In SearchQA Skill Block was intentionally encouraged to load more (to make its overhead visible); alternative guidance would have reduced loads but was not formally ablated across the full set.

Implications for AI Economics

• Token-cost engineering: selectively loading skill text can reduce billed input tokens substantially for large, compressible skills. Savings translate directly into lower per-call input costs under token-billed pricing, but only when unused optional content is large relative to loading/round-trip overhead.
• Billing models and caching matter: provider logs that include cache-read tokens can overstate incremental processing cost if cache reads are treated as newly processed. Deployment cost models should discount cache-read tokens (authors use 10% as a heuristic) when estimating per-turn marginal cost for stateful agents.
• Architectural trade-offs:
  • Hybrid (stubs + fetch) is a strong general-purpose, content-preserving strategy for single-turn and mixed regimes; it balances always-on footprint and fetch frequency.
  • Tool-based Skill Block can achieve large savings for big, compressible skills but is sensitive to guidance (how strongly the model is instructed to fetch) and to additional round trips.
  • Reference selection is useful but often less effective than hybrid or Skill Block when block fetch probabilities are low or when selection itself is frequent.
  • Static pruning (permanent removal of rare blocks) can be cheapest if reduced coverage is acceptable; this is a trade between maintenance/coverage risk and token-cost savings.
• Deployment heuristics:
  • Estimate load probabilities p_i and block sizes s_i; apply the break-even condition (unused mass must outweigh repeated-footprint + tool overhead) before choosing a mechanism.
  • Consider provider-specific system prompts, caching semantics, and tool-call latency; these materially change the break-even point.
  • Account for generated-output sensitivity (clients that pay more attention to output tokens should include output-weighted cost metrics like total_λ).
• Operational caution:
  • Routing failures can degrade task performance; while the paper found no detected quality drop across paired comparisons, this does not prove safety/robustness of routing. Monitoring and execution-feedback recovery mechanisms matter.
  • Results depend on endpoint configuration and on how retrieval instructions are phrased; small instruction changes can alter load frequencies and therefore costs.

Practical rule-of-thumb from the study: - Use conditional loading (hybrid or Skill Block) when a skill is large and much of its text is optional per request, especially in multi-turn tasks where persistent always-on text would reappear across many turns. - Avoid complex on-demand tooling for tiny skills or procedures that are used essentially every turn—static pre-loading or small stubs may be simpler and cost-competitive. - If coverage can be sacrificed, static pruning may yield the biggest token savings with the simplest runtime cost model.