Distilling prior runs into reusable 'skills' stabilises LLM agent execution—reducing setup, format and verification errors—because skills act as procedural anchors rather than mere fact injections; however, as skill libraries grow or contexts change, retrieval failures and contextual mismatch remain the main bottlenecks.

Main Finding

Skills help agents primarily by acting as procedural anchors that stabilize noisy multi-step execution (not by injecting missing factual knowledge). Distilled SKILL artifacts outperform raw workflow memory in matched settings (≈ +6.06 percentage points) because they compress and standardize procedures. However, skills are not universally reliable: retrieval difficulty, confusable distractors, context mismatch, and brittle assumptions create separate failure modes. Exact retrieval of the ground-truth skill is neither necessary nor sufficient for success.

Key Points

• Dominant mechanism: procedural anchoring (65.7% of skill-use cases in the authors’ taxonomy). Skills supply usable ordering, checklists, tool sequences and verification plans that reduce execution-layer failures.
• Knowledge injection is rare: only ~4.5% of cases are explained by skills supplying facts the agent lacked.
• Skills vs Workflow Memory: skills improve task success over workflow memory by ~6.06 percentage points in matched experiments; workflow memory tends to preserve noisy exploration, failed branches and verbose traces that can increase timeouts and drift.
• Retrieval is a critical bottleneck:
  • As candidate-pool size grows from 5 to 100, actual-use precision drops from 29.6% to 3.3%.
  • Hard-negative/confusable distractors substantially reduce offline identification accuracy (embedding ranking and explicit selection), but downstream execution success does not fall in direct proportion: related non-ground-truth skills can still be useful.
  • Selecting the “correct” skill does not guarantee success; invocation compatibility, adaptation, and runtime constraints matter.
• Failure modes: brittle assumptions embedded in skills, applied in incompatible contexts, weak/no adaptation when invoked, resource/timeouts, capability/safety limits, and verification/infrastructure errors.
• Taxonomy: the authors consolidate observations into 3 high-level categories and 12 skill-use modes (examples: guided success, execution & verification failures, invocation & budget problems).

Data & Methods

• Datasets & benchmarks:
  • Controlled experiments on Terminal-Bench (realistic command-line tasks with isolated environments and test-based verification) and SkillsBench.
  • Collected balanced pools of successful and failed trajectories per task; constructed fixed-budget composition grids (mixtures labeled 5s0f → 0s5f).
• Trials and coding:
  • Normalized 8,135 trial records from controlled experiments.
  • Sampled 240 trajectories for open coding; 238 valid unique labels consolidated into the final taxonomy.
• Models and agent pairings:
  • Primary experiments: Codex + GPT-5.3-Codex and Gemini CLI + Gemini-3.1-Pro-Preview.
  • Retrieval diagnostics used Qwen3-Embedding-0.6B; one retrieval set used GPT-5.4 for Codex pairing (availability caveat).
• Experimental conditions:
  • Representation comparison: Raw (no prior experience), Workflow Memory (cleaned traces), Skill (distilled SKILL.md from same traces).
  • No-hint setting: same trajectories but remove explicit success/failure annotations to separate outcome-signal effects.
  • Cross-framework transfer: artifacts built in one agent framework (Codex) tested in another (Gemini CLI).
  • Retrieval experiments (three arms):
    • Arm 1: embedding-based ranking without execution.
    • Arm 2: explicit agent selection (no execution).
    • Arm 3: full-pool real execution (skill selection and invocation during task runs).
  • Pools: each task paired with a candidate pool containing the ground-truth skill plus k−1 real distractors (random / similar / dissimilar).
• Evaluation protocol:
  • Tasks executed in fixed Docker environments; verifier tests used to measure success.
  • Typical downstream execution: n = 5 unique trials per task, parallelism 20.
  • Outcomes logged and trajectories analyzed to attribute mechanism labels.
• Key quantitative metrics reported:
  • Skill advantage over Workflow Memory: +6.06 points (matched comparisons).
  • Procedural anchoring share: 65.7%; Knowledge injection: 4.5%.
  • Retrieval precision fall: 29.6% → 3.3% when pool size increases 5 → 100.

Implications for AI Economics

• Value creation and ROI
  • Distillation investment: producing compact, standardized skills yields outsized operational gains relative to storing raw traces. Firms should prioritize tooling and processes that distill frequent successful procedures into reusable skills.
  • Marginal returns diminish with library size unless retrieval quality scales: adding more skill artifacts without improving retrieval or curation can sharply reduce usable precision and raise search costs.
• Product and market design
  • Skill marketplaces and services should price/position offerings by retrieval cost and confusability: highly specific, high-utility procedural skills are more valuable but require better indexing and disambiguation.
  • Portability and standards matter: skills that are framework-agnostic or follow common interfaces have higher cross-product value; standardizing SKILL.md-like formats increases the transferable economic value of procedural assets.
• Organizational learning and labor
  • Automation of repetitive procedural tasks benefits most from skills that stabilize execution; this lowers repeated rediscovery costs and reduces variance in outcomes, improving predictability for downstream business processes.
  • However, brittle skills can create systemic risk—poorly adapted skills can propagate failures. Organizations need governance (testing, validation, monitoring) and incentives for continuous adaptation and curation.
• Infrastructure and operational costs
  • Retrieval infrastructure (embeddings, hard-negative handling, ranking) becomes a central economic lever. Investment in retrieval quality and candidate-pruning is often more valuable than simply expanding storage.
  • Verification and runtime debugging tooling mitigate the non-sufficiency of correct skill retrieval; investments here reduce downstream execution failure costs.
• Policy and safety
  • Because skills are procedural and can embed brittle assumptions, regulation and auditing standards for automated procedural assets (especially in safety-critical domains) may be warranted.
  • Economic incentives should favor robust, adaptable skill design and explicit provenance (origin, confidence, applicability bounds).
• Practical recommendations (economic priorities)
  • Prioritize distillation pipelines that produce concise procedural anchors rather than raw trace dumps.
  • Invest in retrieval systems that handle confusable distractors (hard-negative training, metadata, context-sensitive ranking).
  • Maintain hybrid stores: keep both distilled skills and some richer workflow traces for debugging and adaptation—use skills for execution, traces for revision.
  • Monitor library growth costs and employ curation/pruning to preserve high precision.
  • Standardize skill representation for transferability across agent frameworks to increase reusability and market scope.

Limitations to consider when applying these results - Experimental scope used specific benchmarks (Terminal-Bench, SkillsBench) and particular agent–model pairings; numeric magnitudes may differ in other domains (e.g., open web, robotics). - Some RQ4 experiments used a different model pairing due to availability (GPT-5.4 vs GPT-5.3), so cross-RQ numeric comparisons should be interpreted cautiously. - The study focuses on short-horizon verification via test-based evaluation; long-run adaptation and continuous learning dynamics may reveal additional economic trade-offs.

Summary takeaway: In economic terms, invest in producing and curating distilled procedural skills and in the retrieval/verification infrastructure that makes them actionable—this yields predictable, reusable value—but avoid assuming that increasing the count of stored artifacts alone will scale benefit.