Reusable agent 'skills' frequently backfire: a contrastive study finds 307 skill-induced failures across popular benchmarks, with most correctness breaks caused by skills misguiding required implementation details and most cost blowouts driven by mandatory verification and heavy implementation pipelines.

Main Finding

Agent "skills" — reusable instruction packages loaded into LLM agents — can meaningfully harm agent performance. In a controlled, contrastive evaluation over two public skill benchmarks, the authors confirm 307 skill‑induced failures (125 functional failures and 182 high‑confidence efficiency regressions). Most harmful cases come from skills that are topically relevant but operationally inappropriate: they misguide implementations, force unnecessary procedures, or impose mandatory validation/overhead that doubles or worse the cost.

Key Points

• Dataset and scope
  • 307 confirmed skill‑induced failures from augmented evaluations on SkillsBench and SWE‑Skills‑Bench.
    • 125 functional failures
    • 182 efficiency regressions
  • Potential paired comparisons expanded from 826 to 20,664 by adding semantically matched public skills.
• Contrastive attribution method
  • Differential (paired) testing: a target run (with a skill) is compared to a reference run (no skill or a semantically matched skill) with task, verifier, model, repo state fixed; only skill setup varies.
  • Functional failure = target FAIL, reference PASS.
  • Efficiency regression = both PASS but both token use and execution time regress (min(rtok, rtime) > 1.0 and max(...) > T), with primary threshold T = 2.0 (i.e., ≥2× on at least one metric).
• Root causes — functional failures
  • The dominant cause is Task‑Implementation Fault (86/125; 68.8%): skills that appear relevant but cause the agent to implement or omit task‑required elements (wrong field, API behavior, formatting, calculation, etc.).
  • Other causes: wrong artifact locations (24 cases) and environment mismatches (13 cases).
  • Harm is rarely from obviously irrelevant skills; instead, subtle operational mismatches dominate.
• Root causes — efficiency regressions
  • Excessive Procedure dominates (114/182; 62.6%), not prompt length alone.
    • Within Excessive Procedure, excessive verification (67 cases) and heavy implementation pipelines (30 cases) are the largest contributors.
  • Context overhead cases are mostly due to mandatory skill body text (43/46).
  • Many skills turn optional checklists and construction recipes into mandatory, costly steps.
• Tooling (SKILLTRIAGE)
  • Built a taxonomy-guided attribution tool that (1) normalizes paired cases, (2) extracts differential evidence, and (3) produces triage reports.
  • Automated attribution accuracy (against manual labels): 111/125 (88.8%) for functional failures, 132/182 (72.5%) for efficiency regressions.
• Implementation details (high level)
  • Augmented curated benchmarks by searching public skill repositories (smithery.ai, skillsmp.com); retrieved candidates via all‑MiniLM‑L6‑v2 embeddings with cosine ≥ 0.7 (kept up to top‑5).
  • Executions run with OpenCode 1.15.1 and Claude Opus 4.6; recorded trajectories, verifier outputs, token use, and execution time.
  • Data refinement removed ambiguous/verifier‑narrow/duplicate cases; final set derived from 665 labeled candidates.

Data & Methods

• Benchmarks used: SkillsBench (84 tasks) and SWE‑Skills‑Bench (490 task instances), both with deterministic programmatic verifiers and executable environments.
• Skill augmentation: semantically matched public skills added to enlarge cross‑skill contrast space (embedding similarity threshold 0.7).
• Paired comparison types:
  • With/no‑skill: compare audited skill run vs. run with no skill.
  • Cross‑skill: compare audited skill run vs. semantically matched alternative skill run.
• Failure labeling:
  • Functional failures: audited skill run fails while reference run passes.
  • Efficiency regressions: audited and reference both pass, but both token and time regress and at least one metric exceeds 2× (T = 2.0).
• Execution environment and measurement:
  • Fixed task instruction, verifier, model, repository/container state, and agent runtime; only skill setup changed.
  • Logged full execution trajectories, tool calls, artifacts, token counts, wall‑clock execution time.
• Dataset counts and refinement:
  • From ~20k possible pairs, execution produced 665 labeled candidates (315 functional, 350 efficiency); after refinement and manual consensus labeling, 307 final confirmed cases.

Implications for AI Economics

• Reuse risk affects expected returns: skills are meant to be reusable assets. When a skill induces failures or large cost regressions, its reuse reduces expected task success rates and increases operational cost, lowering its net value. A marketplace or firm that monetizes or relies on skills must account for negative expected value from harmful skills.
• Hidden and repeatable costs: because skills are reusable and often topically relevant, a single harmful skill can repeatedly impose both functional and efficiency costs across many tasks and users, compounding total cost of ownership (TCO) and operational risk.
• Market design and incentives
  • Skill marketplaces need screening, continuous monitoring, and attribution tools (like SKILLTRIAGE) to detect and label harmful skills before broad distribution.
  • Reputation, certification, or insurance mechanisms should reflect both accuracy and cost impact (not just topical relevance).
  • Incentivize authors to declare optional vs. mandatory steps, provide environment assumptions, and include cost metadata (expected token/time overhead) to allow buyers to make informed choices.
• Procurement, pricing, and risk management
  • Procurement decisions for agent stacks should treat skills as stateful, operational dependencies with measurable risk profiles — price them accordingly or run A/B differential tests before deployment.
  • Cost models for agent-based workflows must include skill‑induced regressions (e.g., verification pipelines, forced rebuilds) when estimating marginal cost per task.
• Product & engineering tradeoffs
  • Defaulting to loading seemingly relevant skills without causal checks is economically suboptimal. Systems should make skill loading conditional, cost‑aware, and explainable to reduce locked‑in operational expense.
  • Design change: allow skills to be loaded in “advisory” mode (non‑mandatory checklists) and surface optional steps to agents/models, preserving benefits while reducing forced overhead.
• Research and policy priorities
  • Invest in automated attribution and audit tooling to rapidly detect skill harms at scale.
  • Develop marketplace standards for skill metadata (assumptions, validation scope, required environment) so buyers can price and manage risk.
  • Consider regulatory or governance guidance around repeated harmful artifacts in widely distributed agent skills (especially when skills affect safety‑critical or high‑cost tasks).

Short takeaway: skills are powerful reuse artifacts but can impose large, repeatable negative externalities — both on correctness and cost. Economically, marketplaces, firms, and procurement processes must treat skill reuse as a risk‑bearing asset and adopt contrastive/audit tooling, cost metadata, and incentive designs to capture true value.