Agent Skills materially improve performance for larger open-source models (≈30B–80B) but fail for very small models; 'thinking' modes add little benefit while raising GPU costs, forcing a trade-off between accuracy and compute for resource-constrained industrial deployments.

Main Finding

Agent Skills (a skill-library / on-demand SKILL.md approach) can substantially improve performance of open-source models in industrial tasks — but only above a model-size threshold. Small models (<30B, especially <4B) frequently fail at skill selection and derive limited benefit; mid-to-large open models (≈30B–80B) gain large accuracy improvements under the Agent Skill Instruction (ASI) paradigm. However, “thinking”-style model variants can incur massive GPU-time costs (overthinking) with little or no accuracy gain, producing an unfavorable cost–performance tradeoff for industrial deployments.

Key Points

• Definition & setup

  • SLMs defined here as models <30B parameters; experiments include open-source models from 270M to 80B and one closed model (gpt-4o-mini).
  • Three prompt/context engineering strategies compared:
    • DI (Direct Instruction): minimal user prompt
    • FSI (Full-Skill Instruction): entire skill repository is provided upfront
    • ASI (Agent Skill Instruction): model decides whether to fetch skill info on demand
  • Live tool execution purposely excluded in main experiments to isolate skill-selection effects.

• Core empirical results

  • ASI is the most effective CE strategy for medium/large open models; it commonly delivers the largest accuracy gains while keeping reasonable GPU-time.
    • Example: Qwen3-80B-Instruct on IMDB: DI ACC 0.307 → ASI ACC 0.950 (Skill ACC 0.997). On FiNER: DI 0.194 → ASI 0.648.
  • Very small models (e.g., Gemma-3-4B-it, Gemma-3-270M-it) struggle to select the correct skill from a short list of distractors and thus gain little from the skill framework.
  • Error decomposition:
    • For SLMs (<12B) the dominant failure mode is skill-selection (retrieval) error.
    • For ≥30B models retrieval errors decline and remaining errors shift to parsing/ reasoning/format compliance.
  • “Thinking” variants (Qwen3-80B-Thinking) often produce very large VRAM·Time (GB·min) increases (e.g., ~18–34× higher than “Instruct” under same method) but without proportional accuracy improvements — demonstrating “overthinking” and extreme computational waste.
  • Robustness to skill-hub size:
    • Small models’ skill-selection accuracy decays quickly beyond ~10–20 skills; models ≥12B (and especially ≥30B) remain robust even when N → 100 skills.
  • Chat history:
    • Including truncated chat history improves accuracy most for very small models but increases VRAM·Time; recommended selectively for lightweight SLMs only.
  • Minor implementation detail:
    • Replacing the keyword “Skill” with synonyms had minimal effect on performance.

Data & Methods

• Datasets
  • IMDB subset (binary sentiment; reviews controlled to 300–500 chars): 300 eval items.
  • FiNER (139 financial XBRL tags): 403 eval items.
  • InsurBench (proprietary insurance claims threads; long, noisy): 200 eval items. InsurBench chosen to minimize dataset contamination from model pretraining.
• Models evaluated (examples)
  • Gemma-3-270m-it (0.27B), Gemma-3-4b-it (4B), Gemma-3-12b-it (12B)
  • Qwen3-30B-Instruct (30B), Qwen3-80B-Instruct / Thinking / Coder (80B variants)
  • GPT-4o-mini used where permitted (not used on InsurBench for privacy).
• Experimental design
  • Temporary skill repository per task: ground-truth skill + 4–5 distractor skills.
  • Metrics:
    • Cls ACC (classification accuracy), Cls F1
    • Skill ACC (skill-selection accuracy)
    • Avg GT (minutes): average processing time per task
    • Avg VRAM Time (GB·min): average GPU-memory × time per task (intended as an operational cost proxy)
  • Main focus on isolating skill selection and subsequent execution correctness; follow-up ReAct-style experiments (with tool calls) reported separately.
• Key quantitative findings (representative)
  • ASI gives large gains for 30B–80B instruct/coder variants (e.g., Qwen3-80B-Instruct ASI: IMDB ACC 0.950; FiNER ACC 0.648; Skill ACC 0.997).
  • Tiny models often show Skill ACC well below usable thresholds (e.g., Gemma-3-4B-it and Gemma-3-270M-it struggled to reliably select correct skill).
  • Qwen3-80B-Thinking VRAM·Time examples: IMDB 94.8 GB·min vs Instruct 4.2 GB·min (≈22×), while accuracy remains lower than Instruct in many tasks.

Implications for AI Economics

• Cost vs. capability tradeoffs
  • Operational costs (GPU-hours and memory residency) are a first-order economic factor in agent deployments. “Thinking” variants can dramatically inflate per-task GPU·time with little accuracy gain, producing poor return on investment.
  • For many industrial use-cases requiring on-prem or private deployments (data security, regulatory constraints), mid-sized open-source models (≈30B–80B) combined with ASI provide a cost-effective alternative to proprietary APIs while preserving good task performance.
• Investment priorities for industry
  • Prioritize improving retrieval/skill-routing pipelines (indexing, prompting for selection, lightweight rerankers). The dominant failure mode in SLM deployments is routing, so investment here yields outsized ROI compared to naively scaling model size.
  • For low-budget deployments, prefer medium-size instruct/coder variants (e.g., 30B) over tiny models: medium-size models achieve much of the skill-related benefit at manageable VRAM·Time.
  • Avoid deploying “thinking” variants in production unless their additional reasoning tokens demonstrably improve accuracy for the specific task; otherwise they impose high OPEX.
• Policy & procurement consequences
  • Organizations that cannot use proprietary APIs (privacy/compliance) should budget for medium-sized on-prem models and GPU capacity; total cost modelling must account for VRAM residency effects that limit concurrency (blocking throughput).
  • Procurement decisions should weigh both model-to-skill alignment capability and operational cost metrics (GB·min), not model parameter count alone.
• Research & product recommendations
  • Develop and benchmark lightweight routing and skill-selection modules (cheap classifiers/rerankers) that front-load selection and keep bigger models idle unless needed.
  • Explore hybrid architectures: small/specialized routers + medium models for execution, to minimize average GPU·time while preserving accuracy.
  • Extend cost–benefit analyses across more industry datasets and tasks to create per-task model-size decision rules, and quantify break-even points vs. proprietary API costs.
• Summary take-away
  • Agent Skills are a practical and economically attractive CE pattern for industrial settings when paired with sufficiently capable open-source models (≈30B–80B) and on-demand skill loading (ASI). Tiny SLMs often fail at the critical routing step, and thinking-type models can waste substantial compute without proportional gains—so deployments should focus on retrieval engineering, selective history use, and careful model-variant choice to optimize cost-performance.