37 cumulative citations
View corpus contextMalicious third‑party 'skills' in LLM agent registries are rare but potent: 157 of 98,380 skills contained confirmed attacks exploiting hundreds of vulnerabilities, largely driven by one templated threat actor and removed after disclosure.
"Do Not Mention This to the User": Detecting and Understanding Malicious Agent Skills in the Wild
A systematic analysis of 98,380 LLM skills from two major registries found 157 confirmed malicious skills exploiting 632 vulnerabilities across 13 attack techniques—often concentrated in a single actor using templated brand impersonation—and all flagged skills were removed after disclosure.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
LLM-based coding agents increasingly rely on third-party extensions called skills, which bundle natural language instructions and helper scripts that execute with full user privileges. Community registries have emerged to distribute these skills, but the security implications remain unstudied due to the absence of labeled threat data. This paper presents a systematic security analysis of 98,380 skills collected from two major registries. Through a combination of static pattern matching and dynamic behavioral verification, we identify 157 skills exhibiting confirmed malicious behavior, encompassing 632 distinct vulnerabilities across 13 attack techniques. Our analysis reveals that these threats are deliberate rather than accidental: each malicious skill contains an average of 4.03 vulnerabilities spanning multiple attack phases. We identify two dominant attack strategies with statistically significant negative correlation -- credential theft via remote code execution, and agent manipulation through adversarial instructions embedded in documentation. Over half of all confirmed cases originate from a single threat actor employing templated brand impersonation at scale. We further observe that attack sophistication correlates with concealment investment, with advanced skills universally employing undocumented capabilities while also exploiting platform-native trust mechanisms. Following responsible disclosure, registry maintainers removed all 157 (100%) of the reported skills. Our dataset and detection pipeline are publicly available to facilitate future research on securing LLM agent ecosystems.
Summary
Main Finding
The authors build the first large-scale, behaviorally-verified dataset of malicious LLM agent “skills” and show that a small but meaningful fraction of public skills are deliberately weaponized. From 98,380 skills crawled from two community registries, a pipeline of static triage plus sandboxed execution produced 157 behaviorally-confirmed malicious skills (0.16%) containing 632 labeled vulnerabilities. Malicious skills tend to layer multiple attack techniques (avg. 4.03 vulnerabilities / skill), cluster into two distinct attacker archetypes (Data Thieves vs. Agent Hijackers), and frequently rely on concealment and platform trust to evade detection. After responsible disclosure, maintainers removed all 157 reported skills.
Key Points
- Scope and scale
- Dataset: 98,380 skills from two public registries (snapshot Jan 2026).
- Static-phase candidates: 4,287 (4.4%).
- Dynamically-triggered candidates: 762 (0.78%).
- Confirmed malicious skills: 157 (0.16%) with 632 vulnerabilities across 13 techniques.
- Attack characterization
- Two dominant, negatively correlated archetypes:
- Data Thieves: credential theft and supply-chain exfiltration, often via remote code execution.
- Agent Hijackers: instruction-level manipulation embedded in SKILL.md (prompt injection / instruction override).
- Multi-phase kill chains are common: median of 3 attack phases per malicious skill.
- A single industrialized actor produced 54.1% of malicious skills using templated brand impersonation.
- Two dominant, negatively correlated archetypes:
- Evasion & sophistication
- Higher sophistication correlates with greater investment in concealment (e.g., undocumented/hidden capabilities, obfuscated payloads).
- Advanced attacks exploit platform-native trust mechanisms (skills run locally with pre-granted user privileges).
- Detection performance & impact
- Behavioral verification pipeline achieves 99.6% precision; static-only baselines had ≤1.1% precision.
- Responsible disclosure resulted in 100% (157/157) removal from the registries.
- Artifacts released
- The labeled dataset, detection pipeline, and analysis scripts are publicly available for future research.
Data & Methods
- Data sources
- skills.rest: 25,187 skills (2,337 GitHub repos).
- skillsmp.com: 73,193 skills (8,909 GitHub repos).
- Combined snapshot: 98,380 skills (January 2026).
- Note: private skills, enterprise deployments, MCP servers, and vendor-curated stores are not covered.
- Static triage
- Defined 14 detection patterns mapped to six attack phases (Reconnaissance, Credential Access, Execution, Defense Evasion, Exfiltration, Impact).
- Code-level detection via regex over bundled scripts (Python/Shell/JS) for patterns like remote execution, external transmission, credential harvesting, obfuscation.
- Instruction-level detection used an LLM analysis agent (GPT-5.2) to detect prompt-injection-like patterns in SKILL.md (hidden instructions, instruction override, behavior manipulation).
- Static scan flagged 4,287 candidate skills (4.4%).
- Behavioral verification (sandboxed dynamic analysis)
- Each candidate executed in isolated Docker containers with monitoring: tcpdump, strace/dtrace, auditd, honeypot credentials.
- Activation modes: documented entry points, LLM-generated synthetic inputs, and multi-round invocations.
- Runtime indicators filtered 762 skills for manual review (e.g., exfiltration of honeypot credentials, network connections to attacker domains, decoded obfuscated payloads).
- Manual labeling
- Two expert reviewers inspected runtime logs and code, labeled 632 vulnerabilities across 157 confirmed malicious skills.
- Labels include both code-level and instruction-level behaviors; confirmation required observed malicious behavior and evidence of intentional abuse (not accidental bugs).
- Validation & metrics
- Behavioral verification precision: 99.6%.
- Static-only precision: ≤1.1% (appendix).
- Post-disclosure remediation: all 157 removed from registries.
- Limitations noted by authors
- Snapshot limited to two community registries and January 2026 timeframe.
- Private/enterprise skills, MCP-based integrations, and platform-curated stores not included.
- Static pattern set may miss novel techniques (false negatives possible).
Implications for AI Economics
- Market externalities and trust
- Negative externalities: even a small fraction of malicious skills can produce outsized harms (credential theft, supply-chain compromise), damaging user trust across platforms and slowing adoption of agentic tools.
- Reputation and information asymmetries: users cannot easily assess skill safety; registries with low vetting create a market for trust and certification services.
- Platform design, governance, and liability
- Local-execution model (skills run with user privileges) creates a clear market need for platform-level governance (vetting, code signing, permission controls, minimum sandboxing) and raises questions about liability allocation between registries, platform vendors, and end users.
- Policy interventions (mandatory vetting, disclosure requirements) or private certification markets could internalize security externalities.
- Concentration risk & attacker economics
- High concentration of malicious supply (one actor ~54% of confirmed cases) indicates low-cost, high-leverage attack strategies (templated impersonation). This implies attackers face low marginal cost to scale abuse, increasing expected returns and lowering required deterrence/investment to exploit the ecosystem.
- Costs and incentives for defenders
- Effective detection requires dynamic verification and manual review—costly processes. Economic models should consider trade-offs between friction (vetting delays, developer onboarding costs) and security benefits.
- The large gap between static-only and behavioral verification precision suggests that investments in runtime monitoring or sandboxed testing have high marginal value.
- Insurance, liability, and pricing of skills
- Potential for an insurance market: insurers could price cyber-insurance for users/orgs based on the provenance and vetting status of installed skills.
- Skills from unvetted registries should carry higher risk premia; market mechanisms (ratings, verified badges, escrow) could emerge to signal lower risk and command higher prices or faster adoption.
- Research and measurement opportunities for economists
- Estimate social costs: combine incidence rates from this dataset with downstream breach costs to estimate expected loss per skill/installation and optimal investment in vetting.
- Study adoption dynamics: how security incidents affect demand for agentic extensions and incumbents’ incentives to provide curated marketplaces.
- Model platform competition: platforms that enforce stricter vetting may incur higher onboarding costs but attract higher-trust users—characterize equilibria and welfare.
- Analyze attacker–defender investment games: optimal deterrence levels given low marginal cost of templated malicious skill publishing.
- Use released dataset as empirical input: actor-level concentration, distribution of techniques, and remediation outcomes (100% removal after disclosure) can calibrate structural models of incentives and enforcement effectiveness.
- Policy and practical recommendations (economic framing)
- Create market signals for trusted skills (third-party audits, code-signing/certificates) to reduce information asymmetry.
- Consider regulatory minimums for marketplaces (basic static scanning + mandatory sandbox execution or attestation) — quantify benefits vs. compliance costs.
- Incentivize platform owners to internalize externalities (liability rules, mandatory disclosure of vetting practices) so that private incentives align with social welfare.
- Support public-good detection infrastructure (shared sandboxes, labeled datasets) to lower collective defense costs and information collection barriers.
Summary note on applicability and limits - The paper provides rich, labeled empirical inputs for modeling the economics of agent-skill ecosystems, but its snapshot excludes private and enterprise distributions and vendor-curated stores. Any economic estimates derived from the dataset should account for this coverage limitation and potential time-variation as marketplaces and vendor policies evolve.
Assessment
Paper Typedescriptive
Evidence Strengthhigh — Large-scale dataset (98,380 skills) from two major registries, combined static pattern matching with dynamic behavioral verification to confirm malicious behavior, and real-world validation via responsible disclosure that led to removal of all reported skills, which strongly supports the descriptive claims about prevalence and tactics.
Methods Rigorhigh — The study uses a systematic pipeline combining static indicators and runtime verification to reduce false positives, quantifies vulnerabilities and attack techniques, performs statistical analysis of tactic correlations, and conducts responsible disclosure with registry removal—demonstrating robust operational and methodological checks, though some detection and sampling limitations remain.
Sample98,380 LLM 'skills' (extensions bundling NL instructions and helper scripts) collected from two major public community registries; analysis included static pattern matching across all skills and dynamic behavioral verification for suspected cases, yielding 157 confirmed malicious skills and 632 distinct vulnerabilities spanning 13 attack techniques.
Themesgovernance adoption
GeneralizabilityData limited to two public registries and may not represent private/enterprise skill stores or proprietary marketplaces, Snapshot in time — adversaries and defenses evolve, so prevalence and tactics may change, Detection pipeline may miss highly stealthy or time-delayed attacks (false negatives) and is tuned to executable/scripted skills, Findings emphasize skill-based agents and may not generalize to other LLM integration modalities (APIs, embedded assistants) or non-coding agent ecosystems
Claims (11)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| We collected 98,380 skills from two major registries. Other | null_result | dataset_size |
Reading fidelity
high
Study strength
high
|
n=98380
|
| We identify 157 skills exhibiting confirmed malicious behavior. Ai Safety And Ethics | negative | count_of_malicious_skills |
Reading fidelity
high
Study strength
high
|
n=157
157
|
| The 157 malicious skills encompass 632 distinct vulnerabilities across 13 attack techniques. Ai Safety And Ethics | negative | vulnerability_count_and_attack_techniques |
Reading fidelity
high
Study strength
high
|
n=157
632 distinct vulnerabilities across 13 attack techniques
|
| Each malicious skill contains an average of 4.03 vulnerabilities spanning multiple attack phases. Ai Safety And Ethics | negative | vulnerabilities_per_skill |
Reading fidelity
high
Study strength
medium
|
n=157
4.03 vulnerabilities
|
| Two dominant attack strategies are credential theft via remote code execution, and agent manipulation through adversarial instructions embedded in documentation, and these two strategies show a statistically significant negative correlation. Ai Safety And Ethics | negative | co-occurrence_of_attack_strategies |
Reading fidelity
medium
Study strength
medium
|
n=157
|
| Over half of all confirmed cases originate from a single threat actor employing templated brand impersonation at scale. Ai Safety And Ethics | negative | share_of_cases_from_single_actor |
Reading fidelity
high
Study strength
medium
|
n=157
Over half
|
| Attack sophistication correlates with concealment investment: advanced skills universally employ undocumented capabilities while also exploiting platform-native trust mechanisms. Ai Safety And Ethics | negative | relationship_between_attack_sophistication_and_concealment |
Reading fidelity
medium
Study strength
medium
|
not reported
|
| Following responsible disclosure, registry maintainers removed all 157 (100%) of the reported skills. Governance And Regulation | positive | removal_rate_after_disclosure |
Reading fidelity
high
Study strength
high
|
n=157
100%
|
| LLM-based coding skills bundle natural language instructions and helper scripts that execute with full user privileges. Ai Safety And Ethics | negative | privilege_level_of_skill_execution |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Security threats in skills are deliberate rather than accidental. Ai Safety And Ethics | negative | intentionality_of_threats |
Reading fidelity
medium
Study strength
speculative
|
n=157
|
| Our dataset and detection pipeline are publicly available to facilitate future research on securing LLM agent ecosystems. Research Productivity | positive | data_and_tool_availability |
Reading fidelity
high
Study strength
low
|
not reported
|