An AI agent nearly rivalled human pentesters in a live campus network: ARTEMIS discovered nine valid vulnerabilities with an 82% validation rate and outperformed nine of ten professionals, with some variants operating at about $18/hour versus $60/hour for human testers; existing AI scaffolds lagged, and agents still suffer higher false positives and struggle with GUI-driven tasks.

Main Finding

ARTEMIS, a purpose-built multi-agent AI scaffold, achieves near-human performance in a live enterprise penetration test. In a head-to-head study on a university network (~8,000 hosts, 12 subnets), ARTEMIS placed second overall: it produced high-quality, technically sophisticated exploits (9 validated vulnerabilities; ~82% valid submission rate as reported) and outperformed most existing AI scaffolds and the majority of participating human pentesters. Key limitations remain: higher false-positive rates and difficulty with GUI-based tasks. ARTEMIS-style agents offer systematic, parallel, and much lower-cost offensive capability relative to human professionals.

Key Points

• Comparative setup: 10 professional pentesters vs six prior AI agents plus ARTEMIS in a live university environment (mixed Unix/IoT/Windows, Kerberos auth). Participants had ~10 hours; agents were run up to 16 hours (first 10 hours used for direct comparability).
• ARTEMIS architecture: supervisor + dynamically spawned sub-agents + automatic triager. Features dynamic prompt generation, task-specific system prompts, long-horizon session management, and a reproduction/severity triage pipeline.
• Performance summary:
  • ARTEMIS placed second on the leaderboard (second-most severe/complex submissions).
  • Reported ARTEMIS outcome: 9 valid vulnerabilities, ~82% valid submission rate, and outperformed 9 of 10 human participants in overall ranking (per abstract).
  • Human cohort found 49 validated unique vulnerabilities in total; individual human results varied (3–13 valid findings each).
  • Existing scaffolds (Codex, CyAgent, Claude Code, MAPTA, Incalmo) underperformed: many produced only low-complexity scanner-style findings or refused/failed due to model refusals or brittle architectures.
• Strengths of ARTEMIS/agents:
  • Systematic enumeration and repeatable scanning at scale.
  • Parallelism (multiple concurrent sub-agents) enabling faster multi-host exploitation.
  • Cost efficiency: certain ARTEMIS variants estimated ~$18/hour vs ~$60/hour for professional testers.
  • Long-horizon task persistence—ARTEMIS operated across sessions and resumed work without losing context.
• Weaknesses/risk vectors:
  • Higher false-positive rate compared to humans (triage module mitigates but does not eliminate).
  • Struggles with GUI-driven tasks and complex human-interactive steps.
  • Agents can be brittle/adversarially attacked; require monitoring and safe-harbor controls in live environments.
• Safety/operational controls: dual-layer monitoring during runs (researcher and IT), VDP and IRB procedures, session termination capability. No agent went out of scope in the study.

Data & Methods

• Target: a research university Computer Science network — ~12 subnets, ~8,000 hosts; 7 public subnets, 5 VPN-only subnets; heterogeneous hosts (mostly Unix, some Windows, IoT).
• Participants: 10 cybersecurity professionals recruited via industry channels; self-reported experience; each given a standardized Kali VM on GCP; compensated $2,000 each; asked for ≥10 hours engagement.
• Agents tested: ARTEMIS (two configurations: A1 with GPT-5 supervisor/subagents; A2 ensemble supervisor with Claude Sonnet 4 subagents), Codex, CyAgent (variants), Claude Code, Incalmo, MAPTA.
• ARTEMIS design:
  • Supervisor issues work, spawns arbitrary sub-agents with dynamically generated expert prompts.
  • Note-taking, structured task lists, summarization to maintain long runs.
  • Triager: three-phase verification (initial review, repro/validation, severity classification/CVSS-like scoring).
• Scoring framework:
  • Unified metric combining technical complexity (detection complexity + exploit complexity; exploitation rewarded, verification-only penalized) and weighted criticality (exponential-like weights mirroring bug-bounty payouts: Critical=8, High=5, Medium=3, Low=2, Informational=1).
  • MITRE ATT&CK mapping used to categorize techniques.
• Safety: informed consent, recording, VDP-defined safe harbor, dual monitoring to terminate sessions if needed.
• Benchmarks: ARTEMIS showed no scaffold uplift on single-host CTF benchmark (Cybench) beyond model effects — scaffold improves execution/planning in complex, noisy production environments rather than model knowledge.

Implications for AI Economics

• Labor substitution and cost structure:
  • Significant unit-cost advantages for routine/scaleable penetration tasks: reported ~$18/hour for ARTEMIS variants vs ~$60/hour for human professionals. This suggests AI agents can undercut human labor for many standard scanning/enumeration/exploitation activities.
  • Reduced marginal cost per task and ability to horizontally scale (spawn many sub-agents) imply higher throughput and lower average cost of offensive capabilities.
• Demand-shift within cybersecurity labor:
  • Routine pentest work (scanning, standard exploit chains, verification-lite tasks) may be increasingly automated, depressing demand—and potentially wages—for lower-skill pentesters.
  • Demand will likely rise for higher-skill roles: oversight, triage, reproducibility verification, GUI/human-interaction tasks, red-team strategy, and incident response. Human labor will shift toward verification, judgment, and handling corner cases where agents fail.
• Productivity and market structure:
  • Higher productivity from agents could lower costs of defensive testing (if defenders adopt) and increase the supply of “red-team” labor, potentially improving baseline security if widely used by defenders.
  • Conversely, attackers adopting similar tools lower the cost of attack, increasing expected frequency of incidents—creating asymmetric adoption externalities and potential adverse selection for insurers.
  • Markets for pentesting and bug-bounty programs may reprice: downward pressure on routine payouts, while payouts for complex/novel exploits may rise due to scarcity of human expertise.
• Verification/triage as a new bottleneck:
  • The study shows higher false-positive rates for agents; economic value shifts toward triage/verification labor and tooling. A new market may emerge for standardized triage services, reproducibility infrastructure, and automated exploit-proofing.
• Systemic risk and externalities:
  • Horizontal scalability of agentic attacks increases tail risk: an effective automated agent can be replicated/parallelized cheaply, raising potential aggregate loss and systemic cyber risk.
  • Cyber insurance markets, firms, and regulators may need to factor in agents’ marginal impact on attack frequency and severity when pricing coverage or mandating controls.
• Policy and investment implications:
  • Regulatory attention may be warranted (access controls, model governance, usage auditing) given dual-use harms and the potential for cheaper, faster offensive operations.
  • Investment priorities should include detection techniques oriented toward agentic behavior (e.g., anomalous parallelized probes, AI-driven exploit patterns), robust triage pipelines, and architectural mitigations that raise the cost for automated exploitation (hardening GUIs, multi-factor human interactions).
• Open research directions (economic questions):
  • Formal modeling of attacker-defender adoption equilibria when agentic tools are available to both sides.
  • Quantification of marginal productivity of agents across pentest task types; decomposition of how much value accrues to scaffolding vs model improvements.
  • Impact on wages and employment composition in cybersecurity (task-level automation vs augmentation).
  • Market design for bounties/inspections when automated submissions have higher false-positive rates—mechanisms to discourage spamming while not disincentivizing defender use.

Takeaway: ARTEMIS demonstrates that scaffolded, multi-agent AI can reach near-top human performance on complex, real-world penetration tasks and do so at much lower cost—this creates both productive opportunities (cheaper defensive testing) and heightened risks (lower-cost offensive capabilities). Economically, the most important near-term effects will be reallocation of human effort toward triage/oversight and structural shifts in pricing, insurance, and regulation to address the externalities introduced by scalable agentic offensive tooling.