Autonomous AI agents will make tailored cyberattacks cheap and scalable, overturning defenders' reliance on attacker labor scarcity; to contain the threat, governments and firms must develop and tightly govern offensive AI capabilities and testing infrastructure.

Main Finding

AI agents will make large-scale, economically viable cyberattacks inevitable by automating vulnerability discovery, exploitation, and post-exploitation customization at low marginal cost. Traditional model-centric safeguards (data filtering, alignment, guardrails, access controls, representation edits) are insufficient against adversaries who can run or fine-tune open-weight agentic models. Defenders must therefore build and operate offensive AI agents—under strict governance—as essential defensive infrastructure to predict, stress-test, and mitigate attacker behavior at scale.

Key Points

• Threat model: financially motivated, technically capable adversaries with access to state-of-the-art agentic models (APIs or local), seeking to maximize aggregate profit across many heterogeneous victims rather than damage to single high-value targets.
• Why agents change the economics:
  • Automation reduces per-attempt cost dramatically; attackers only need low success rates to be profitable.
  • Agents can parallelize scanning/exploitation across thousands of targets, making long-tail, niche systems viable targets.
  • Agents can adapt, chain exploits, and perform tailored post-exploitation actions, raising effective yield per compromise.
• Limitations of current defenses:
  • Data governance: removing exploit examples doesn't stop reasoning and synthesis capabilities; agentic models can acquire new info at inference time.
  • Safety alignment: jailbreaks and objective distortion, plus alignment degradation under fine-tuning, undermine effectiveness.
  • Representation engineering: brittle across new contexts and long-horizon agentic behaviors.
  • Output guardrails: fail to detect malicious multi-step agent workflows or self-hosted/open-weight deployments.
  • Access/deployment controls: leak/replication and low-cost fine-tuning make gated controls porous once models proliferate.
• Proposed defensive shift:
• Build comprehensive, dynamic benchmarks that cover the full attack lifecycle (reconnaissance, exploit chaining, command-and-control, post-exploitation, etc.) and realistic system variations.
• Move from workflow-based tooling to trained offensive agents capable of discovering in-the-wild vulnerabilities and composing multi-step attacks.
• Institute governance: restrict offensive agents to audited cyber ranges, require strict logging/auditing, and distill offensive findings into defensive-only agents and mitigations.
• Empirical snapshot: Table summarizing SOTA performance on varied security benchmarks shows uneven capabilities—agents do better on local/small-scale generative tasks (e.g., short-function patching) than on large-project analysis, exploit chaining, or full PoC generation—suggesting important capability gaps but rapid progress.

Data & Methods

• Paper type: conceptual/position paper combining threat modeling, literature synthesis, and a curated summary of existing benchmark results.
• Evidence sources:
  • Cited empirical and theoretical work on AI-assisted code generation, agent capabilities, economics of cybercrime, and cybersecurity practice.
  • Aggregated benchmark performance (Table 1) across multiple public/red-team/bench datasets (e.g., CyberSecEval, AutoPenBench, VulnLLM, CyberGym, SWE-bench) showing SOTA agent metrics by task (attack generation, CTF, vulnerability detection, PoC generation, patching).
• Methodological proposals:
  • Construct new benchmarks based on MITRE frameworks and the cyber kill chain with dynamic execution environments (containerized/simulated systems) and playbook-driven tasks to better emulate real, multi-step attacks.
  • Train and evaluate agentic offense/defense agents within controlled ranges; measure lifecycle coverage, exploit-chaining ability, adaptive tool use, and post-exploitation monetization steps.
• Limitations acknowledged: much of the paper is analytic and normative rather than experimental; benchmark snapshot shows current capabilities but is not an exhaustive empirical evaluation of deployed adversarial agents.

Implications for AI Economics

• Lowered attacker costs and scale effects:
  • Automation shifts attacker cost structure from high fixed costs per exploit to low marginal costs per target, increasing expected ROI even at low per-target success rates.
  • This makes previously uneconomical targets (long-tail, niche, poorly maintained systems) profitable to attack, expanding the market size for cybercrime.
• Market and incentive effects:
  • Demand for offensive AI capability will grow (black/gray markets and legitimate defenders), creating incentives for actors to develop or repurpose agents even where model governance exists.
  • Defenders will need to internalize offensive capability (invest in agents, cyber ranges, skilled staff), shifting budgets and labor markets in cybersecurity toward agent development and range operations.
  • Entities controlling audited cyber ranges, curated benchmarks, and defensive-distilled agents may capture strategic economic value and influence.
• Externalities and public-good problems:
  • Widespread agent-driven exploitation creates systemic risk (fast propagation, cross-domain compromise) that individual firms may underinvest to mitigate—heightening the role for public-sector coordination, standards, and shared defensive infrastructure.
  • Insurance markets and cyber risk pricing may be disrupted by rapidly changing attack surfaces and correlated failures.
• Regulatory and governance trade-offs:
  • Restricting offensive agents imposes costs (slower defensive learning, need for tightly controlled research), while permissive openness accelerates defensive knowledge but also abuse risk—policy must balance learning versus misuse.
  • Transparency via shared benchmarks and audited ranges can reduce information asymmetries but requires strong access controls to avoid leakages that lower attacker R&D costs.
• Potential for an arms race:
  • If defenders adopt offensive agents, attackers are incentivized to develop more capable agents, amplifying compute and talent races and potentially increasing concentration of economic power among well-resourced actors.

Overall, the paper reframes offensive AI capability not as an optional or purely malicious capability to be suppressed, but as an essential component of defensive strategy. From an AI-economics perspective this creates new markets, shifts incentives, increases externalities and public-good needs, and calls for governance architectures that reconcile defensive learning with abuse containment.