0 cumulative citations
View corpus contextScaled open-weight models can match proprietary systems on offensive CTF tasks given more compute and remain cost-competitive; but defensive SOC investigations do not improve simply by adding reasoning budget—successful blue-team work requires disciplined tool use and SOC-native, cost-aware evaluations.
Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents
When evaluated by cost as well as success, offensive CTF tasks improve with additional inference compute so scaled open-weight models can be cost-competitive with proprietary systems, while defensive SOC investigations rely less on raw reasoning budget and more on disciplined tool use, telemetry navigation, and selective enrichment.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Security-agent evaluations commonly measure peak offensive capability under generous inference budgets, emphasizing vulnerability discovery, exploit development, penetration testing, and CTF completion. Such measurements are useful but incomplete: in operational security, every reasoning step, tool call, telemetry query, and enrichment request consumes budget. We evaluate language-model security agents through this cost-success lens on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges. Instead of reporting only best-case success, we compare models at fixed cost levels and decompose performance by inference spend and tool spend. Our results show distinct scalingregimes for red- and blue-team tasks. Offensive CTF performance improves with additional test-time compute, and scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive. Defensive SOC investigation does not scale in the same way: success depends more heavily on disciplined tool use, telemetry navigation, and selective enrichment than on raw reasoning budget alone. We argue that security-agent benchmarks should measure economic efficiency and operational fit alongside task success. Cost-aware, SOC-native evaluations provide a clearer picture of which models are practically useful today and where defensive agents still need to improve. We present an interactive website with our results https://evals.frontier.security.
Summary
Main Finding
Evaluating security agents only by peak success rates misses a key operational dimension: cost. When you measure performance at fixed per-sample budgets and decompose spend into inference tokens and priced external tool calls, offensive (red-team/CTF) tasks scale primarily with reasoning/compute budget, while defensive (SOC/investigation) tasks depend much more on disciplined tool use, telemetry navigation, and selective enrichment. Cost-aware, SOC-native benchmarks reveal which models are practically useful and where further investment should go.
Key Points
- Two distinct scaling regimes:
- Offensive (Cybench CTFs): success rises predictably with extra reasoning budget; scaled open-weight models can approach proprietary-model performance while remaining cost-competitive.
- Defensive (BOTS v1 SOC investigations): additional token/compute budget often yields little improvement; success correlates with efficient and selective tool use rather than raw compute.
- Cost decomposition matters: authors separate model-token cost (inference) from priced-tool spend (e.g., Brave Search, WhoisXMLAPI) and report operational metrics like $/solve (CTF) and $ per 1,000 BOTS points.
- Guardrails and refusals materially change observed outcomes: some models refuse many offensive samples (policy gating) and appear weaker, while others answer many BOTS questions without tool calls — a contamination risk.
- Decontamination controls (no-tools probes) show strong, model-dependent retrieval of public benchmark answers; scores must be interpreted with contamination checks.
- Practical cost examples from the paper:
- Cybench: GPT-5.5 solved 94.1% of challenges (at $2.10 cap) with ~$1.16 per solved-equivalent; DeepSeek v4 Flash rose from 76.1% at a $0.80 cap ($1.03/solve) to 86.4% at $2.10 ($1.45/solve).
- BOTS v1: Claude Opus 4.8 scored 9,666.7/10,300 points (93.9%) with model+tool spend $28.80 and ~$2.98 per 1,000 points; DeepSeek v4 Flash scored ~73% with much higher tool usage and ~$5.79 per 1,000 points.
- Tool-volume can be negatively correlated with score: some blue-team agents plateau despite heavy tool usage; disciplined, selective enrichment is more effective.
Data & Methods
- Benchmarks:
- Offensive: Cybench (hard variant) — 39 sandboxed CTF challenges with bash/Python allowed, flag submission judged by substring match, up to 3 submission attempts, three-epoch averaging.
- Defensive: Splunk BOTS v1 — 31 scored, sequential Splunk investigation questions (10,300 total points), official scoring including hint penalties; agents had Splunk discovery/search/event drill-down plus limited external enrichment (Brave Search, WhoisXMLAPI, VirusTotal, DNS/WHOIS).
- Agent & harness:
- ReAct-style agent with auto-compaction (90% context cap) implemented in the Inspect framework; same agent across suites with different tool sets.
- Budgets imposed via per-sample cost caps (agents unaware of the cost cap; Inspect aborts runs that exceed cap and marks failures).
- Refusal detection: heuristic on assistant text and provider metadata; first detected refusal aborts that sample-epoch.
- Cost accounting:
- Decomposed into model-token (inference) spend and priced tool spend.
- Pricing assumptions (examples reported): Brave Search $0.005/request; WhoisXMLAPI preview $0.0258/request or purchase $1.29/request; VirusTotal and many Splunk actions treated as $0 marginal cost in reported dollars.
- Reported metrics include run cost, $/solve (Cybench), model+tools $ and $ per 1k BOTS points, total tool calls.
- Contamination controls:
- No-tools single-epoch probes (no Splunk/web/bash/python) to test how much models can answer from memorized/training data and prerequisite Q&A context.
- Substantial, model-dependent retrieval observed, especially when prerequisite Q&A were provided.
Implications for AI Economics
- Procurement & ROI: pick models by operating point (task family + cost cap), not just peak accuracy. Use marginal-cost metrics like $/solve and $ per 1k SOC points to compare ROI across providers and model sizes.
- Where to invest:
- Offensive automation: extra reasoning budget (bigger context / more tokens) yields predictable returns — invest in compute/long-context pricing and caching to maximize capability per dollar.
- SOC automation: invest in tool integration, query optimization, and policy/agent workflows that enforce selective enrichment and efficient telemetry navigation. Improving tool-discipline or query costs often yields more practical ROI than scaling raw model compute.
- Pricing and product design:
- Priced external enrichments (WHOIS history purchases, search calls) can be a nontrivial operational cost and should be priced and budgeted separately from model inference.
- Caching (host-side or provider prompt caching) and budget-aware tooling can materially change per-sample economics.
- Safety vs availability tradeoffs: stronger guardrails reduce risky behavior but can cause operational refusals that appear as capability loss; buyers must weigh safety-policy settings against operational coverage (and measure this tradeoff in cost-aware evaluations).
- Benchmarking and policy:
- Benchmarks should report cost-aware, task-native operating points (cost caps, tool limits, cost-per-outcome) and include contamination/no-tools controls to avoid overestimating production utility.
- Standardized economic metrics (e.g., $/solve, $ per 1k investigation points, tool-calls-per-success) would make cross-provider comparisons more actionable for procurement and operational planning.
- Market signal: the paper suggests open-weight scaled models can be competitive on compute-cost grounds for offensive tasks, while defensive value depends on agent-tooling ergonomics — implying different vendor/product positioning and pricing strategies for red-team vs blue-team products.
Short actionable takeaway: measure and compare agents using per-sample budgeted operating points and explicit cost decomposition (tokens vs priced tools). For SOC automation, prioritize investments that improve tool discipline and telemetry navigation rather than only scaling model compute.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper provides direct empirical comparisons across tasks and cost regimes and decomposes performance by inference vs. tool spend, giving useful evidence about cost–performance tradeoffs. However, evidence is limited to selected benchmark suites, a particular set of model and tool integrations, and simulated/benchmarked SOC tasks rather than deployment in live operational environments, reducing external validity.
Methods Rigormedium — The study uses systematic, controlled evaluations (fixed budgets, multiple tasks, cost decomposition) and compares open-weight and proprietary models, which is methodologically sound for benchmarking. Rigor is reduced by likely limitations in model selection transparency, potential benchmark-specific tuning, lack of reported statistical uncertainty or robustness checks in the summary, and absence of real-world deployment validation.
SampleBenchmark datasets: offensive Cybench Capture-the-Flag (CTF) challenges and defensive Splunk BOTS v1 SOC investigation challenges; evaluated multiple scaled open-weight LLMs and proprietary systems; metrics include success rates at fixed inference compute budgets, counts of tool calls/telemetry queries/enrichment requests, and cost decompositions between inference and tool use. (Paper also provides an interactive results website.)
Themesadoption org_design
IdentificationControlled benchmarking comparison: models are evaluated on the same offensive (Cybench CTF) and defensive (Splunk BOTS v1) challenge suites at fixed inference and tool-cost budgets, and performance decomposed by inference spend vs. tool/spend; no formal causal identification strategy beyond controlled experimental comparisons.
GeneralizabilityBenchmarks (Cybench, Splunk BOTS v1) may not capture the full diversity and noise of real-world SOC environments and adversaries, Telemetry and enrichment in benchmarks may be synthetic or simplified compared with production telemetry, Model set evaluated may exclude newer or specialized models and architectures; results may change with model updates, Cost and latency findings depend on specific infrastructure, pricing, and tool integration details that vary across organizations, Focus on red/blue CTF-style and investigation tasks limits applicability to other security workflows (e.g., threat hunting at scale, incident response across heterogeneous stacks), Performance under adversarial or adaptive red-team behavior in live settings is not measured
Claims (11)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Security-agent evaluations commonly measure peak offensive capability under generous inference budgets, emphasizing vulnerability discovery, exploit development, penetration testing, and CTF completion. Other | null_result | evaluation_practice (peak offensive capability under generous inference budgets) |
Reading fidelity
high
Study strength
speculative
|
not reported
|
| Such measurements are useful but incomplete: in operational security, every reasoning step, tool call, telemetry query, and enrichment request consumes budget. Other | null_result | operational_cost (budget consumption per action) |
Reading fidelity
high
Study strength
speculative
|
not reported
|
| We evaluate language-model security agents through this cost-success lens on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges. Other | null_result | benchmark_performance_on_Cybench_and_Splunk_BOTS_v1 |
Reading fidelity
high
Study strength
high
|
not reported
|
| Instead of reporting only best-case success, we compare models at fixed cost levels and decompose performance by inference spend and tool spend. Task Completion Time | null_result | performance_by_cost (inference vs tool spend decomposition) |
Reading fidelity
high
Study strength
high
|
not reported
|
| Our results show distinct scaling regimes for red- and blue-team tasks. Task Completion Time | mixed | scaling_behavior_of_task_performance |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Offensive CTF performance improves with additional test-time compute. Task Completion Time | positive | CTF_task_success / performance as a function of test-time compute |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive. Adoption Rate | positive | relative_performance_and_cost_efficiency (open-weight vs proprietary) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Defensive SOC investigation does not scale in the same way: success depends more heavily on disciplined tool use, telemetry navigation, and selective enrichment than on raw reasoning budget alone. Organizational Efficiency | negative | SOC_investigation_success (sensitivity to reasoning budget vs tool use) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| We argue that security-agent benchmarks should measure economic efficiency and operational fit alongside task success. Governance And Regulation | null_result | benchmark_design_principles (economic efficiency and operational fit) |
Reading fidelity
high
Study strength
speculative
|
not reported
|
| Cost-aware, SOC-native evaluations provide a clearer picture of which models are practically useful today and where defensive agents still need to improve. Adoption Rate | positive | practical_utility_of_models_under_cost_constraints |
Reading fidelity
high
Study strength
medium
|
not reported
|
| We present an interactive website with our results https://evals.frontier.security. Other | null_result | availability_of_results_website |
Reading fidelity
high
Study strength
high
|
not reported
|