Large language models are dissolving many Capture-the-Flag challenges and threatening CTFs' role as tests of human cyber skill; the authors map which challenge subcategories still resist AI and propose tiered divisions, LLM-resistant design, investigative telemetry and a community code to preserve fair play.

Main Finding

Large language models (LLMs) and agentic tool-chains have materially changed what Capture the Flag (CTF) competitions measure. Easy and intermediate tasks in cryptography, many web-exploitation patterns, and a large share of binary-exploitation “jeopardy” problems are now routinely solvable by agents. The core community dispute (“is AI cheating?”) is misplaced: the decisive question is competition purpose. The authors map the human–machine capability boundary, document community reactions, and propose a four-component safeguard framework (tiered divisions; LLM-resistant challenge design; investigative telemetry; community code of conduct) plus a decision tool that links safeguards to a competition’s declared purpose.

Key Points

• Disruption timeline: five overlapping phases from ChatGPT (Nov 2022) → agentic systems → long-context specialist models (late 2025–early 2026) that can ingest whole challenges and run tool chains.
• Category-specific capability boundary:
  • Cryptography: “settled.” Many classical and intermediate exploits (RSA attacks, padding-oracle, classical weaknesses) are automatable. Hard math-first-principles problems (e.g., advanced lattice, novel ECC isogenies) still resist LLMs.
  • Web exploitation: “in transition.” Common classes (SQLi, SSRF, JWT misconfigs, SSTI, prototype pollution) are increasingly solvable; remaining frontier involves novel business-logic flaws, desync/request-smuggling, race conditions, heavy obfuscation, or physical/interactive components.
  • Binary exploitation (pwn): held out longest but has eroded fastest once agents could drive debuggers and tool-chains; only the hardest kernel/complex cases currently resist.
• Technical drivers:
  • Agentic systems that can call debuggers, browsers, and local tools closed the representational and tooling gaps.
  • Large context windows (~100k–1M tokens) let models ingest entire challenges and long tool-output traces.
  • Economic factors: competing at the top now often involves paying for model credits — an emerging tooling budget/arms race.
• Community dynamics:
  • Debate centers on whether competitions are outcome-focused (maximize score; AI is a tool/skill) or process-focused (develop human tacit knowledge; AI undermines purpose). Without explicit purpose declarations, unregulated competitions default to score-maximization, disadvantaging human-skill development.
• Safeguard framework (four components):
• Tiered competition divisions (e.g., “human-only” tracks, AI-allowed tracks).
• LLM-resistant challenge design (novelty, physicality, interaction, bespoke protocols).
• Telemetry used investigatively (server-side logs, submission patterns) to detect anomalous AI-style solves — noting methodology is still immature and privacy/implementability concerns exist.
• A community code of conduct and transparency norms (declare purpose, tooling policies).
• Decision tool: a practical mapping that ties which safeguards to apply based on the declared purpose of the event (education, recruitment, entertainment, talent ID).
• Broader claim: findings extend beyond CTFs to any cybersecurity setting where an observed artifact is used as evidence of underlying capability (certification, hiring pipelines, assessments).

Data & Methods

• Mixed-methods design with four strands:
• Synthesis of published benchmarks against public CTF datasets (Cybench, NYU CTF Bench, InterCode) plus a recent government evaluation showing agentic systems raise solve rates substantially.
• Observational case studies of live competitions across cryptography, web, and binary exploitation during the 2025–26 season, including DiceCTF and Insomni’hack, and authors’ own competitive play.
• Structured observation of public community channels (subreddits, Discords, CTFtime threads) for discourse context.
• Semi-structured interviews (three 30–60 min) with experienced players and organisers (pseudonymised); ethical approval obtained and anonymity protections applied.
• Limitations noted by authors:
  • Small, self-selected interview sample (illustrative rather than representative).
  • Fast-moving capability landscape — the mapped boundary is a snapshot; direction-of-travel matters.
  • Telemetry and detection methodologies are still immature and raise practical/privacy concerns.

Implications for AI Economics

• Signalling and credentialing:
  • CTFs and similar practical tasks are an important credential/talent pipeline. When LLMs produce demonstrable outputs detached from human competence, the signal value of flags/achievements is diluted. Economists should expect credential inflation or a reconfiguration of which signals employers trust (e.g., interviews, supervised tasks, in-person assessments).
• Labor-market impacts:
  • Short-term: easier entry for novices (wider on-ramp), which can increase applicant pools but complicate filtering by true skill.
  • Medium-term: firms may shift hiring practices to emphasize supervised/interaction-based evaluation or to price in tooling budgets as hiring offsets.
• Competitive dynamics and arms races:
  • A tooling-budget arms race emerges — teams paying for inference/agent credits can capture leaderboards. This creates rent-seeking and unequal access; richer teams/organisations can outcompete skilled but resource-constrained individuals.
  • Markets for specialised models, agentic orchestration tools, and long-context security models will grow, altering R&D and procurement incentives.
• Assessment and mechanism design:
  • Organizations that rely on CTF-style evidence (universities, recruiters, certifiers) need mechanism redesign: tiered tracks, supervised assessments, or new observables that better reflect human competence.
  • There is market value in “LLM-resistant” assessment design and in third-party verification services that assert human-in-the-loop integrity.
• Policy and standards:
  • Standardization bodies and institutions may need to set norms (e.g., declared purpose, acceptable tooling) to avoid endogenous rules that favor score-maximization and privilege resource-rich actors.
  • Regulatory interest may arise around fairness, procurement of AI for competitive advantage, and privacy/ethics of telemetry-based detection.
• Research and measurement agenda for economists:
  • Quantify causal effects of agentic LLM availability on hiring outcomes, wage premia, and recruitment spending.
  • Measure credential devaluation: how employer trust in CTF-derived signals changes over time and across sectors.
  • Model the equilibrium between human-skill investment and reliance on tool budgets — predict specialization, divisioning of markets (human-certified vs agent-assisted tracks).
  • Study distributional impacts: who gains/loses (students, independent competitors, firms with bigger budgets).
  • Evaluate cost-benefit trade-offs of safeguards (implementation costs, enforcement, false positives in telemetry) and potential unintended consequences (privacy harms, gaming of detection signals).

Suggested short policy/economic responses for actors: - Recruiters: adopt supervised or interactive technical screens; treat CTF scores as one signal among many. - Universities/organisers: declare competition purpose; use tiered tracks and design challenges that reward human-only skills if pedagogy is the goal. - Firms/market designers: expect and budget for tooling expenditures if operating in AI-assisted competition markets; consider developing verification services or certification standards. - Researchers: prioritize causal studies on labor-market impacts and on the effectiveness and costs of proposed safeguards.