Giving coding agents interactive debuggers boosts bug-fix success by over 20% on standard benchmarks; better tool design can let cheaper models match pricier ones, narrowing the gap between model size and practical performance.

Main Finding

Integrating interactive debugging into coding agents via a specialized Debug Subagent (Debug2Fix) substantially improves repository-level bug-fixing. The approach yields >20% relative improvement over baseline agents on some models and benchmarks, and—crucially—better tool and architecture design can let cheaper/weaker models match or exceed stronger models’ performance.

Key Points

• Problem: Current coding agents rely on static analysis or iterative print-debugging (trial-and-error). They lack direct access to precise runtime state, which limits effectiveness and efficiency on harder bugs.
• Solution (Debug2Fix): Introduce a Debug Subagent that encapsulates interactive debugger usage behind a simple, high-level question-answer interface for the main agent.
  • The main agent invokes the Debug Subagent with a Runtime Question plus metadata (test(s), path, lines, variable hints).
  • The subagent runs an internal debugging loop (set breakpoints, step, inspect) and returns a structured factual answer with evidence.
• Tool design (LLM-friendly): concrete tools exposed to the subagent include
  • debug_start_session (atomic: build, launch test, attach, set initial breakpoints),
  • debug_control (continue / step over / step in / step out),
  • debug_inspect (evaluate expressions, inspect locals/objects, view call stacks),
  • debug_breakpoint (add/remove/list breakpoints),
  • plus file navigation/grep/read tools.
  • Output format enforces a block with Question, Answer, Evidence, Location.
• Important engineering decisions:
  • Expose only the Debug Subagent to the main agent (not raw debugger tools), because direct exposure led to under‑utilization.
  • Disable file-editing tools until the main agent has invoked the Debug Subagent at least once—encourages root-cause analysis before edits.
  • Make debug_start_session atomic to avoid race conditions/timeouts seen when exposing lower-level steps.
• Empirical claims:
  • Implemented with Java (JDB) and Python (pdb) debuggers.
  • Evaluated on repository-level bug-fix benchmarks (GitBug-Java, SWE-Bench-Live).
  • Observed >20% relative improvement over a vanilla agent baseline for certain model+setup combinations.
  • Using Debug2Fix enabled weaker/cheaper models (e.g., GPT-5, Claude Haiku 4.5) to match or exceed performance of stronger models (e.g., Claude Sonnet 4.5).
• Ablations: show both the subagent architecture and the debugger integration are essential; exposing tools directly to the main agent or omitting atomic tooling lowers performance / utilization.

Data & Methods

• Datasets / Benchmarks:
  • GitBug-Java (Java repository-level bug-fix benchmark).
  • SWE-Bench-Live (real-world software engineering bug-fix tasks; includes Python examples).
  • Motivating example: a deep repo Python bug (faker library) where _safe_now() fell back to epoch (1970-01-01), causing a negative timestamp; baseline agent did heavy print-debugging and proposed an incorrect change while Debug2Fix found the root cause quickly via breakpoints and variable inspection.
• Models tested (representative mentions):
  • GPT-5, Claude Haiku 4.5, Claude Sonnet 4.5 (used to show model/tool tradeoffs).
• System & Architecture:
  • Main agent: general-purpose repo-level coding agent with tools (bash, read, grep, etc.) plus Debug Subagent as a tool.
  • Debug Subagent: separate LLM instance with its own system prompt, context, and restricted toolset. It orchestrates the debugger tools and returns a concise evidence-backed answer.
  • Debug start session builds the project, launches tests, attaches debugger and sets breakpoints (defaults to test entry).
  • Debug Subagent performs iterative inspect/control/breakpoint actions until it answers the runtime question or exhausts turns.
• Evaluation methodology:
  • Compare baseline agent (no debugger subagent) versus Debug2Fix on the benchmarks.
  • Measure repair/fix success rates (and report relative gains; >20% in some configurations).
  • Ablation studies: (a) expose debugger tools directly to the main agent vs via subagent, (b) disable/enable file-editing gating, (c) remove atomic start_session—assessing effects on utilization and success.
• Qualitative analysis: traced trajectories show Debug2Fix mirrors expert developer debugger workflows (set breakpoint at test, step, inspect variables, find root cause) and avoids repeated print-debug cycles.

Implications for AI Economics

• Tool design as a lever for cost-effective performance:
  • Investing in richer tool integration and agent architecture (subagents + debuggers) can substitute for upgrading to more expensive LLMs. The paper shows weaker/cheaper models can match stronger models when paired with better tooling—this changes the marginal value of model size vs tool engineering.
  • For firms evaluating product upgrades, ROI calculations should include engineering cost to integrate interactive debugging rather than only model API costs.
• Pricing and product differentiation:
  • Coding-agent vendors can create tiered offerings where advanced runtime tooling (interactive debugger subagent) is a premium feature. Because tooling can boost cheaper models, vendors might monetize tooling rather than raw model access.
• Productivity and labor effects:
  • Debug2Fix-style agents should reduce time-to-fix for complex bugs, increasing developer throughput and productivity. That has downstream effects on project timelines, staffing needs, and developer value-add metrics.
  • The technology complements developer skills (debugger-driven reasoning); demand may shift toward developers who can design/deploy tool-augmented workflows and integrate agents into CI/dev environments.
• Cost structure and compute:
  • Fewer trial-and-error test runs and faster convergence to root causes reduce compute resources spent per bug-fix (fewer runs of tests / LLM calls), lowering operational costs for hosted agent services.
• Evaluation and procurement:
  • Procurement decisions should weight agent-tool co-design and integration costs. Benchmarks that only measure static LLM performance will understate the value of integrated toolchains.
• Risks & maintenance costs:
  • Integrating debuggers into production agents introduces operational complexity: build-system coupling, portability across languages/environments, security/sandboxing of runtime debugging, and ongoing maintenance as repos and toolchains change. These costs must be accounted for in economic assessments.
• Market impact on benchmarking & competition:
  • Benchmarks and vendor comparisons should include tooling and orchestration quality. Firms emphasizing agent orchestration, subagent design, and engineering robustness may outcompete those that only optimize model choice.

If you’d like, I can: - Extract the paper’s reported numeric results and table(s) exactly (if you can provide the results section/tables). - Produce a short slide-style summary for stakeholders comparing model-upgrade vs tool-investment ROI scenarios.