Pre-execution planning can hurt code-repair agents when planners lack task information or face tight token budgets, but giving planners the issue or more inference capacity sharply raises resolution rates — turning a harmful overhead into a 30-percentage-point advantage.

Main Finding

Extra inference capacity (larger token budgets) improves task success only insofar as that capacity relaxes active bottlenecks inside a workflow — and that effect depends on how inference and task information are organized across stages. A separate pre‑execution planning stage can (a) hurt performance when it consumes tokens without access to the task issue, (b) be redeemed by giving the planner issue visibility, and (c) outperform one‑shot execution once capacity is no longer scarce.

Key Points

• Experimental setting

  • 40 externally verified software‑engineering tasks (SWE‑bench Verified; 35 Django tasks + 5 others).
  • External evaluation via a frozen Docker harness; success = passing frozen tests.
  • Two model backends, three replicates per task → 240 assignments per workflow × cell.
  • Three workflow contracts:
  • TG (Direct execution): no separate planner; execution sees issue and repository and performs edits/runs.
  • TGP (Issue‑hidden planning): separate pre‑execution planner that inspects repository with the issue hidden (diagnostic tools in resource panels; read‑only in strict campaign). Execution later receives issue + plan.
  • TGPA (Task‑informed planning): same strict planning contract but planner also sees the issue.
  • Logical token ceilings B ∈ {12,000, 24,000} create a shared budget between planning and execution; all ledger use is metered.

• Main empirical estimates (externally verified success)

  • Direct execution (TG): identical success 59.6% (143/240) at both 12k and 24k ceilings.
  • Issue‑hidden planning (TGP): success rises from 36.2% (87/240) at 12k to 51.2% (123/240) at 24k.
  • The change in the planning workflow’s effect when ceiling doubles:
  • ΔB = τP(24k) − τP(12k) ≈ +0.150 (i.e., a 15.0 percentage‑point narrowing of the disadvantage relative to TG).
  • 95% task‑cluster bootstrap interval: [4.2, 25.8] pp; p = .008.
  • Strict 12k information experiment (read‑only planner):
  • TGP (12k): 29.2% (70/240).
  • TGPA (12k): 45.6% (109/239) → issue visibility improves success by ≈ +16 pp (statistically significant after Holm correction).
  • TG (12k): 55.4% (133/240). TGPA vs TG at 12k: ≈ −10 pp (CI roughly −20 to ≈0).
  • At 24k, TGPA attains 82.5% success — about +29.6 pp above direct execution.

• Process evidence and mechanism

  • At 12k, an issue‑hidden planner often consumes a large share of tokens on plans irrelevant to the true issue: ~46.2% of runs hit the planner budget (binding).
  • Raising the ceiling to 24k almost eliminates planner binding (binding rate falls to 0.8%); downstream execution token use rises (mean execution use from ~5,207 → 6,531 logical tokens).
  • Direct execution typically leaves substantial unused capacity under both ceilings (no displacement cost).
  • When the planner has issue visibility, planning reallocates effort toward productive coordination and yields substantial performance gains, especially when capacity is ample.

Data & Methods

• Tasks and verification
  • 40 SWE‑bench Verified repository tasks, each with a repo, issue, and executable tests run in isolated Docker harnesses.
  • External verifier is frozen (no tailoring to treatments); success is binary (tests pass).
• Experimental design
  • Randomized run order within panels.
  • Resource panels: compare TG vs TGP at B=12k and B=24k. Each panel includes both workflows across the 40 tasks × 2 backends × 3 replicates per workflow (n=480 outcomes per panel).
  • Strict information panel: read‑only planner and comparison of all three workflows at 12k (and also at 24k for TGPA).
• Measurement and metrics
  • Outcome: externally verified success (pass/fail).
  • Token accounting: shared logical ledger; planner and execution token use recorded separately; measured binding rates (fraction of runs that hit ceiling).
  • Statistical inference: task‑cluster bootstrap intervals (95%) and multiple‑comparison correction where noted.
• Robustness
  • Results robust under missing endpoint imputation reported in paper; supplementary analyses in online appendix (process and backend checks).

Implications for AI Economics

• Scale vs organization
  • Increasing aggregate inference capacity (larger token budgets) is not a universal win: the productive returns depend on where extra capacity is absorbed and whether it eases a real bottleneck inside the workflow.
  • Organizational design (who plans, what information they see, and how stages share budget) determines how much of that capacity is transformed into higher output.
• Information allocation matters
  • Planners deprived of task‑defining information can produce coherent but misdirecting plans whose coordination value is negative once you account for the opportunity cost of displaced execution.
  • Providing planners with the relevant issue can flip planning from harmful to beneficial — and at high capacity can make a planning workflow substantially better than direct execution.
• Opportunity cost of intermediate stages
  • Any intermediate computational stage (planning, search, verification) consumes scarce inference that could otherwise be used for execution, repository inspection, or recovery; that opportunity cost must be included when valuing metareasoning or multi‑stage agents.
• Design and policy implications
  • System architects should jointly optimize token budgets and information flows: e.g., read‑only/security constraints that hide task‑defining information from planners may significantly reduce productivity unless budgets are increased or planners are given alternative signals.
  • Benchmarking and model evaluation should account for workflow organization and resource metering (not just aggregate token counts), because differences in orchestration change realized returns to compute.
• Limits and generalizability
  • Results are from repository code‑repair tasks with specific verifiers and fixed planner/executor contracts; other domains, interleaved planning–execution policies, different toolsets, or adaptive allocation rules could change magnitudes.
  • External verification tests are an operational success metric but do not imply full software correctness or robustness to unseen distributional issues.

Short takeaway: tokens alone don’t determine performance — how you allocate them across stages and what information each stage gets are central economic margins that determine whether additional compute improves real task outcomes.