An open-source harness shows structural retrieval dramatically improves AI refactoring: AST-aware chunking increases multi-file refactoring pass rates by roughly 25–30 percentage points, and retrieval-augmented single-agent pipelines reach up to 86% success on a 100-task Python benchmark while costing no more per successful refactor than the baseline.

Main Finding

RefactorPlatform is an open-source, auditable harness for evaluating repository-scale refactoring agents under controlled conditions. Using 100 multi-file RefactorBench tasks, the platform shows that (1) AST-aware structural retrieval (AST chunking + vector/BM25 search) substantially improves refactoring pass rates versus both a retrieval-free baseline and naive token-window chunking, (2) multi-agent delegation underperforms a lean, retrieval-augmented single-agent pipeline, and (3) retrieval’s token overhead is largely offset by its accuracy gains, leaving cost per successful refactor essentially unchanged.

Key Points

• Platform design
  • Isolated per-task workspaces, live terminal streaming, per-turn telemetry (tokens, diffs, transcripts), and exportable logs for auditability.
  • Explicitly controllable axes: model backbone, execution regime (S1 baseline, S2 retrieval-augmented, S3 sub-agent delegation), prompt specificity (Descriptive / Base / Lazy), and benchmark selection.
  • AST-based verification pipeline: Python via AST unit tests; Java via extraction → compile → test → RefactoringMiner checks.
• Retrieval and chunking
  • AST-aware chunking preserves syntactic/semantic units; beats naive token-window chunking by roughly 25–30 percentage points across prompt modes (Descriptive/Base/Lazy).
  • Retrieval engine: CocoIndex chunking, nomic-embed-code embeddings (GGUF Q4_K_M), hybrid cosine+BM25 retrieval fused with Reciprocal Rank Fusion (RRF).
• Execution regimes & prompt sensitivity
  • S1 (baseline terminal assistant): sensitive to prompt specificity. Switching Descriptive → Base → Lazy produced large drops (baseline example: -9 pp then -16 pp).
  • S2 (retrieval-augmented single agent): substantial improvement over S1; structural retrieval is primary driver.
  • S3 (agent-native sub-agent delegation): worse than S2 on these tasks. In the 100-task run (Descriptive prompts) S2 = 86% vs S3 = 66% overall (70.4% on tasks where delegation actually triggered). There were 20 tasks that passed under S2 but failed under S3; no tasks exhibited the reverse.
• LSP diagnostics
  • Local LSP enforcement provided only small, inconsistent gains (~+2–5 pp), suggesting linting feedback does not substitute for broad structural context.
• Cost-efficiency
  • Retrieval raises $/attempt (indexing/query overhead) but reduces failed attempts; cost per successful refactor ($/pass) stayed effectively constant (differences ≤ $0.01 in their examples).
  • Example best configuration: qwen3.6-flash S2 reached 86% pass rate at ~$0.13 per successful refactor.
• Limitations
  • Experiments used mid-tier models (qwen3.6-flash, minimax-m3, kimi-k2.6, deepseek-v4-pro) for cost reasons — frontier models not evaluated.
  • Primary focus on Python (RefactorBench); Java (SWE-Refactor) explored only preliminarily.
  • Results are single-run campaigns (no uncertainty estimates reported).

Data & Methods

• Benchmark and tasks
  • Primary dataset: RefactorBench (100 multi-file Python navigation/refactoring tasks spanning 9 repositories, 2–31 files changed per task).
  • Supplemental integration with SWE-Refactor (Java) supported but only preliminary results included.
• Verification
  • Python: AST-based unit tests; tasks count as pass only if all associated AST tests pass (no partial credit).
  • Java: extraction → compile → run tests → RefactoringMiner structural check; CodeBLEU recorded as a diagnostic.
• Experiment setup
  • Execution regimes:
  • S1: single terminal assistant with local filesystem and shell tools.
  • S2: S1 + task-scoped structural retrieval (MCP-based APIs: search_codebase, list_indexed_files, read_indexed_file).
  • S3: agent-native sub-agent delegation (no imposed topology).
  • Retrieval pipeline: CocoIndex chunking → nomic-embed-code embeddings → hybrid cosine + BM25 ranking → RRF fusion.
  • Prompt modes: Descriptive (What+Where+How), Base (What+Where), Lazy (What-only).
• Metrics
  • Task Pass Rate (PRtask): fraction of tasks where all AST tests pass.
  • Cost-efficiency (CE): average token dollar cost per successful task = (sum costs) / (number of successful tasks).
• Empirical runs
  • Two-stage demonstration: full ablation on qwen3.6-flash (Table 1) and cross-model S1 vs S2 comparisons across four models (Table 2).
  • Representative numeric results (Descriptive prompts):
  • qwen3.6-flash: S1 73% → S2 86%
  • minimax-m3: S1 75% → S2 81%
  • kimi-k2.6: S1 71% → S2 78%
  • deepseek-v4-pro: S1 77% → S2 89%
  • Ablation highlights (qwen3.6-flash):
  • S1 Descriptive with LSP: 73% (no LSP: 71%).
  • S2 Descriptive AST chunking: 86%; naive chunking: 57%.
  • S3 Descriptive (Copilot sub-agents): 66%.
  • Cost table example (Descriptive mode):
  • qwen3.6-flash S1: 73% pass, $0.096/task, $0.13/pass
  • qwen3.6-flash S2: 86% pass, $0.113/task, $0.13/pass
  • deepseek-v4-pro S1: 77% pass, $1.072/task, $1.39/pass
  • deepseek-v4-pro S2: 89% pass, $1.234/task, $1.39/pass

Implications for AI Economics

• Tooling investment vs model scaling
  • Structural tooling (AST-aware chunking + targeted retrieval) yields material performance improvements that can substitute for—at least partially—simply using a larger or more expensive model. For procurement and ROI analysis, investing in indexing/retrieval infrastructure can be a high-leverage cost-effective strategy.
• Cost per successful outcome matters more than per-call cost
  • Retrieval increases per-attempt token costs but reduces failures; resulting cost-per-success remained stable in experiments. Economic evaluations should therefore use cost-per-success (or cost-per-deployed-change) rather than per-inference cost when comparing agent designs.
• Multi-agent orchestration carries hidden economic risks
  • Multi-agent delegation introduced communication and orchestration overheads that reduced pass rates in this setting. Beyond direct token costs, organizational and engineering costs (complex orchestration, monitoring, debugging handoffs) can erode expected benefits. Enterprises should be cautious: multi-agent architectures may increase operational complexity without guaranteed gains.
• Auditable harnesses reduce deployment risk and enable repeatable ROI measurement
  • RefactorPlatform’s per-task telemetry, AST verification gates, and exportable logs enable credible, repeatable measurement of both efficacy and cost. This reduces model/devops risk and enables better contracting and budgeting decisions (e.g., pay-for-performance or benchmarking claims).
• Procurement and deployment recommendations
  • Prioritize investment in structural code understanding (AST-aware chunking, codebase-oriented retrieval) and robust verification pipelines before scaling to multi-agent stacks or heavier compute models.
  • When evaluating vendors or agent architectures, request cost-per-success and audited pass rates on representative, repository-scale refactoring tasks (not just per-call latency or token cost).
• Caveats for economic generalization
  • Results are from mid-tier models and Python tasks; frontier models or strongly-typed compiled languages may change the cost/benefit calculus. Decision-makers should run domain-specific, repeated trials (enabled by platforms like RefactorPlatform) before large-scale procurement.

If you want, I can (a) produce a compact table of the key numeric results for quick inclusion in a report, or (b) draft suggested procurement evaluation criteria for enterprise teams based on these findings.