AI agents turn a developer's single rename into a safe, project-wide refactoring plan, cutting tedious manual propagation and false positives; demonstrated via a 609K-commit analysis, a 205-developer survey, and an implemented multi-agent refactoring tool.

Main Finding

CoRenameAgent — a novel multi-agent, human-in-the-loop system integrated with the IDE — substantially improves automated coordinated rename refactorings. Compared to prior tooling and vanilla LLM approaches, it delivers much higher precision and recall (2.3–3.1× F1 improvements on benchmarks), eliminates compilation/semantic errors by leveraging IDE refactoring APIs, reduces developer time from ~1 hour to ~5 minutes on average, and has demonstrated real-world acceptances (5 of 10 submitted PRs merged).

Key Points

• Problem targeted: coordinated renaming — a frequent, time-consuming, and error-prone refactoring that often requires consistent renames across many identifiers/files (57% of rename commits, average 5 renames across 4 files; some historical examples involve hundreds).
• Architecture: multi-agent workflow with three cooperating agents plus episodic memory:
  • Scope Inference Agent: observes a developer’s seed rename and infers an explicit natural-language Declared Scope; elicits and records developer feedback to refine scope.
  • Planned Execution Agent: maps the Declared Scope to concrete program elements in a file, asks for developer approval, and applies approved renames via the IDE’s trusted refactoring APIs (ensures safe execution).
  • Replication Agent: finds other files to propagate scoped renames using semantic search + program slicing (controls project-scale propagation).
• Human-in-the-loop: developer supervises and approves suggestions; ablation shows removing supervision drops precision ~4× — developer oversight is essential for correctness and containment.
• Safety & robustness: by invoking IDE refactoring engines rather than emitting raw edits, CoRenameAgent avoids hallucination-driven compile/semantic errors reported with vanilla LLMs (vanilla LLMs produced 45–92% of projects with compile errors; avg ~5 compile errors per corename set).
• Implementation: IntelliJ IDEA plugin built with LangGraph and OpenAI o4-mini; episodic memory to learn developer preferences and support inter-agent coordination.
• Empirical performance:
  • RenasBench (potentially contaminated): F1 = 54.6% (2.3× over best baseline).
  • New uncontaminated Co-renameBench (post-2025 dataset): F1 = 48.5% (3.1× improvement).
  • Usability: average 5 minutes per coordinated rename vs ~1 hour for prior SOTA; automates ~23 actions for ~$0.30 cost per operation.
• Real-world validation: generated and submitted 10 PRs; 5 accepted/merged, 2 rejected for socio-technical reasons, 3 pending — evidence of practical adoption potential.
• Limitations noted by authors: potential dataset contamination in prior benchmarks, evaluation focused on Java, dependence on LLMs and IDE integration, socio-technical review issues can block some automated PRs.

Data & Methods

• Repository mining and formative study:
  • Analyzed 609K commits across 100 open-source projects to characterize frequency, scope, and cost of coordinated renames.
  • Surveyed 205 developers about their refactoring practices and pain points.
• Tool implementation:
  • CoRenameAgent implemented as an IntelliJ plugin; uses LangGraph and OpenAI o4-mini; integrates with IDE refactoring APIs.
• Evaluation methods (multi-methodology):
  • Comparative studies against state-of-the-art research tools (e.g., RENAS) and vanilla LLMs / code agents.
  • Benchmarks:
    • RenasBench: 1349 renames across 161 co-rename sets (may include training data contamination).
    • Co-renameBench (new, uncontaminated): 1573 recent renames (post-2025) curated by the authors to avoid contamination.
  • Ablation study: measured effect of removing human-in-the-loop on precision.
  • Replication of real-world refactorings and user/case studies.
  • Real-world deployment: generated PRs to active repositories to test acceptability.
• Metrics reported:
  • F1-score for identifying and effecting co-renames; precision/recall tradeoffs; compilation/semantic-error rates; time-to-complete; cost-per-run; acceptance of PRs.

Implications for AI Economics

• Productivity and direct ROI:
  • Time savings: typical coordinated rename task reduced from ~60 minutes to ~5 minutes. At common developer rates (example: $50/hr), a single automated run saves ~55 minutes ≈ $45 value, while tool cost is trivial (authors report ~$0.30 per 23 automated actions), implying large per-task ROI.
  • Scaling effects: frequent repetitive refactoring tasks mean cumulative time savings across teams; high reuse potential in large codebases and active projects.
• Labor composition and complementarities:
  • Task-shift from manual execution to supervision: developers spend less time on mechanical edits and more on intent-definition, review, and policy decisions. This raises demand for oversight and higher-level judgment skills rather than purely mechanical coding.
  • Augmentation rather than replacement: the human-in-the-loop design demonstrates a likely path for tools to augment developer capacity while preserving control — decreasing adoption friction due to ownership/control concerns.
• Risk reduction and value of trusted automation:
  • Integrating with IDE refactoring APIs mitigates hallucination risks of raw LLM edits, reducing costs of bug-fix cycles and review overhead — significant economic value in avoiding costly post-release errors.
• Market & platform implications:
  • Opportunity for IDE vendors and toolchains: agentic integrations with IDEs (safe API usage + scope-inference UX) can become differentiators for commercial IDEs and DevTools.
  • Service bundling/pricing: low per-run compute cost suggests business models could charge modest subscription fees or embed in platform licensing; firms could offer paid enterprise-grade agents with policy controls and audit logs.
• Adoption frictions & externalities:
  • Socio-technical hurdles: some PRs rejected for non-technical reasons, implying organizational/coordination costs remain; governance, naming conventions, and review norms matter.
  • Data/benchmark quality matters for evaluation and procurement: authors created an uncontaminated benchmark — buyers and regulators should prefer uncontaminated, recent-evaluation datasets to avoid overestimating capabilities.
• Broader economic questions for further study:
  • Aggregate productivity gains across firms and markets, and how these translate to labor demand (complementary upskilling vs substitution).
  • Pricing and diffusion dynamics for agentic IDE plugins across startups vs large enterprises.
  • Liability, auditability, and contract implications when agents make project-wide changes (who is accountable for semantic errors missed by human reviewers).
  • Long-term effects on tooling labor (e.g., code review patterns, maintenance staffing needs).

Limitations to economic interpretation: the evaluation is concentrated on Java open-source projects and the reported cost/time figures derive from the authors’ experimental setup; firm-level ROI will vary by developer wages, task frequency, integration costs, and organizational adoption barriers.