Code-writing AIs often rewrite more than they must when fixing bugs, increasing review burden; a simple preservation prompt cuts excess edits and slightly raises test-passing rates, and reinforcement learning can teach models to prefer minimal, faithful repairs.

Main Finding

Large language models commonly “over-edit” code when asked to repair localized bugs: they produce functionally correct patches that change substantially more of the original implementation than the minimal required edit. Over-editing is measurable, widespread across frontier LLMs, can be substantially reduced by a simple preservation prompt, and can be learned during post-training (RL outperforms naive supervised fine-tuning for out-of-domain edit-fidelity without degrading general coding ability).

Key Points

• Definition: over-editing = a passing repair that changes more code than the known minimal fix (measured as excess normalized token-level Levenshtein distance relative to the injected minimal repair).
• Benchmark and scope: focused on local, function-level repairs (brownfield setting). Open-ended refactors or feature additions are outside the scope.
• Prevalence: many state-of-the-art coding models achieve high Pass@1 while still making large, unnecessary edits (examples include large excess edits from GPT-5.x and other frontier models).
• Prompting: adding a short preservation instruction (“keep as much of the original code as possible”) reduced aggregate excess Levenshtein distance from 0.195 to 0.131, lowered added cognitive complexity by 26.6%, and increased Pass@1 by 2.3 percentage points.
• Reasoning and scale: enabling chain-of-thought/reasoning or increasing model size does not reliably reduce over-editing; effects are model-specific and non-monotonic.
• Post-training: supervised fine-tuning tends to overfit to corruption patterns seen in training; reinforcement learning produced the best out-of-domain trade-off — reported example: 0.782 out-of-domain Pass@1 with 0.050 excess Levenshtein distance and no loss in general coding ability.
• Nature of over-editing: often a granularity mismatch — models identify the bug but rewrite data flow, add defensive checks, or restructure code instead of performing the small local change.
• Human validation: metrics align well with developer judgments — excess Levenshtein matched human majority on reviewability in ~95% of cases; audit showed ~82% of high-excess passing repairs contained genuinely unnecessary edits.

Data & Methods

• Benchmark construction:
  • Base: 400 problems sampled from BigCodeBench (Python functions with executable tests).
  • Corruptions: inject 1–2 controlled AST-level corruptions per reference solution from a predefined corruption set; keep only examples where corrupted solutions fail the original tests. Total = 568 corruption applications.
  • Characteristics: functions average 10.4 executable lines; gold repairs are intentionally tiny (50.2% require 1 token edit, 91.8% ≤ 2 tokens, none > 2 lines).
  • Code and data: authors provide code (link in paper).
• Metrics:
  • Functional success: Pass@1 (task passes all tests).
  • Edit size: token-level normalized Levenshtein distance on function bodies. Define Dgold = d(G,C) (minimal repair size) and Dmodel = d(M,C). Excess = ELev = Dmodel − Dgold (positive → over-editing).
  • Structural overhead: added cognitive complexity (Δ cognitive complexity between model output M and gold G).
• Evaluations:
  • Compared many frontier models (reasoning and non-reasoning variants) under generic and explicit preservation prompts.
  • Human annotation: three experienced developers annotated blind pairs for reviewability and faithfulness to validate metrics; also a targeted audit of high-excess cases.
• Analyses:
  • Prompt ablation (generic vs. preservation).
  • Reasoning vs. non-reasoning variants within model families.
  • Size scaling study on Qwen2.5-Coder-Instruct family (0.5B → 32B).
  • Error-type breakdown showing over-editing concentrated for ambiguous, boundary, and list/conditional bugs.
  • Post-training experiments: supervised fine-tuning vs. RL for minimizing excess edits while preserving overall performance.

Implications for AI Economics

• New quality axis affects adoption economics: edit fidelity (minimal, reviewable fixes) is distinct from functional correctness and materially impacts developer review costs, time-to-merge, and operational risk in brownfield maintenance.
  • Over-edited patches increase reviewer cognitive load and diff churn, which raises human labor costs and slows integration — these are measurable hidden costs not captured by Pass@1 alone.
• Procurement and evaluation: organizations buying or deploying coding models should include edit-fidelity metrics (excess edit distance, added cognitive complexity) in evaluation and SLAs, not just Pass@k. Procurement decisions that ignore fidelity risk underestimating the total cost of model-assisted maintenance.
• Incentives for providers: model providers can gain value by optimizing for edit fidelity (via prompting, RL fine-tuning, or offering “preservation-aware” modes). This can reduce downstream human review costs and improve product-market fit in maintenance-heavy enterprise settings.
• Cost-benefit of post-training: simple supervised fine-tuning may overfit to specific corruption patterns and not generalize, potentially wasting investment; RL-style fine-tuning appears to yield better out-of-domain fidelity without degrading coding performance. Firms should weigh the additional engineering/training costs of RL against expected savings in review time and reduced risk.
• Product design and workflow changes: integrating preservation prompts or “minimal-edit” modes into developer tooling (patch suggestions, PR bots) is a low-cost intervention with measurable gains and could be adopted quickly to lower friction. Vendors can surface a fidelity toggle (conservative vs. robust fix) to match user needs.
• Risk management: unseen over-editing (passing tests but changing large swaths) can introduce latent regressions in production—risk models and expected cost of faults should account for over-editing-induced regression probability. Monitoring and post-hoc audits should track edit-fidelity as a risk signal.
• Benchmarking & marketplace effects: standardized fidelity benchmarks will reshape competition — models that preserve code succinctly will command premium in maintenance contexts. This also signals an economic opportunity for niche models or fine-tuned variants targeted at brownfield repair tasks.

Limits noted by authors (relevant for economic interpretation): - Benchmark focuses on local repairs; intentional refactors or feature changes that legitimately edit broadly are out of scope. - Minimal repair objective assumes the injected corruption’s reversal is the best/desired fix; real-world fixes can validly differ (but the controlled setup isolates over-editing behavior).

Overall, the paper argues that edit fidelity is measurable, economically relevant, and actionable: simple prompts and appropriate post-training (especially RL) can reduce unnecessary edits and their downstream costs.