A proactive self-distillation framework, SkillForge, raises LLM-based code-repair success rates by roughly 4–6 percentage points on standard benchmarks by synthesizing repository-specific bugs from tests and distilling reusable diagnostic and intervention skills.

Main Finding

SkillForge is a proactive self-distillation framework that synthesizes executable, test-grounded “project-specific” issues from a repository and distills the resulting agent trajectories into a dual-level skill repository. Feeding these skills back to the agent (macro-level diagnostic priors + micro-level just-in-time intervention) substantially improves automated issue-resolution performance on standard benchmarks, outperforming both history-driven and online per-issue adaptation baselines.

Key quantitative results reported: - On SWE-bench Verified, SkillForge yields Pass@1 of 72.2% (DeepSeek-V3.2) and 60.6% (GPT-5-mini), improving over a Mini-SWE-Agent baseline by +5.8 and +5.6 percentage points, respectively. - On SWE-bench Pro, SkillForge improves over Mini-SWE-Agent by +5.8 and +4.1 percentage points (same two models). - Outperforms evaluated history-driven and online project-specific adaptation baselines.

Key Points

• Cold-start problem: LLM-based SWE agents lack project-specific conventions and repeatedly rediscover the same patterns when deployed on a new repository.
• Proactive synthesis: SkillForge uses coverage-guided traces and a masked reimplementation prompting strategy to create realistic, test-driven synthetic issues (pass→fail instances) without relying on historical issue reports.
• Dual-level skill repository:
  • Mext (global diagnostic skills): entity-linked records capturing an entity’s purpose, playbook (repository-specific reasoning heuristics), and related_apis (co-involved APIs from trajectories).
  • Mint (local intervention skills): entity-specific repair guidance and pitfall-avoidance cues distilled from both successful and failed resolution trajectories.
• Retrieval and injection:
  • Macro initialization: BM25 retrieval of top-k Mext records appended to the agent’s initial prompt as project priors.
  • Micro JIT injection: Mint records are appended to the agent’s context dynamically when it accesses specific files/entities (based on parsed shell commands).
• Distillation pipeline: normalize agent trajectories, align accessed coordinates to AST-derived entities, contrast successful/failed edits to extract actionable lessons, structure records (purpose, playbook, related_apis, intervention_skills).
• Implementation & evaluation: implemented with Mini-SWE-Agent and evaluated using DeepSeek-V3.2 and GPT-5-mini on SWE-bench Verified and Pro. Code and data released.

Strengths highlighted: - Does not depend on prior issue history and avoids heavy per-issue online exploration. - Entity-grounded retrieval reduces retrieval ambiguity and provides targeted guidance exactly when the agent examines relevant code. - Uses repository tests as the supervision signal, producing executable and behaviorally-grounded synthetic issues.

Limitations noted: - Requires a test suite that exercises core functionality—effectiveness depends on test coverage and quality. - Upfront synthesis and distillation requires compute; gains must offset that investment. - Distilled knowledge quality depends on LLM rewriter and trajectory quality; risk of noisy or incomplete skills in low-signal repos. - Possible dependence on model capabilities; gains reported for particular model families.

Data & Methods

• Synthetic instance generation:
  • Identify passing tests that exercise core functionality via coverage instrumentation.
  • Extract contiguous traced code segments; rank top-k critical segments using an LLM given test purpose and segment summaries.
  • For selected segments, use a “strict-mask” code-rewriting prompt: LLM must reimplement the segment without seeing the original implementation (given only limited surrounding lines and test intent). This creates plausible implementation variants that often fail tests.
  • Assemble pass→fail instances: buggy repo snapshot, failing tests, derived problem statement (LLM translates failure evidence to a user-facing prompt), and reference patch.
• Issue resolution & trajectory collection:
  • Use a SWE agent (Mini-SWE-Agent) to attempt fixes in isolated environment following SWE-bench protocol; log full action-observation trajectories (file edits, shell commands, test runs).
• Skill distillation:
  • Normalize trajectories; parse shell commands to extract accessed files/line ranges; map to AST-derived entities to produce Ecand.
  • For each entity, distill:
    • Mext: purpose, playbook (agent-tested heuristics), related_apis (co-involved APIs observed during resolution).
    • Mint: intervention_skills assembled from successful and failed patches (e.g., “avoid overriding X because...”, “ensure setup_query called before...”).
• Skill adaptation:
  • Macro-level: BM25 retrieval of relevant Mext records using the new issue description; prepend to initial prompt.
  • Micro-level: monitor agent’s real-time file accesses; inject matching Mint records as auxiliary observations just-in-time.
• Benchmarks and baselines:
  • SWE-bench Verified and SWE-bench Pro.
  • Models: DeepSeek-V3.2, GPT-5-mini.
  • Baselines include Mini-SWE-Agent and both history-driven (e.g., EvoCoder, SWE-Exp) and online adaptation methods where available.
• Experimental metrics: Pass@1 improvements reported; ablations and hyperparameter sensitivity explored (paper includes RQ2/RQ3 analyses).

Implications for AI Economics

1. Upfront investment vs per-issue cost trade-off

   • SkillForge converts some recurring, per-issue exploration cost into an upfront distillation cost. Economically, this is an investment problem: if a repository expects many future issues or frequent edits, the amortized savings (faster fixes, fewer token/test cycles per issue) can justify the upfront compute and prompting expense.
   • Suggests a cost model: one-time distillation cost C_d versus expected per-issue savings s and number of future issues N. Break-even when N*s > C_d. This frames purchasing/licensing decisions for teams and vendors.

2. Value of test coverage as an economic lever

   • The method’s reliance on tests makes test coverage a tangible productivity asset with direct economic value: better tests enable higher-quality skill distillation and hence greater agent effectiveness. Firms may find it economically rational to invest more in test suites to unlock automated-maintenance returns.

3. Productization and markets for skills

   • Distilled skill repositories (Mext/Mint) are project-specific, reusable artifacts. They could be productized:
     • Sold as part of a maintenance subscription (distill once, reuse across issues).
     • Exchanged in marketplaces for popular frameworks or libraries (common modules with many consumers).
   • Ownership and privacy implications: in closed-source or enterprise environments, skill distillation may be proprietary IP; marketplaces would require governance and pricing models.

4. Labor and task reallocation

   • By reducing repeated rediscovery and speeding fixes, SkillForge can shift human developers toward higher-level design and non-routine tasks. The effect is partial substitution for routine debugging labor; however, complex or low-test-coverage work still requires human expertise.
   • Firms should consider reskilling and reallocation strategies as agents handle more maintenance work.

5. Heterogeneous returns across repositories

   • Repos with rich tests, modular architecture, and recurring patterns will see larger marginal returns from SkillForge. Conversely, low-coverage, ad-hoc codebases have lower returns and possibly higher distillation cost per unit benefit. This implies market price discrimination: agents/offerings could tier pricing by repo-health metrics.

6. Compute and model-choice economics

   • Gains depend on base LLM capabilities. Higher-performing models improve synthesis and distillation quality but increase cost per token/call. Firms must choose model/compute trade-offs—e.g., cheaper model + more distillation iterations vs. fewer runs on a stronger model.
   • There is potential for hybrid strategies: distill with larger model once and run downstream resolution with smaller models supplemented by skills.

7. Measurement & ROI

   • To evaluate ROI, organizations should track per-issue resolution time, number of test runs, tokens consumed, and downstream bug-recurrence. Key KPIs: mean time to fix, median token/test runs per fix, and percent of issues resolved autonomously.
   • A/B tests across repos or teams can quantify labor savings and inform pricing/investment decisions.

8. Externalities and incentives

   • If proactive distillation becomes standard, incentives to maintain high test coverage and reusable modules increase—this is a positive feedback loop improving software quality and lowering maintenance costs.
   • Potential negative externalities include concentration of tooling value in large platforms that can pay upfront distillation costs, possibly widening productivity gaps between firms.

Practical recommendations for economists and product managers - Estimate expected issue volume and test-coverage quality before investing in distillation; prioritize high-variance or high-frequency repos. - Use a staged rollout: distill on core modules first, measure per-issue savings, then expand. - Consider offering “skill as a service” subscriptions for customers with similar stacks to amortize distillation costs across multiple repos. - Monitor and audit distilled skills to avoid propagation of subtle incorrect repair heuristics (quality control matters financially).

Limitations relevant to economic assessment - The paper reports relative Pass@1 gains but does not publish per-instance compute/token/time cost for distillation vs. online adaptation—needed for precise ROI models. - Effectiveness is conditional on test availability and LLM quality; these constraints should be modeled in economic projections.

Overall, SkillForge reframes part of software maintenance as an investable, reusable knowledge asset (skills grounded to code entities). That reframing opens pricing, productization, and labor-allocation opportunities, but also requires careful cost-benefit analysis tied to test-quality and expected issue flows.