Auto-generated 'SKILL' documents give coding agents modest help: GEPA-produced files raise agent performance by roughly 5 percentage points on average across three Kotlin repositories, but the improvement is indistinguishable from the agent's run-to-run variability given the small number of real pull-request tasks.

Main Finding

Automatically synthesized in-repository SKILL.md files can contain valuable, project-specific knowledge and sometimes improve a coding agent’s performance on real merged-PR tasks, but measured performance gains are small, within the agent’s own run-to-run variance, and costly to discover given the limited number of usable tasks a single repository provides.

Key Points

• Task mining: The authors mine real, harder tasks by reverting merged pull requests to a single frozen base commit and using the tests that fail after the reversion as the task specification (reverse-PR mining). This avoids historical leakage that arises when tasks are anchored to differing historical commits.
• Proposers compared: Two reflection-driven optimizers were evaluated:
  • GEPA (free-form rewrites + Pareto frontier sampling).
  • SkillOpt (bounded add/delete/replace edits with an improvement gate and edit-budget).
• Scoring: Candidates are evaluated by pairing each candidate rollout with the seed (empty SKILL) rollout on the same task. A candidate only gains credit when it improves the agent’s outcome on that exact task compared to the seed rollout.
• Aggregate results (three Kotlin repos: JetBrains/koog, kotest/kotest, ktorio/ktor):
  • GEPA’s SKILLs raised the mean paired score by 4.9 percentage points on average across repos.
  • SkillOpt’s SKILLs changed the mean by +0.1 percentage points.
  • The same agent (Claude Code, Sonnet 4.6) solved ~53% of reverse-PR tasks without any SKILL, leaving nontrivial headroom for improvement.
• Statistical power and variance:
  • Held-out test splits were small (≈20–26 tasks per repo); the observed gains are not statistically separable from agent run-to-run variance.
  • The run-to-run variance of the agent compounds with scoring variance, so differences of the observed magnitude are inside the agent’s own spread.
• Practical artifacts and human review:
  • Optimized SKILLs read well and contained repository-specific operational knowledge a maintainer recognized as useful.
  • On two live issues in koog, using either generated SKILL reduced wall-clock time and cost for the agent to finish the tasks.
• Implementation caveats:
  • Reverse reversion uses a three-tier strategy: git apply --reverse, structural file add/remove, and (for hard cases) an LLM to reconstruct pre-change source to produce a real patch.
  • The LLM tier is responsible for most usable tasks but also introduces defects that must be filtered.
  • Mining yield is limited: roughly 1 in 5 merged PRs survived the filters (e.g., 119/660 for koog).
• Cost of optimization:
  • Average agent rollout cost ≈ $0.84.
  • Total experimental spend ≈ $2,014 across three repos and full optimization runs; per-repo spends ranged from ≈$182 to ≈$485 in these runs (table in paper).

Data & Methods

• Repositories: Three Kotlin/JVM projects: JetBrains/koog, kotest/kotest, ktorio/ktor.
• Task construction:
  • For each merged PR, the implementation change is reverted at a single frozen base commit.
  • The tests that go from passing→failing after the reversion form the FAIL_TO_PASS target set for the task; these tests are used for grading.
  • Tasks are validated (build/test once at base, then with reverted patch) and rejected if they don't discriminate candidates.
• Scoring & pairing:
  • Every candidate rollout is paired with the seed rollout on the same task; the seed (empty .md) run establishes a baseline of 0.5.
  • Pairwise comparison proceeds by (1) correctness vs test file edits and contradictions, (2) satisfying all FAIL_TO_PASS tests, then tie-breakers including fraction of tests passed, honesty of claims, diff distance, tool-call counts, and run cost (bounded).
• Optimizers:
  • GEPA: reflection LM proposes full rewrites; maintains a Pareto frontier; accepts when improvements shown on minibatches then larger selections.
  • SkillOpt: bounded edits with edit-budget, rejected-edit buffer, epoch-level momentum, strict held-out selection improvement required to accept changes.
• Agent & tooling:
  • Agent: Claude Code (Sonnet 4.6), frozen for all candidates.
  • Runs executed inside fresh Docker containers built per repository; repository history removed before each attempt to prevent leakage.
• Experimental budget:
  • GEPA allowed 200 scored attempts per repository; SkillOpt run as three epochs.
  • Typical rollout cost ~ $0.84; wall-clock totals and spends reported per repo (see paper table).

Implications for AI Economics

• Cost vs. benefit: Finding small improvements to in-repo documentation (SKILLs) by directly optimizing against the deployed agent is expensive. Even modest optimization runs cost hundreds to thousands of dollars because each candidate evaluation requires a full agent rollout. For a single repository, the marginal benefit (a few percentage points) may not justify the monetary and compute cost under current pricing.
• Measurement & statistical power: Small held-out task pools (dozens of tasks) and high agent variance make it difficult to detect real improvements economically. In practice, robust detection of small SKILL effects requires either (a) many more tasks (e.g., pooling data across repositories or time), (b) cheaper proxy evaluations with good transfer, or (c) variance-reduction techniques (multiple rollouts per candidate, controlled randomness), each with its own cost trade-offs.
• Infrastructure vs. simplicity trade-off:
  • In-repo SKILLs (plain .md) have governance and integration advantages (versioning, PR review, no external memory infra), which reduce organizational friction and operational costs compared to retrieval+vector stores and separate memory services.
  • However, optimizing those artifacts via expensive LLM-driven pipelines reintroduces costs and complexity at development time. Organizations should weigh these one-time or episodic optimization costs against ongoing infrastructure costs of external memory systems.
• Economies of scale and pooling:
  • Because single-repo histories yield few usable tasks, there is an economic argument for pooling task data across similar repositories or organizations to increase statistical power per optimization budget. This implies coordination benefits (shared benchmarks, standardization) and possible market opportunities (services that aggregate tasks and provide optimization at scale).
• Value beyond measured pass rates:
  • Quantitative gains can understate economic value: human review found SKILLs contained maintainer-only operational knowledge and reduced agent wall-clock time and cost on live issues. Even if aggregate pass-rate deltas are small/insignificant, downstream labor savings and faster issue resolution are economically meaningful.
• Recommendations for practitioners and researchers:
  • Before running expensive direct-optimization loops, estimate expected task yield and required sample size to achieve detectable improvements; often this will require pooling tasks or accepting proxies.
  • Consider hybrid approaches: use cheaper proxy agents or smaller simulators to pre-screen candidate edits, then validate a small set of promising candidates with full rollouts.
  • Invest in variance reduction (multiple rollouts per candidate; fixed seeds where possible) to increase signal-to-noise and avoid wasting budget on indistinguishable candidates.
  • Account for non-metric benefits (documentation quality, reviewability) when evaluating the ROI of SKILL automation pipelines.
  • For platforms offering SKILL-optimization as a product, provide transparent cost estimates per repo and encourage shared-task markets to improve statistical power.

Overall, the paper shows that while automated SKILL synthesis can produce usable, project-specific knowledge and occasionally improve agent behavior, the economic case for direct in-repo optimization—given current agent costs and limited task samples per repo—is weak unless organizations can pool data, reduce evaluation costs, or prioritize the non-quantified operational benefits (knowledge capture, reviewability, faster issue handling).