0 cumulative citations
View corpus contextPackaging operational know-how from code and papers into compact, verified 'skills' sharply raises autonomous research agents' output: with the same LLM, harness and execution budget, skill-equipped agents outperform skill-free ones across four research benchmarks—up to a 134% boost on MLE-bench.
Repo-To-Skill: Distilling GitHub Repositories Into AI4AI Skills
Automatically distilling operational 'skills' from ML repositories and papers into a reusable skill library substantially improves autonomous ML-research agent performance (e.g., +134% on MLE-bench) when the model, harness, and compute are held fixed.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Autonomous agents are beginning to carry out machine-learning (ML) research end to end. These agents combine a model backbone with a harness for planning, execution, memory, and verification, but this architecture still leaves domain-specific know-how outside the agent. We call this missing layer operational knowledge, the know-how that separates knowing a method from making it work. That knowledge is not absent from the field. It appears in repositories and papers, but in forms written for human readers and too large to load during a task. Once distilled into compact, verified skills, this knowledge can be reused across tasks rather than rediscovered during each run. We present DisCo, a skill-powered research agent that creates skills and uses them during research. Its distillation runs in two complementary forms: task-agnostic, condensing the field's widely used repositories into reusable skills, and task-oriented, producing the skills a concrete task calls for. The former, applied across the open ecosystem, yields the AREX-Skill Library, with 5,000+ verified skills distilled from 1,000 widely used ML repositories and organized into 20 areas and 178 capability families. With the GPT-5.5 backbone, research harness, and downstream execution budget held fixed, the skill-equipped research agent scores 134.3% higher on MLE-bench, 34.4% higher on PaperBench, 9.2% higher on FrontierCS, and 14.0% higher on PassNet than the same agent without skills. These gains come from adding distilled operating context under that fixed setup.
Summary
Main Finding
The paper identifies "operational knowledge"—the task-level know-how that turns methods and code into working research—as the missing layer for autonomous ML research agents, and shows that distilling this knowledge into compact, verified "skills" substantially improves agent performance. The authors introduce DisCo, a system that automatically converts repositories and papers into skill graphs (the AREX-Skill Library: 5,000+ skills from 1,000 repos) and demonstrate large performance gains on multiple ML research benchmarks when the same agent is given these skills versus not: +134.3% (MLE-bench), +34.4% (PaperBench), +9.2% (FrontierCS), and +14.0% (PassNet), holding model backbone, harness, and execution budget fixed.
Key Points
- Operational knowledge: the procedural, contextual know-how (e.g., API usage, configs, failure modes) that lets an agent execute methods rather than just reason about them.
- Skills: self-contained, agent-facing units encoding capability + usage policy. Each skill has three layers:
- SKILL.md (knowledge interface; summary and SOP)
- references/ (substrate; docs, evidence)
- scripts/ (execution interface; wrappers)
- Skill graphs: multiple skills per source organized into entry nodes and links (routing/dependencies). Progressive disclosure lets agents load only the small subset needed for a task.
- Distillation pipeline (four stages): scope → ground → construct → verify. Verification is mandatory; gaps are recorded and repaired where possible.
- Two distillation modes:
- Task-agnostic (repo-to-skill): precompute reusable skills from repositories/papers.
- Task-oriented: on-demand skill generation targeted to a particular research task.
- DisCo operates in two modes:
- Creator mode: distills sources into skill graphs and deposits them in the AREX-Skill Library.
- Researcher mode: agent uses the relevant skills as operating context when solving tasks.
- Empirical isolation: experiments hold backbone (GPT-5.5), harness, and downstream execution budget fixed; skills are the only variable at runtime.
- Public artifact: AREX-Skill Library and code (link provided in paper).
Data & Methods
- Data sources:
- 1,000 widely used ML repositories (task-agnostic distillation).
- 153 papers for paper-derived skills.
- Produced artifacts:
- AREX-Skill Library: 5,000+ verified skills organized into 20 areas and 178 capability families.
- Each distilled source yields a skill graph with entry point, links, scripts, and references; construction records capture evidence and unresolved gaps.
- Distillation method:
- Automated LLM-driven pipeline (DisCo) that scopes capabilities, extracts grounding evidence, constructs SKILL.md/references/scripts, and verifies by running checks or trials. Verification repairs or records remaining gaps.
- Evaluation design:
- Baseline: same research agent (GPT-5.5 backbone + standardized harness) without skills.
- Treatment: identical agent with access to the distilled skill library and/or task-oriented skills.
- Benchmarks: MLE-bench, PaperBench, FrontierCS, PassNet.
- Execution budget and harness held fixed; skill construction completed before downstream runs so skills are the only runtime variable.
- Reported gains (skill-equipped vs skill-free, matched settings):
- MLE-bench: +134.3%
- PaperBench: +34.4%
- FrontierCS: +9.2%
- PassNet: +14.0%
Implications for AI Economics
- Productivity and cost efficiency:
- Distilled skills reduce redundant trial-and-error, lowering wasted compute and experimental budgets and increasing per-dollar research output. This implies higher returns to computational R&D budgets and potentially faster innovation cycles.
- Labor and comparative advantage:
- The value of routine operational know-how becomes automatable; human labor may shift toward higher-order roles (skill curation, verification, creative research planning). Demand for ML engineers who do repetitive integration/testing may decrease while demand for oversight, verification, and skill-authoring increases.
- Returns to scale and public goods:
- Prebuilt, reusable skill libraries create strong increasing returns: one-time creator-mode cost amortized across many tasks. If skill libraries are public goods (open), they can democratize research productivity; if proprietary, they can generate platform lock-in and concentration effects.
- Market structure and rents:
- Entities that build and control high-quality, verified skill repositories can capture value by becoming indispensable operating-context providers for research agents—analogous to how curated datasets or infrastructure can confer competitive advantage.
- Innovation diffusion and replication:
- Easier replication and reuse of operational knowledge could accelerate diffusion of methods across firms and academia, lowering barriers to entry for advanced ML research and reducing duplicative R&D spend.
- Incentives around documentation and maintenance:
- Repositories and papers that include actionable operational details (good SKILL.md-like documentation) will be more valuable. This creates incentives for better documentation but also for strategic withholding of operational details if actors seek competitive advantage.
- Externalities and compute waste:
- Verified skills that avoid misconfigurations and failed runs reduce negative externalities from wasted energy/compute, which has social welfare implications in contexts where compute is costly or carbon-intensive.
- Intellectual property and governance:
- Distillation of public repos into skill artifacts raises IP and licensing questions (reuse, redistribution, attribution). Economic models and regulation may be needed to govern monetization, licensing, and liability for incorrect/unsafe skills.
- Measurement opportunities for economists:
- Quantify cost-per-successful-experiment with and without skills.
- Estimate labor reallocation and wage impacts across ML occupations.
- Model adoption dynamics of skill libraries and potential market concentration.
- Evaluate social welfare trade-offs between open-source public skill libraries and proprietary skill platforms.
Limitations and cautions relevant for economic analysis: - Results depend on backbone (GPT-5.5) and the chosen harness; gains may vary with other models or orchestration systems. - Skill coverage and freshness matter: repositories drift, so maintenance and verification costs are ongoing. - Verification is necessary but not foolproof—residual gaps and domain-specific edge cases remain. - Potential selection bias in the 1,000 repos and 153 papers chosen for distillation.
For economists studying AI-driven R&D, this paper supplies a concrete mechanism (skill distillation) that can materially change the cost structure and organization of ML research—making it a useful case for empirical work on productivity, labor demand, platform effects, and the governance of AI knowledge assets.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper provides quantitative, within-system comparisons showing large relative gains across several research-focused benchmarks while holding major components constant, which supports a causal interpretation that skills produced the gains. However, evidence is limited to specific benchmarks, a single backbone (GPT-5.5) and a curated set of repos/papers; the excerpt lacks detailed statistical tests, ablation checks for alternative explanations (e.g., tuning, overlap between skill sources and benchmarks), and full reproducibility details.
Methods Rigormedium — The design isolates a single variable (skills) by fixing backbone, harness, and budget and verifying skills before use, which is a strong experimental control. But the supplied text (excerpt) does not show full experimental protocols, randomization, statistical significance tests, baseline tuning parity, or robustness/ablation studies in detail; potential selection/overlap between training sources and benchmarks and limited external-validity checks reduce rigor.
SampleTask-agnostic skills distilled from ~1,000 widely used ML repositories (yielding 5,000+ verified skills, organized into 20 areas and 178 capability families) and additionally paper-derived and task-oriented skills from 153 papers; evaluations run with a fixed GPT-5.5 backbone and a fixed research harness and execution budget, on four benchmarks: MLE-bench, PaperBench, FrontierCS, and PassNet (reported gains: +134.3% MLE-bench, +34.4% PaperBench, +9.2% FrontierCS, +14.0% PassNet). Skills undergo automated construction and verification steps (scope, ground evidence, construct candidate skill graph, verify/repair), and are exposed via a router and progressive-disclosure interface.
Themesproductivity innovation
IdentificationControlled within-system A/B comparisons: the authors hold the LLM backbone (GPT-5.5), the research harness, and downstream execution budget fixed, construct skills ahead of downstream runs, and compare the same agent with and without the distilled skill layer on four benchmarks (MLE-bench, PaperBench, FrontierCS, PassNet); skill construction is completed before evaluation so skills are the only run-time variable.
GeneralizabilityResults rely on a single proprietary/specified backbone (GPT-5.5); gains may differ with other LLMs or smaller models., Skills were distilled from a curated set of 1,000 ML repositories and 153 papers — performance may depend on this corpus and its overlap with benchmark tasks., Benchmarks are research-centric; findings may not generalize to non-ML tasks or non-research productivity outcomes., Paper excerpt lacks details on statistical robustness, hyperparameter parity, and potential tuning differences between conditions., Skill maintenance and freshness over time (as repositories evolve) may affect long-run effectiveness., Verification and skill construction procedures may be engineering-intensive and hard to replicate exactly in other environments.
Claims (8)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| The AREX-Skill Library contains more than 5,000 verified skills distilled from 1,000 widely used machine-learning repositories, organized into 20 areas and 178 capability families. Adoption Rate | positive | Scale and coverage of the reusable AI skill library |
Reading fidelity
high
Study strength
medium
|
n=1000
5,000+ skills from 1,000 repositories
|
| DisCo performs skill distillation in two forms: task-agnostic distillation that builds reusable skills from sources such as repositories and papers, and task-oriented distillation that produces skills for a concrete research task. Task Allocation | positive | Task allocation and reuse strategy for producing operational knowledge |
Reading fidelity
high
Study strength
medium
|
not reported
|
| The skill-equipped research agent scored 134.3% higher on MLE-bench than the same agent without skills. Research Productivity | positive | Performance on MLE-bench |
Reading fidelity
high
Study strength
medium
|
134.3% higher
|
| The skill-equipped research agent scored 34.4% higher on PaperBench than the same agent without skills. Research Productivity | positive | Performance on PaperBench |
Reading fidelity
high
Study strength
medium
|
34.4% higher
|
| The skill-equipped research agent scored 9.2% higher on FrontierCS than the same agent without skills. Research Productivity | positive | Performance on FrontierCS |
Reading fidelity
high
Study strength
medium
|
9.2% higher
|
| The skill-equipped research agent scored 14.0% higher on PassNet than the same agent without skills. Research Productivity | positive | Performance on PassNet |
Reading fidelity
high
Study strength
medium
|
14.0% higher
|
| Under the reported evaluation setup, the addition of distilled skills was the only runtime variable between the skill-equipped and unequipped research agents. Research Productivity | positive | Attribution of research-agent performance differences to skill augmentation |
Reading fidelity
high
Study strength
medium
|
not reported
|
| The paper claims that no skill is admitted to the library without verification, and that unresolved gaps are recorded rather than hidden. Ai Safety And Ethics | positive | Verification and reliability of distilled operational knowledge |
Reading fidelity
high
Study strength
medium
|
not reported
|