Public agent skill repositories are maintained in human-governed, AI-assisted loops: in a five-repository study every substantive SKILL.md edit passed through a named human account while 62% disclosed an AI co-author, and most edits add or correct content rather than prune skills.

Main Finding

Public maintenance of LLM agent "skill" Markdown files (SKILL.md) currently operates as a human‑governed, AI‑assisted loop: most substantive edits (62%) disclose AI co‑authorship in commits, but every substantive edit in the studied repositories is authored or merged through a named human account. Maintenance is mostly constructive (additions/expansions) rather than pure repair, and reliably measuring whether edits produce generalizable "rules" remains an open problem.

Key Points

• Corpus and scope
  • Purposive sample: 5 public AI‑tooling repositories (getsentry, trailofbits, obra, cloudflare, anthropics).
  • Pinned clones spanning Oct 2025–Jun 2026: 873 commits, 143 SKILL.md files, 631 commit-file records, 254 substantive post‑creation edits.
  • Released artifacts: mining scripts, codebooks, corpus, and a replay protocol for automated curators (link in paper).
• Governance
  • 62% (158/254) of substantive edits include a Co‑Authored‑By trailer indicating AI co‑authorship; 38% have no AI trailer.
  • Every substantive edit (non‑bot set) is authored or merged via a named human account; no agent‑only substantive commits observed.
  • Governance regime varies sharply by organization (bimodal patterns): some repos are largely AI‑trailer‑heavy, others largely human‑merged.
• What maintenance does
  • Operation taxonomy (8 leaf ops collapsed into corrective vs enhancement):
  • Most common leaf ops: content expansion (72), factual correction (56), restructure (50), fix‑from‑failure (41), description tuning (19), consolidation/merge (10), deprecation (1).
  • Collapsed: 60% enhancement (perfective) vs 38% corrective (repair), 2% other.
  • Many factual‑correction rows are concentrated in multi‑file mass‑refactor commits; excluding those reduces the corrective share.
• Where edits occur & dynamics
  • Instructions (skill body) are the most edited components: 85% of edits touch instructions; 56% touch embedded code; router/frontmatter/examples/references edited far less.
  • Cadence: median interval between successive commits touching a skill ≈ 5 days.
  • Size change: among 120 skills with ≥2 observations, 32 grew >10%, 7 shrank >10%, 81 stayed roughly stable.
• Measurement and reliability
  • Coding pipeline: commits coded along governance (deterministic metadata), operation (8‑way), and trigger‑evidence (0–3) axes.
  • Coding used an LLM (Opus 4.8) in two passes; reliability on 8‑way operation taxonomy moderate (κ ≈ 0.62 inter‑pass) but stronger after collapsing to corrective vs enhancement (validated by human recode, κ ≈ 0.72).
  • A pre‑registered "rule‑likeness" axis (whether edits are generalizable rules vs instance‑bound) failed reliability checks (inter‑rater κ ≈ -0.02); cannot report distribution.
• Limitations noted by authors
  • Purposive (non‑representative) sample and pinned commits; results descriptive of these repositories.
  • Trailer‑visible AI is a proxy for AI involvement (construct validity constraints).
  • LLM‑based coding, potential repository‑revealing inputs, right‑censoring window, and mass‑refactor sensitivity.

Data & Methods

• Data collection
  • Git mining: tracked SKILL.md files with git --follow, froze repository clones to pinned commits to create reproducible snapshots.
  • Substantive edit definition: post‑creation commits changing ≥5 lines in a SKILL.md file, excluding bot‑authored/trivial commits → 254 substantive edits.
• Coding & analysis
  • Three pre‑registered codebooks:
  • Governance (deterministic metadata): bot author; AI Co‑Authored‑By trailer present/absent; human author with web‑merge committer; human author+committer.
  • Curation operation: 8‑way taxonomy (content‑expansion, restructure, description‑tuning, consolidation‑merge, deprecation‑retirement, factual‑correction, fix‑from‑failure, other); collapsed to corrective vs enhancement for headline claims.
  • Trigger evidence: 0–3 ladder (L3 explicit issue/failure link → L0 none).
  • Automated coding: Opus 4.8 applied across all edits in one pass and to a stratified 50‑edit sample in a second pass; 50‑edit human blind recode used to validate the corrective/enhancement collapse.
  • Statistical checks: Cohen’s κ for inter‑pass agreement, sensitivity analyses excluding mass‑refactors, repository‑stratified tests (Holm correction, Cochran–Mantel–Haenszel).
• Released materials: all scripts, codebooks, and processed data to reproduce mining and enable replay evaluations.

Implications for AI Economics

• Labor vs automation: evidence of human governance with frequent AI assistance implies current automation substitutes for parts of the drafting/authoring work but not for the final release/merge decision. Economically, this suggests partial automation rather than full labour replacement—demand for human maintainers (reviewers, integrators, curators) may persist even as AI raises per‑worker productivity.
• Complementarity and task decomposition: AI tools appear to complement human maintainers (co‑authorship trailers), performing additions/corrections while humans merge and govern. Models of task allocation should treat skill curation as a joint human‑AI production process with complementarities (supervision, quality control, and integration tasks remaining human‑intensive).
• Heterogeneous adoption and firm regimes: sharp org‑level differences (bimodal regimes) indicate adoption heterogeneity. Economic analyses should model firm‑level adoption choices, governance norms, and workflow constraints rather than assuming uniform technology diffusion. Returns to automating skill curation will vary by organization, incentive structures, and attribution practices.
• Measurement and valuation challenges: the failed "rule‑likeness" measurement highlights a core obstacle for valuing automation: we cannot reliably distinguish edits that generate broadly reusable rules from one‑off fixes using commit artifacts alone. This complicates estimating the long‑run productivity gains from automated generalization and building markets for reusable skill assets.
• Quality and welfare considerations: prior benchmarks report higher human‑authored skill performance; the observed human‑in‑the‑loop governance suggests firms prioritize human oversight possibly to protect quality. Economically, this could justify continued human oversight as a form of quality assurance, with implications for pricing, liability, and investment in verification tools.
• Markets & productization opportunities:
  • Demand for tools that support human governance (review UIs, audit trails, provenance accounting) and that can certify generalizability of edits.
  • Replay protocol and corpus enable more realistic evaluations of automated curators—critical for vendors seeking to monetize automated skill‑curation systems.
• Policy and incentives:
  • Attribution practice (Co‑Authored‑By trailers) is observable and affects measurement; incentives or regulation around disclosure could influence apparent adoption and labor accounting.
  • Standards for measuring generalizability and maintenance outcomes would improve market transparency and help regulators and firms assess automation impacts.
• Research directions for economists
  • Model labor demand for curators under joint human‑AI production with heterogenous firm regimes.
  • Estimate the productivity elasticity to AI assistance: how much time/cost saved per human curator per unit of AI assistance, accounting for quality differences and rework.
  • Study incentives for firms to disclose AI contributions and how disclosure affects adoption, labor bargaining, and competition.
  • Investigate welfare effects of automation when measurement failures (e.g., rule‑likeness) impede capitalization of reusable skill assets.

Limitations to keep in mind for economic inference: the dataset is purposive (not population representative), the AI‑trailer signal is a proxy (not a verified measure of AI contribution), coding relies on LLMs with validated but imperfect reliability, and results are right‑censored to a limited time window. Despite these caveats, the released corpus and replay protocol provide concrete inputs for empirical economic work on the productivity and labor effects of AI assistance in software/agent maintenance.