Instruction files for code-writing AI are hit-or-miss: in an analysis of 15,549 agentic pull requests across 148 projects, only 27.7% of projects raised merge rates by at least 20% after adding instructions while 26.4% fell—projects that gained tended to use longer, more structured instructions.

Main Finding

Adding repository-specific instruction files for agentic coding tools (e.g., Copilot, Claude) does not reliably improve agent PR performance. Outcomes are heterogeneous: a substantial share of projects see large gains in merge rate and task complexity after adding instructions, but a comparable share see declines. Projects that do improve tend to have longer, more deeply structured instruction files.

Key Points

• Dataset and scope

  • 15,549 agentic pull requests (Agentic-PRs) from 148 GitHub projects.
  • Projects drawn from the AIDev dataset and extended via GitHub GraphQL (data through 2025-12-10).
  • Projects required ≥5 agentic PRs before and after the first instruction file creation.

• High-level quantitative results

  • Merge rate (RQ1):
    • 27.7% of projects increased merge rate by ≥20% after adding the first instruction file; 26.35% decreased by ≥20%.
    • Considering all instruction files, 31.93% increased ≥20% while 27.73% decreased ≥20%.
  • Task complexity (RQ2; merged PRs only):
    • Statistically significant increases observed in: description length (35.35% of projects), code churn (10.10%), number of commits (12.12%), number of modified files (8.08%).
    • Some projects show statistically significant decreases for these metrics (smaller fractions).
  • Effort to merge (RQ3; merged PRs only):
    • 13.13% of projects showed an increase in time-to-merge; 15.15% showed an increase in number of discussion comments.
    • 8.8% of projects showed decreases in both time and discussion.
  • Instruction-file verbosity & structure (RQ4):
    • Projects with ≥20% merge-rate increase had substantially longer instruction files (median 976 words) vs. projects with ≥20% decrease (median 569 words); difference significant (p≈0.0029).
    • These improving projects also had more headers (median H1/H2/H3 counts ≈ 3.5 / 7.5 / 9 vs. 1 / 5 / 1), i.e., finer-grained structure.

• Takeaway

  • Creation of instruction files alone is not sufficient; quality, length, and structure matter. The authors propose treating instruction development as a software engineering artifact — "Instructions-as-Code" — with standards and quality metrics.

Data & Methods

• Data collection

  • Base: AIDev dataset of agentic PRs extended to Dec 10, 2025 using GitHub GraphQL.
  • Project filtering: ≥20 stars, excluded toy projects/templates, no forks/archived, ≥50 commits, ≥50 PRs.
  • Instruction files located via filename predicates for common agent APIs (e.g., .github/copilot-instructions.md, CLAUDE.md, AGENTS.md, etc.). Creation time taken from first commit adding the file.
  • Final sample: 148 projects with ≥5 agentic PRs before and after instruction-file creation → 15,549 agentic PRs.

• Metrics compared (before vs. after)

  • Success: merge rate (per-project).
  • Complexity (merged PRs): description length, code churn, number of modified files, number of commits.
  • Effort (merged PRs): number of comments, time to merge.
  • Instruction-file features: word count and number/counts of Markdown header levels (H1–H3 as proxy for structure).

• Statistical analysis

  • Per-project comparisons using Mann–Whitney U tests (α = 0.05) and effect-size reporting via Cliff’s delta.
  • Two comparison modes: before vs. after first instruction file; before first vs. after last instruction file (if multiple).

• Limitations noted by authors (implicit/observational)

  • Observational before/after design — association not proven causation.
  • GitHub-only and selection criteria may bias toward larger/popular projects.
  • Merge rate is an imperfect proxy for PR quality or long-term maintainability.

Implications for AI Economics

• Heterogeneous returns to investment in instruction files
  • The mixed outcomes imply that investing developer time to author instruction files yields variable returns. Economically, this introduces uncertainty in the productivity gains from deploying/optimizing agentic tooling.
• Cost–benefit and adoption decisions
  • Organizations should weigh upfront costs (developer time to author & curate instructions; possible short-term increases in discussion/merge time) against expected productivity gains. Richer, well-structured instructions appear to offer higher expected returns, suggesting diminishing returns to minimal or poorly structured guidance.
• Standardization and public goods
  • Standards, quality metrics, and best-practice templates for instruction files could lower search/coordination costs and increase aggregate returns — an area where platform providers, open-source foundations, or public-good initiatives could add value.
• Market and product opportunities
  • There is potential for tools and services (e.g., instruction-file linting, automated generation/optimization, marketplaces for instruction templates) that increase ROI for organizations using agentic AIs.
• Labor-market and task-allocation effects
  • If high-quality instructions raise the autonomous capability of agents for complex tasks, this could shift developer time toward higher-level work (design, architecture, review) and increase productivity per developer — but gains will be uneven across teams depending on instruction quality.
• Measurement and monetization
  • To make firm investment decisions, firms and researchers need models that translate observed changes (merge rate, code churn, time-to-merge) into economic outcomes (developer-hours saved, defect costs avoided, release velocity gains). The paper motivates work to quantify expected value per unit of instruction-file effort.
• Policy for platforms and firms
  • Platform owners (e.g., GitHub) can influence economic outcomes by providing better instruction-file UX, defaults, analytics, and incentives (e.g., exemplars, tooling) that reduce the cost of creating high-quality instructions.

Suggested next steps for researchers and practitioners from an AI-economics viewpoint - Run controlled experiments (A/B) to estimate causal impact and compute expected ROI of diverse instruction-file interventions. - Develop standardized quality metrics for instruction files tied to economic outputs (time saved, acceptance rate lift). - Build tools to automate generation or evaluation of instruction files and measure marginal productivity gains. - Explore market design for instruction-as-code (templates, audits, certification) and platform incentives to foster high-quality instruction artifacts.