Short repository instruction files materially speed up AI coding agents: adding an AGENTS.md cut median execution time by ~28.6% and trimmed output token consumption by ~16.6% across tests on 124 GitHub pull requests, suggesting simple config changes can lower runtime and API costs.

Main Finding

The presence of a repository-level AGENTS.md file (with project description, architecture, and coding conventions) measurably improves operational efficiency of an autonomous AI coding agent (OpenAI Codex gpt-5.2-codex) on real GitHub pull-request reconstruction tasks: median wall-clock time-to-completion decreased by 28.64% and median output-token consumption decreased by 16.58%. Mean reductions were similar or larger for some metrics (wall-clock mean −20.27%, mean output tokens −20.08%, mean total tokens −9.93%). Wall-clock and output-token differences were statistically significant (Wilcoxon signed-rank, p < 0.05).

Key Points

• Dataset and scope
  • 10 sampled repositories (from a filtered pool), 124 pull requests (paired runs).
  • PRs chosen were merged, small-scope code changes (≤100 LoC, ≤5 files), and occurred after the AGENTS.md introduction.
  • AGENTS.md files included only if they contained conventions/best practices, architecture/project structure, and project description.
• Experimental design
  • Paired within-task design: each PR replayed twice on the identical pre-merge snapshot — (a) with the repository’s root AGENTS.md present, and (b) with it removed.
  • Agent: OpenAI Codex (gpt-5.2-codex) via a reproducible Python wrapper; experiments run in isolated Docker sandboxes.
  • For PRs lacking descriptive metadata, task descriptions were generated via a local LLM to standardize input.
• Metrics
  • Measured total tokens (input + cached input + output), and wall-clock time-to-completion.
  • Reported reductions: median wall-clock time from 98.57s → 70.34s (−28.64%); median output tokens from 2,925 → 2,440 (−16.58%).
  • Mean statistics show larger absolute reductions in some cases (e.g., mean output tokens −1,153 tokens ≈ −20.08%).
  • The reduction pattern suggests the largest gains occur in the high-cost tail (a few expensive runs reduced substantially), while medians show consistent general improvements.
• Limitations called out by the authors
  • Single agent/model family (Codex) used.
  • Focused on small PRs and on a filtered subset of AGENTS.md contents.
  • No full automated correctness/alignment evaluation (only a manual sanity check on outputs).
  • Results are initial evidence and motivate broader replication and deeper behavioral analyses.

Data & Methods

• Repository selection and filtering
  • Started from prior corpus of repositories using agent instruction files; restricted to repositories with a single AGENTS.md at root.
  • Further filtered AGENTS.md files using an LLM classifier + manual verification to keep files containing architecture, conventions, and project description; final pool used for sampling was 26 repos, 10 sampled for experiments.
• Pull-request reconstruction
  • For each PR, repository checked out at the pre-merge commit and the agent was tasked to recreate the merged change from that state.
  • Up to 15 merged PRs per repo were sampled under the size/scope constraints, yielding 124 task instances.
• Task input standardization
  • Where PR text was insufficient, a local LLM (gpt-oss-120b) generated a GitHub-issue-style task description from the PR diff and repo structure.
• Experimental conditions
  • Paired runs: identical repository snapshots and task descriptions, differing only by presence/absence of the root AGENTS.md.
  • Runs executed in fresh Docker containers; no cross-run state reuse.
• Metrics and analysis
  • Collected input tokens, cached input tokens, output tokens, total tokens, and wall-clock completion time.
  • Statistical comparison used Wilcoxon signed-rank tests; wall-clock and output-token differences were significant at p < 0.05.

Implications for AI Economics

• Direct cost savings (compute/token billing)
  • Token usage reductions translate directly to lower per-request compute billing in token-priced models. The paper reports mean total tokens down ≈9.9% and mean output tokens down ≈20.1%; so, ceteris paribus, an organization heavily using similar agents could expect proportional cost reductions per task when repos include informative AGENTS.md files.
  • Because gains are concentrated partly in the tail (fewer very expensive runs), AGENTS.md can disproportionately reduce outlier costs and thus reduce variance and peak spending.
• Latency and throughput (operational economics)
  • Median wall-clock latency reduced by ~28.6%; faster agent runs improve developer feedback loops and increase throughput of automated CI/agentic workflows, which has second-order economic value (reduced human wait times, faster merges, higher pipeline utilization).
• Scalability and marginal economics
  • Small per-task savings compound at scale: organizations automating many PR-level tasks can realize measurable savings and improved SLA guarantees.
  • The cost of producing and maintaining AGENTS.md (authoring time, upkeep) is an investment that may have attractive ROI if agents are used frequently. The paper motivates empirical ROI studies quantifying break-even points.
• Product and marketplace design
  • Repositories that are “agent-ready” (clear AGENTS.md) could be priced or signaled as higher-value in commercial marketplaces or internal platform teams; platform features could surface or require AGENTS.md to reduce downstream agent costs.
  • Cloud/LLM providers and internal platforms could offer tooling that ingests AGENTS.md to optimize context delivery (reducing redundant context tokens) and to implement cost-saving defaults.
• Policy and governance
  • Standardizing repository-level agent instructions could enable cost predictability and contractual SLAs for agentic automation (e.g., predictable latency and bounded compute usage).
• Research and measurement priorities (economic-focused)
  • Estimate aggregate cost savings across agent types, repo sizes, and enterprise-scale workloads to build decision models for whether to invest in authoring AGENTS.md.
  • Quantify maintenance costs of keeping AGENTS.md up to date vs. the operational savings they produce.
  • Study interactions between AGENTS.md specificity/length and token-cost trade-offs (longer AGENTS.md adds input tokens; but here input-token median was roughly unchanged while output-token and latency fell).
  • Explore pricing and billing models that account for repository-level context artifacts (e.g., discounted context reuse, repo-level context caching).
• Cautions for economic interpretation
  • Results are conditional on a single agent/configuration and small PRs; generalization to other models, larger tasks, or different AGENTS.md designs is not yet proven.
  • Authors did not measure functional correctness comprehensively; efficiency gains that degrade correctness would have negative downstream economic consequences.

Summary recommendation for practitioners and platform designers: encourage concise, actionable repository-level agent instructions (architecture, conventions, project description) as a low-effort, potentially high-return measure to reduce agent latency and generation cost; evaluate ROI at scale and validate across the specific agents and workloads used in production.