1 cumulative citations
View corpus contextRepository AGENTS.md files do not improve coding agents' pass rates on gold tests across two leading models, with correctness unchanged and only narrow process/efficiency shifts; agent-specific task difficulty bands explain why single-agent studies previously reported conflicting results.
Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories
arxiv
quasi_experimental
medium evidence
7/10 relevance
Full text
usable extracted full text
Source
PDF
Across 15–17 real PR-based coding tasks and 288 evaluated runs on two leading coding agents, persistent repository context files (AGENTS.md) did not measurably increase correctness (bounded to ≤10–15 percentage points), though small agent- and repo-specific process/efficiency signals were observed.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Persistent context files (AGENTS.md, CLAUDE.md) are standard practice for guiding AI coding agents, yet evidence for their effectiveness is contradictory. We present a controlled ablation of context-injection strategy across two frontier agents (Claude Code and Codex), 17 real tasks from 3 repositories (15 shared + 2 Codex-only), and 288 evaluated runs with gold-test evaluation. Context strategy does not measurably move correctness on either agent (bounded to <=10-15pp via equivalence testing). A failure-mode triage reveals why: agents fail on implementation skill---feature design, pattern selection, exact wiring---not missing repository knowledge that a context file could supply; a manipulation probe confirms the real AGENTS.md never converts a near-miss to a pass on either agent. We further show that borderline task difficulty is agent-specific (Spearman rho=0.75), offering a candidate explanation for prior contradictions: single-agent studies draw tasks from different agents' informative bands. We release all code, data, and analysis.
Summary
Main Finding
Persistent repository context files (AGENTS.md, CLAUDE.md) do not measurably improve coding-agent correctness under the tested conditions. Across two frontier agents (Anthropic Claude Code sonnet-4-6 and OpenAI Codex gpt-5.5), 3 injection strategies, and 288 evaluated runs on real merged-PR tasks with hidden gold tests, strategy-driven context injection changed pass-rates by at most ~≤10 percentage points for Claude and ≤15 pp for Codex (TOST-bounded). A focused probe and failure-mode analysis show that when agents fail, it is because of implementation-skill errors (design, pattern choice, wiring), not missing repository facts that an AGENTS.md could supply.
Key Points
- Experimental design
- Three injection strategies: NONE (no AGENTS.md), ALWAYS ON (AGENTS.md injected each turn in system prompt), SELECTIVE (topic wiki files in workspace + hint to consult them).
- AGENTS.md removed from workspace in all arms so only the injection channel supplies that content.
- SELECTIVE corpus caveat: for one repo (opshin) it matched AGENTS.md; for pdm and firebase it used a larger auto-generated wiki (10×–18× words), so SELECTIVE varies both delivery mechanism and corpus for those repos.
- Tasks & runs
- Repositories: pdm, firebase-admin-python, opshin (Python only).
- Tasks: 17 for Codex, 15 for Claude (two high-effort tasks dropped from Claude due to budget).
- Repeats: 3 independent runs per (task, strategy); total completed runs 291, evaluated cells 288 (gold-test correctness).
- Evaluation: SWE-bench Tier-C style — agent never sees gold tests; task considered passed iff all gold tests pass.
- Correctness results
- Pass-rates (aggregate):
- Claude (15 tasks): NONE 53.3% (24/45), ALWAYS ON 55.6% (25/45), SELECTIVE 55.6% (25/45).
- Codex (17 tasks): NONE 58.8% (30/51), ALWAYS ON 56.9% (29/51), SELECTIVE 52.9% (27/51).
- Omnibus permutation tests: no significant strategy effect (Claude p=1.00; Codex p=0.66).
- Equivalence (TOST): pairwise strategy differences bounded to <10pp (Claude) and <15pp (Codex) on the observed sample.
- Manipulation-validity probe: the real AGENTS.md never converted near-miss failures (1–4 failing tests) into full passes for either agent.
- Pass-rates (aggregate):
- Failure-mode triage
- Inspected near-miss failures; examples show failures driven by:
- subtle implementation bugs (miswiring an optimization),
- wrong architectural/pattern choice (reactive retry vs. proactive token refresh),
- mistaken behavioral checks (misapplied validators),
- complex type-system reasoning.
- None were gaps in repository-private factual knowledge.
- Inspected near-miss failures; examples show failures driven by:
- Agent-specific borderline difficulty
- Per-task pass-rates correlate across agents (Spearman ρ ≈ 0.75) but the informative/borderline band differs by agent.
- About ~40% of shared tasks were borderline/floor/ceiling in different ways across agents — explaining why single-agent studies may reach contradictory conclusions.
- Efficiency/process signals
- Largely null across strategies on portable metrics (tool calls, tokens, duration) for Codex.
- Claude: SELECTIVE reduced cache-creation tokens (mechanistic consequence of delivery mechanics), and on opshin context reduced blind full-suite pytest runs and wall-clock time (dose-dependent: NONE > ALWAYS ON > SELECTIVE). These are process effects, not correctness gains.
- Methodological lessons
- Power limits: with n≈17 tasks and 3 repeats, the study has low power — a 30pp true effect is detected only ~57% of the time; detecting a 10pp effect at 80% power would need ~120–200 tasks. More tasks (not more repeats) are needed to scale detection power.
- Metric portability: turn-count thresholds are nonportable (Codex emitted a single turn.completed event per session); prefer tool-call or other portable metrics across agent architectures.
Data & Methods
- Agents: Claude Code (sonnet-4-6) via system-prompt injection; Codex CLI (gpt-5.5) via user-prompt prepend.
- Independent variable: injection strategy (NONE, ALWAYS ON, SELECTIVE). Workspace AGENTS.md removed in all conditions.
- Tasks: mined from merged PRs; PR description = agent prompt; base commit = starting state; PR test files = gold evaluation. Selected/prescreened to include borderline tasks (screening used Codex initially).
- Evaluation: run agent, apply PR gold tests to agent output, pass iff all gold tests pass (SWE-bench Tier-C protocol) on an egress-locked, sanitized execution environment.
- Metrics: correctness (binary), portable efficiency metrics (tool calls, wall-clock, output tokens), agent-specific token accounting (Claude cache reads/creates).
- Statistical analysis:
- Unit of analysis: task (3 repeats averaged).
- Omnibus permutation test (labels permuted within-task).
- Paired Wilcoxon signed-rank tests (Holm-Bonferroni correction) for efficiency metrics.
- TOST on task-clustered bootstrap (10k) for equivalence bounds.
- Monte Carlo power simulation to compute MDE and required sample sizes.
- Reproducibility: code, data, and analysis released by the author.
Implications for AI Economics
- Low marginal ROI of authoring AGENTS.md for correctness: Organizations investing developer time to craft detailed AGENTS.md files should temper expectations that this will improve automated PR-level correctness. The measured effect on correctness is negligible within the tested bounds; small outcome gains (≤10–15 pp) are not supported by this study at current sample sizes.
- Where AGENTS.md may still be valuable
- Process efficiency: context can sometimes change agent behavior in ways that reduce wasted computation (e.g., avoiding blind full-suite runs when a repo warns tests are slow). That can reduce compute costs in specific repos/tasks, implying situational ROI for authoring high-value warnings or test-scope guidance.
- Non-correctness benefits: prior work (Lulla et al.) found token/time efficiency improvements; this study shows limited and repository-specific process signals. So AGENTS.md could be cost-saving on compute/token bills in some operational contexts even if correctness is unchanged.
- Resource allocation trade-offs
- Investments that improve agents’ implementation skills (model capabilities, fine-tuning on repo-level code patterns, agent tooling/harnesses, better retrieval/RAG pipelines capturing specific files) are more likely to raise correctness than adding static project-level context files.
- Given agent-specific informative bands, vendor/tool selection matters: a repository’s marginal gain from context depends on which agent is used. Firms should evaluate agent × task interactions before standardizing on authoring effort.
- Experimental design & auditing for product teams
- When auditing or A/B testing context interventions, screen tasks per agent and use substantially larger task pools to detect realistic effect sizes (expect to need O(100) tasks to detect ~10 pp effects).
- Be cautious about cross-study generalization: single-agent studies drawing different task mixes can yield contradictory conclusions; economic decisions based on a single-agent result risk misallocating effort.
- Policy and tooling recommendations
- Platform designers: auto-loading AGENTS.md as a universal intervention is unlikely to improve correctness broadly; invest instead in higher-bandwidth channels (retrieval, tool integration, bug-level examples) and in surfacing repo warnings that reduce wasted compute.
- Cost-accounting: quantify repo-specific compute savings from context-driven process changes (e.g., fewer full-suite runs) before large-scale efforts to standardize AGENTS.md authoring.
- Bottom line for decision-makers: prioritize investments that increase agent implementation competence and better retrieval/tooling over broad authoring of static AGENTS.md for the purpose of raising automated-coding correctness. Use small pilots to measure repo-specific process savings where AGENTS.md content is expected to influence behavior (e.g., slow test suites, explicit test-scoping guidance).
If you want, I can: - extract the numeric power curves or MDE tables for planning A/B tests; - draft a short checklist teams can use to decide whether to author AGENTS.md content for a given repo; or - produce a one-page recommendation memo for engineering leadership summarizing expected ROI and suggested experiments.
Assessment
Paper Typequasi_experimental
Evidence Strengthmedium — The study uses a careful, within-task paired design on real merged PRs with gold-test evaluation and appropriate statistical tools (permutation tests, equivalence testing, power analysis), which supports causal interpretation of null/limited effects of context injection; however, the number of task clusters is small (15–17), repeats are limited (n=3), there are known confounds (SELECTIVE corpus differs from AGENTS.md for two repos, injection-channel asymmetry between agents), and power is low for small-to-moderate effects, limiting the strength of inferences and generalizability.
Methods Rigormedium — Strengths: pre-specified strategies, gold-test pass/fail outcome, within-task pairing, use of permutation and equivalence testing, and Monte Carlo power analysis; also careful safety/isolation and reproducibility (release of code/data). Limitations: small number of repositories/tasks, screening that was calibrated on one agent (Codex) introducing agent-specific sample issues, confounded SELECTIVE corpus, injection-channel asymmetry (system prompt vs user-turn prepend), limited repeats, and some exploratory post-hoc analyses.
Sample17 Codex tasks and 15 Claude tasks drawn from merged pull requests in three Python repositories (pdm, firebase-admin-python, opshin); tasks selected to include medium/complex and borderline difficulty via a Codex-based screening sweep; each (task, strategy) run with 3 independent repeats yielding 291 completed runs and 288 evaluated gold-test cells (3 Claude runs excluded for crash/timeouts); AGENTS.md files ranged ~248–1236 words; runs executed in an egress-locked pod with gold PR test files held out for evaluation.
Themeshuman_ai_collab productivity
IdentificationWithin-task controlled ablation: each real PR task is run under three context-injection strategies (NONE, ALWAYS ON, SELECTIVE) with 3 independent repeats per cell and two different coding agents; within-task paired contrasts plus permutation tests, paired Wilcoxon for efficiency metrics, TOST equivalence testing (task-clustered bootstrap), and Monte Carlo power analysis are used to attribute differences to the injection strategy while holding task and workspace constant.
GeneralizabilityOnly three repositories (limits domain breadth) and Python-only tasks, Small number of task clusters (15–17) limits power to detect modest effects and external validity, SELECTIVE condition uses a different, larger wiki corpus for two repositories (confound between delivery channel and corpus content), Injection-channel asymmetry across agents (system prompt for Claude vs. user-prompt prepend for Codex) confounds cross-agent comparisons, Tasks were screened using Codex (so the borderline set may be biased toward Codex informative band), and two high-effort tasks were excluded from Claude due to budget, Egress-locked environment excludes tasks requiring external credentials/services so results may not generalize to real-world runs with networked dependencies, Findings apply to the tested agent versions/providers and may not transfer to other models or future agent architectures
Claims (10)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Repository-context injection strategy did not measurably change coding-task correctness for either Claude Code or Codex. Output Quality | null_result | Binary correctness based on whether all hidden gold tests passed |
Reading fidelity
high
Study strength
medium
|
n=288
Pairwise differences ≤2.3pp for Claude and ≤5.9pp for Codex
|
| The study's equivalence analysis bounded the correctness difference between context strategies to less than 10 percentage points for Claude and less than 15 percentage points for Codex. Output Quality | null_result | Difference in gold-test pass rates between context-injection strategies |
Reading fidelity
high
Study strength
medium
|
n=32
<10pp for Claude; <15pp for Codex
|
| On four Codex-borderline tasks, context injection reduced rather than increased the observed pass rate: NONE achieved 58% versus 42% for both ALWAYS ON and SELECTIVE. Output Quality | negative | Gold-test pass rate on borderline tasks |
Reading fidelity
high
Study strength
low
|
n=4
16 percentage-point lower pass rate for ALWAYS ON and SELECTIVE versus NONE
|
| For Claude, SELECTIVE retrieval used significantly fewer cache-creation tokens than NONE. Organizational Efficiency | positive | Cache-creation token usage |
Reading fidelity
high
Study strength
medium
|
n=11
11/11 tasks lower; p=0.001; pHolm=0.012
|
| Codex showed no meaningful differences across context strategies in tool calls, output tokens, or wall-clock duration. Developer Productivity | null_result | Tool calls, output-token count, and wall-clock duration |
Reading fidelity
high
Study strength
medium
|
n=17
Tool calls 32/32/32; output tokens ±3.8%; duration ±3.8%; all |dz|<0.2
|
| For Claude on the opshin repository, context was associated with approximately 24% lower wall-clock time, mainly by reducing blind full-suite test runs. Task Completion Time | positive | Wall-clock task duration and number of blind full-suite test invocations |
Reading fidelity
high
Study strength
low
|
n=5
∼24% lower duration; 2689s vs. 2066s/2032s
|
| Across the 15 tasks attempted by both agents, task-level pass rates were positively correlated between Claude and Codex, but borderline difficulty was agent-specific. Output Quality | mixed | Per-task gold-test pass rate and task difficulty band by agent |
Reading fidelity
high
Study strength
medium
|
n=15
Spearman ρ=0.75; Pearson r=0.77
|
| The paper's failure-mode triage indicates that near-miss failures were primarily caused by implementation skill problems rather than missing repository knowledge that context files could provide. Ai Safety And Ethics | negative | Failure mechanism in coding-agent task attempts |
Reading fidelity
high
Study strength
low
|
n=4
|
| In the manipulation-validity probe, the real AGENTS.md did not convert a near-miss failure into a pass for either agent. Output Quality | null_result | Conversion of near-miss coding-task failures into passing gold-test outcomes |
Reading fidelity
high
Study strength
low
|
n=2
|
| With 17 tasks and three repeats, the simulation estimated that even a 30-percentage-point effect would be detected only 57% of the time, and detecting a 10-percentage-point effect with 80% power would require approximately 120–200 tasks. Other | mixed | Statistical power to detect context-strategy effects on pass rates |
Reading fidelity
high
Study strength
high
|
n=17
57% detection probability for Δ=30pp; ∼120–200 tasks for 80% power at Δ=10pp
|