10 cumulative citations
View corpus contextA new GitHub-derived benchmark finds that when coding agents are given correctly retrieved prior cases they solve issues more accurately and cheaply, but noisy or mis-selected context can nullify or reverse those gains; the dataset (1,100 base + 376 related tasks across 51 repos and 9 languages) provides a standard to study context management in coding assistants.
SWE Context Bench: A Benchmark for Context Learning in Coding
SWE-ContextBench demonstrates that accurately retrieved and summarized prior coding cases substantially improve coding agents' resolution accuracy and reduce runtime and token costs—especially on harder problems—while irrelevant or incorrect context can offer little or negative benefit.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Large language models are increasingly used as coding agents for software engineering tasks. Current benchmarks mainly evaluate whether the agent can correctly solve the request or fix the bugs. They largely treat tasks as independent and do not assess whether agents can reuse previous experience across related problems. As a result, the efficiency gains from reusing the previous experience remains difficult to measure. We introduce SWE-ContextBench, a benchmark designed to explicitly evaluate context understanding and retrieval in coding agents. SWE-ContextBench consists of 1,100 base tasks with another 376 related tasks derived from real dependency and reference relationships among GitHub issues and pull requests. SWE-ContextBench groups base tasks and related tasks with shared context across 51 unique repositories and 9 programming languages. The benchmark evaluates how accurately and efficiently agents solve related issues when prior cases are available in context. Using SWE-ContextBench, we study the behavior of multiple coding agents across varying context reuse settings and retrieval strategies. Our results show that accurately summarized and retrieved previous experience can significantly improve resolution accuracy and reduce runtime and token cost, particularly on harder tasks. In contrast, unfiltered or incorrectly selected context provides limited or negative benefits. These findings highlight the importance of context management and retrieval accuracy, and position SWE-ContextBench as a principled benchmark for studying context learning in coding agents.
Summary
Main Finding
SWE-ContextBench is a new benchmark (1,476 tasks: 1,100 “experience” tasks + 376 related tasks) that explicitly measures whether coding agents can summarize, retrieve, and reuse prior problem-solving context across real GitHub issues/PRs. Accurate, compact summaries of prior experience materially improve task resolution accuracy and — importantly for AI economics — reduce token use, runtime, and API cost. By contrast, unfiltered or incorrectly retrieved context can give little benefit or be harmful.
Key Points
-
Benchmark scope
- 1,476 tasks from 51 real GitHub repos across 9 languages (Python dominant). A lightweight variant (SWE-ContextBench Lite) has 399 tasks (300 experience, 99 related) for lower-cost evaluation.
- Each related task includes pre-fix repo state (base_commit), an issue description, and ground-truth solution/test patches. Validation uses FAIL_TO_PASS (tests fixed) and PASS_TO_PASS (regressions avoided).
- Average related-task test coverage: 4.8 FAIL_TO_PASS tests and 211.5 PASS_TO_PASS tests.
-
Context representations
- Full recorded interaction trajectories (context pool) average ~25,634 tokens each.
- Compact summaries (final-summary paragraph of a trajectory) average ~217 tokens each.
- Context pool was generated by running an agent (Claude Sonnet 4.5) on the 1,100 experience tasks and saving full trajectories.
-
Evaluation dimensions (desiderata)
- Accuracy variation: how much performance changes with prior context available.
- Time efficiency: wall-clock/search time improvements from reusing context.
- Cost efficiency: token consumption and monetary/API cost savings from avoiding redundant reasoning.
-
Experiment design
- Five context-access settings: No-Context (baseline), Free Context Learning, Oracle Context, Free Summary Learning, Oracle Summary Learning.
- Five coding agents evaluated (closed- and open-weight): Claude Sonnet 4.5, GPT-5.3 Codex, Qwen3.5-plus, GLM-5, MiniMax 2.7.
- All related-task runs executed in isolated environments; context was provided only via explicit retrieval.
-
Quantitative outcomes (high-level)
- Oracle summaries (gold relevant summaries) raised resolution rates for most models; improvements vary by model (examples below).
- Summaries reduced token consumption and often reduced runtime and API cost (magnitude varies by agent).
- Examples from Table 3 (baseline → oracle summary):
- GPT-5.3 Codex resolved 22.60% → 23.94% (tokens: ~858k → ~719k; cost: $0.31 → $0.27).
- Claude Sonnet 4.5 resolved 19.68% → 23.40% (tokens: ~1.70M → ~1.35M; runtime: 344s → 281s; cost: $0.67 → $0.55).
- Qwen3.5-plus showed a large improvement in resolution (16.22% → 21.54) and a notable drop in reported cost ($0.33 → $0.18).
- Gains are larger when the retrieved context is accurate and compact; noisy/unfiltered retrieval gives limited or negative gains.
Data & Methods
-
Data construction
- Base instances drawn from SWE-Bench Lite / Multilingual / Verified (1,100 instances across 51 repos).
- Related tasks identified by manually analyzing PR/issue cross-references (six relationship types: multi-issue, PR→issue, PR→PR, issue→issue, issue→PR, multi-references) plus recursive expansion; 376 related tasks created and manually verified.
- Tasks created via GitHub API: base commit, issue text, solution patch and test patch extracted from PR diffs.
-
Evaluation pipeline
- For each related task: clone base_commit, run original test suite (results_before), optionally provide retrieved context, let agent generate implementation patch(s), apply patch(s), run tests (results_after).
- PASS_TO_PASS ensures no regressions; FAIL_TO_PASS verifies the intended bug was fixed.
-
Retrieval/Context settings
- Full trajectories vs compact final summaries; settings include oracle (give the correct prior context/summary) and free (agent may retrieve from the pool).
- Context pool created by executing an agent on each experience task and saving its full interaction.
-
Agents & metrics
- Metrics: localization accuracy (file/function/line), resolved (% tasks fixed), avg tool calls, avg tokens, avg runtime, API monetary cost.
- Experiments compare baseline (no context) vs oracle-summary and other context settings to isolate retrieval and summarization effects.
Implications for AI Economics
-
Direct cost savings from context reuse
- Compact, accurate summaries dramatically reduce token consumption compared to replaying full trajectories (~217 vs ~25,634 tokens). That translates into lower API usage and direct monetary savings per task.
- Reduced token usage often (but not always) correlates with lower runtime and fewer tool calls, lowering compute and operational costs.
-
ROI of building/maintaining memory systems
- There is an economic trade-off: building and storing a context pool (recording trajectories, generating summaries, maintaining retrieval indices) costs engineering time and storage; SWE-ContextBench quantifies the potential downstream savings in token/API spend and developer time to assess ROI.
- For organizations running frequent automated codefix/CI tasks, even modest per-task token savings can accrue large savings at scale.
-
Model-choice and procurement decisions
- Different models show distinct trade-offs: some (e.g., GPT-5.3 Codex) are most token-efficient and accurate; others benefit more from summaries (open-weight models sometimes gain larger relative improvement). Procurement should consider both baseline performance and sensitivity to context-augmentation when estimating total cost of ownership.
-
Product and pricing implications for LLM providers
- Providers can capture value by offering: compact-context encoding/summarization services, retrieval-as-a-service (accurate relevance ranking), and memory APIs that reduce downstream token consumption for customers.
- Pricing models that charge per token without accounting for value of compact summaries may incentivize customers to build third-party summarization/retrieval layers; providers could monetize integrated memory features.
-
Incentives for engineering investments
- Benchmarks like SWE-ContextBench create measurable KPIs (accuracy variation, time, token cost) that help justify investment in retrieval, summarization, and context-selection algorithms.
- Economic gains are largest when retrieval accuracy is high; hence investment should prioritize precise relevance scoring and concise representation rather than naive dumping of prior interactions.
-
Cautions & risks
- Misleading or incorrectly selected context can harm accuracy; this implies an additional cost/risk dimension (incorrect fixes, developer time to undo regressions).
- Organizations should account for potential negative externalities (debugging incorrect retrieved suggestions) when calculating net benefits.
Overall, SWE-ContextBench supplies a principled way to quantify the economic value of memory/retrieval and summarization strategies for coding agents, enabling better-informed cost-benefit analyses for both system builders and buyers of code-generation services.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper provides systematic empirical evidence from experiments on a large, purpose-built benchmark (1,100 base tasks + 376 related tasks) showing consistent patterns across multiple coding agents and retrieval strategies; however, results are confined to the constructed benchmark and experimental settings, so external validity and causal claims about real-world productivity gains are limited.
Methods Rigormedium — The authors assemble a sizable, multi-language dataset drawn from real GitHub issues/PRs, test multiple agents and retrieval strategies, and measure accuracy, runtime, and token costs, indicating careful experimental design; nonetheless, potential selection biases in repository/task choice, limited details on agent/configuration/hyperparameter variation, and lack of field-validation or human-in-the-loop evaluation reduce methodological rigor.
SampleSWE-ContextBench: 1,100 base coding tasks plus 376 related tasks derived from dependency and reference relationships in real GitHub issues and pull requests; grouped into shared-context sets across 51 repositories and covering 9 programming languages; experiments compare multiple coding agents under varying context-reuse and retrieval strategies, measuring resolution accuracy, runtime, and token usage.
Themesproductivity human_ai_collab
GeneralizabilityTasks drawn from 51 GitHub repositories may not represent the full diversity of codebases or enterprise settings, Nine programming languages cover common cases but omit many ecosystems and domain-specific languages, Benchmarked LLM agents and configurations may change rapidly, limiting temporal generalizability, Derived related tasks reflect issue/PR linkage heuristics that may not capture all real-world context-dependency patterns, Performance measured on automated benchmarks may not translate directly to human developer productivity or team workflows
Claims (9)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| SWE-ContextBench consists of 1,100 base tasks with another 376 related tasks derived from real dependency and reference relationships among GitHub issues and pull requests. Other | null_result | number of tasks in benchmark (base and related) |
Reading fidelity
high
Study strength
high
|
n=1476
|
| SWE-ContextBench groups base tasks and related tasks with shared context across 51 unique repositories and 9 programming languages. Other | null_result | number of repositories and programming languages represented |
Reading fidelity
high
Study strength
high
|
n=51
|
| Current benchmarks mainly evaluate whether the agent can correctly solve the request or fix the bugs and largely treat tasks as independent, not assessing whether agents can reuse previous experience across related problems. Other | null_result | scope and focus of existing coding-agent benchmarks (independence of tasks, evaluation criteria) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| The benchmark evaluates how accurately and efficiently agents solve related issues when prior cases are available in context. Other | null_result | accuracy and efficiency of agents solving related issues with prior context |
Reading fidelity
high
Study strength
high
|
not reported
|
| Using SWE-ContextBench, we study the behavior of multiple coding agents across varying context reuse settings and retrieval strategies. Other | null_result | agent behavior under different context reuse settings and retrieval strategies |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Accurately summarized and retrieved previous experience can significantly improve resolution accuracy. Output Quality | positive | resolution accuracy |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Accurately summarized and retrieved previous experience can reduce runtime and token cost, particularly on harder tasks. Task Completion Time | positive | runtime and token cost (resource efficiency), with emphasis on harder tasks |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Unfiltered or incorrectly selected context provides limited or negative benefits. Output Quality | negative | benefit to agent performance (accuracy/efficiency) when using unfiltered or incorrect context |
Reading fidelity
high
Study strength
medium
|
not reported
|
| These findings highlight the importance of context management and retrieval accuracy, and position SWE-ContextBench as a principled benchmark for studying context learning in coding agents. Other | positive | importance of context management and retrieval accuracy; suitability of SWE-ContextBench as a research benchmark |
Reading fidelity
high
Study strength
speculative
|
not reported
|