3 cumulative citations
View corpus contextA new executable benchmark finds state-of-the-art LLM agents often fail real-world backend engineering tasks; across 224 repository-level problems spanning 8 languages and 19 frameworks, agents struggle to configure, containerize and pass end‑to‑end API tests, revealing limits to their readiness for practical backend work.
ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development
ABC-Bench is an executable benchmark of 224 backend engineering tasks that shows current agentic LLMs frequently fail to complete end-to-end repository-level development, deployment, and API tests, exposing a gap between model capabilities and real-world backend engineering requirements.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
The evolution of Large Language Models (LLMs) into autonomous agents has expanded the scope of AI coding from localized code generation to complex, repository-level, and execution-driven problem solving. However, current benchmarks predominantly evaluate code logic in static contexts, neglecting the dynamic, full-process requirements of real-world engineering, particularly in backend development which demands rigorous environment configuration and service deployment. To address this gap, we introduce ABC-Bench, a benchmark explicitly designed to evaluate agentic backend coding within a realistic, executable workflow. Using a scalable automated pipeline, we curated 224 practical tasks spanning 8 languages and 19 frameworks from open-source repositories. Distinct from previous evaluations, ABC-Bench require the agents to manage the entire development lifecycle from repository exploration to instantiating containerized services and pass the external end-to-end API tests. Our extensive evaluation reveals that even state-of-the-art models struggle to deliver reliable performance on these holistic tasks, highlighting a substantial disparity between current model capabilities and the demands of practical backend engineering. Our code is available at https://github.com/OpenMOSS/ABC-Bench.
Summary
Main Finding
ABC-Bench demonstrates that current LLM-based agents—even top proprietary models—struggle to reliably complete full-lifecycle backend engineering tasks that require repository exploration, environment configuration, containerized deployment, and external end-to-end API validation. Shortcomings are concentrated in environment configuration and deployment rather than pure code logic, implying that practical automation of backend engineering remains partial and system-dependent.
Key Points
- What ABC-Bench is
- A realistic, execution-driven benchmark of 224 full-lifecycle backend tasks drawn from open-source repositories.
- Covers 8 programming languages and 19 web frameworks; 92 tasks explicitly require autonomous environment configuration and container startup.
- Task construction and verification
- ABC-Pipeline: automated three-phase workflow (Repository Exploration → Environment Synthesis → Task Instantiation).
- Each task packaged with masked repository, instructions, generated Docker/environment files (removed for env-challenge tasks), and API-level verification tests.
- Two-stage verification ensures ground-truth builds/tests pass and masked repos cause tests to fail.
- Evaluation protocol
- Agents run inside an isolated outer container with full autonomy to edit code, install deps, and provide Dockerfiles; solutions are built and launched in a separate inner container.
- Success judged only by external API-level integration tests against the deployed service.
- Models, agent frameworks, and metrics
- Evaluated a mix of open-source and proprietary models (e.g., Qwen3 variants, DeepSeek-V3.2, GLM 4.7, Nex-N1, GPT-5, Gemini, Claude Sonnet 4.5).
- Default agent framework: OpenHands (also tested other frameworks).
- Metric: pass@1 (three independent runs per task); decomposition into Build stage (S1) and Functional execution given build success (S2).
- Main empirical results (high-level)
- Overall pass@1 range: top proprietary Claude Sonnet 4.5 = 63.2%; GPT-5 ≈ 49.4%; top open-source ~50.1% (DeepSeek-V3.2); many smaller models < 10% (e.g., Qwen3-8B ≈ 8.3%).
- Environment configuration is the primary bottleneck: many models that score high on functional logic (S2 > 80%) fail at S1 (build/start) < 50%.
- Language heterogeneity: performance varies substantially by language/stack; Rust tasks were especially challenging (many models score ~0% on Rust).
- Interaction depth strongly correlates with success (r = 0.87): higher-performing systems tend to run longer, iterative debugging traces.
- Agent framework matters: choice of framework substantially affects final performance (OpenHands outperforms some lightweight frameworks).
Data & Methods
- Dataset construction
- Source pool: 2,000 MIT-licensed open-source backend repositories.
- ABC-Pipeline produced ~600 candidate tasks; final curated set = 224 tasks balanced across languages/frameworks.
- 92 tasks require agents to synthesize environment/container files (no pre-provided Dockerfile).
- Task makeup
- 224 tasks total; diverse domains (analytics, search, commerce, payments, dev tooling).
- 8 languages: e.g., Python, Go, JavaScript, Java, C#, PHP, Ruby, Rust.
- 19 frameworks: FastAPI, Flask, Express, Spring Boot, Rails, Laravel, Gin, etc.
- Evaluation setup
- Agent runs in isolated container with CLI access; allowed to modify repository, install dependencies, and craft Dockerfiles.
- After agent finishes (or interaction budget exhausted), the system attempts to build a Docker image from the agent’s output and launches the inner container.
- External HTTP(s) integration tests (generated by pipeline) are executed to determine pass/fail.
- Models and configurations
- Models: a mix of open-source (Qwen3, Nex, DeepSeek, GLM) and proprietary (GPT-5, Gemini 2.5 Pro, Claude Sonnet 4.5).
- Default agent framework: OpenHands; also tested Claude Code and mini-SWE-agent for ablation.
- Repeated three runs per task; typical sampling temperatures: 0.7 (standard) and 1.0 (reasoning-enhanced).
- Key quantitative findings
- Overall pass@1 examples: Claude Sonnet 4.5 = 63.2%; DeepSeek-V3.2 = 50.1%; Qwen3-Coder-480B ≈ 43.1%; Qwen3-8B ≈ 8.3%.
- Environment build vs. functional success: Claude S1 ≈ 78% and S2 ≈ 80%; GPT-5 and DeepSeek show high S2 (>80%) but low S1 (<50%).
- Correlation agent turns ↔ pass@1: r = 0.87 (longer interaction trajectories correlate with higher success).
Implications for AI Economics
- Productivity gains are conditional and partial
- Agents are effective at generating correct business logic when they can run in a prepared runtime (high S2 for many models), suggesting potential productivity gains in code authoring and routine logic tasks.
- However, pervasive failures in environment configuration and deployment (S1) limit end-to-end automation value: real productivity gains require solving the deployment/DevOps bottleneck, not just code synthesis.
- Complementarity and skill-biased impacts
- Models perform unevenly across language stacks; high difficulty on some stacks (e.g., Rust) implies persistent wage premia for specialists in those areas. Skills in environment provisioning, DevOps, and debugging remain complementary to models and more valuable than pure implementation skills.
- The need for long-horizon, iterative interaction implies human/agent collaboration patterns rather than wholesale task replacement for complex backend work.
- Market and investment signals
- High variance across agent frameworks and system integration means productization and deployment tooling (platforms that reliably convert model outputs into working containers) are high-leverage investment targets. Returns may accrue to companies that integrate models with robust DevOps automation rather than to model vendors alone.
- Proprietary models showed notable lead (e.g., Claude Sonnet 4.5). This suggests commercial advantage—and pricing power—for high-performing models in enterprise contexts where full-lifecycle reliability matters.
- Cost structure & adoption thresholds
- Economic value of deploying such agents depends on probability of full-lifecycle success (pass@1), expected cost of human oversight/fixes, and costs of failed deployments (downtime, security, testing). Given current success rates, the marginal cost of supervision is non-trivial and must be included in ROI calculations.
- For firms, adoption is likely through hybrid processes: agents handle code logic generation and routine edits, humans manage provisioning, complex configs, and final validation—implying labor redeployment rather than large-scale layoffs in the near term.
- Sectoral and task-level displacement risk
- Tasks confined to logic writing in well-configured environments are more automatable; tasks requiring environment discovery, dependency resolution, and system orchestration are less automatable and thus less at-risk. This creates a segmented labor market: automation for some developer roles (e.g., feature scaffolding) and sustained demand for DevOps/infra engineers.
- Policy and standards implications
- Benchmarks like ABC-Bench provide crucial, economically-relevant metrics (end-to-end success rates under deployment conditions) that should inform procurement, regulatory validation, and risk assessment. Policymakers and enterprise buyers should prefer measures that account for deployment reliability, not just code-generation benchmarks.
- Suggested analyses for economists and decision-makers
- Cost-benefit modeling using pass@1 and S1/S2 decomposition to estimate net productivity gains after human supervision and failure costs.
- Scenario analysis on adoption thresholds: compute the break-even pass@1 where agents reduce total labor costs net of oversight.
- Examine returns to investing in DevOps automation platforms that raise S1 success—such investments may yield outsized productivity gains relative to pure model improvements.
- Monitor language/stack distribution to forecast which developer skill premiums will persist or grow.
Suggested actionable metrics to accompany future economic studies - End-to-end pass rate (deployment + API test pass) per task distribution. - Conditional success rates: S1 (build/start) and S2 (functional given S1). - Average human-in-the-loop intervention time/cost per failed deployment. - Variance by stack/language to estimate heterogeneity in displacement risk.
Overall, ABC-Bench highlights that while LLMs can materially improve parts of software engineering, full economic gains require solving system-integration and deployment reliability problems. Investment and policy focus on tooling, deployment automation, and evaluation standards will be key to realizing the labor- and capital-market effects of agentic coding systems.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper provides extensive empirical evaluation using a sizable, executable benchmark (224 tasks) and end-to-end automated testing, which gives credible evidence about current model capabilities; however, it does not establish causal effects on economic outcomes or worker productivity and may be sensitive to task selection, environment/configuration choices, and specific model/tooling versions.
Methods Rigormedium — The authors build a scalable automated pipeline, curate realistic tasks from open-source repositories across multiple languages/frameworks, require containerized execution and external API tests, and release code; nonetheless, potential limitations include task selection bias, possible dependencies on specific environment/tooling setups, limited detail on evaluated model versions/configuration in the summary, and the inherent brittleness of end-to-end execution that may conflate model shortcomings with infrastructure/setup fragility.
Sample224 practical backend engineering tasks curated from open-source repositories, covering 8 programming languages and 19 frameworks; tasks require repository exploration, environment configuration, containerized service instantiation, and passing external end-to-end API tests; evaluated using an automated pipeline against several state-of-the-art agentic LLMs (code and evaluation pipeline released on GitHub).
Themesproductivity human_ai_collab
GeneralizabilityTasks drawn from open-source repos may not represent proprietary, large-scale production backend systems, Coverage limited to 8 languages and 19 frameworks — other stacks and niche frameworks not evaluated, Evaluation dependent on the specific environment, containerization, and tooling choices which may advantage or disadvantage certain agents, Performance may vary with different model versions, tool integrations, or increased compute/time allowances, End-to-end tests capture functional correctness but may not capture code quality, maintainability, or team workflow integration
Claims (6)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| We curated 224 practical tasks spanning 8 languages and 19 frameworks from open-source repositories. Other | positive | dataset size / task coverage |
Reading fidelity
high
Study strength
high
|
n=224
224 tasks
|
| ABC-Bench is explicitly designed to evaluate agentic backend coding within a realistic, executable workflow requiring agents to manage the entire development lifecycle, including repository exploration, instantiating containerized services, and passing external end-to-end API tests. Output Quality | positive | ability to complete end-to-end executable workflows / pass external API tests |
Reading fidelity
high
Study strength
medium
|
n=224
|
| Our extensive evaluation reveals that even state-of-the-art models struggle to deliver reliable performance on these holistic backend engineering tasks, indicating a substantial disparity between current model capabilities and practical backend engineering demands. Output Quality | negative | model performance / task success rate / reliability on end-to-end tasks |
Reading fidelity
high
Study strength
medium
|
n=224
|
| Current benchmarks predominantly evaluate code logic in static contexts and neglect the dynamic, full-process requirements of real-world engineering, particularly for backend development. Other | negative | scope/coverage of existing benchmarking practices |
Reading fidelity
medium
Study strength
medium
|
not reported
|
| Backend development demands rigorous environment configuration and service deployment, which static code-logic benchmarks fail to evaluate. Other | negative | coverage of environment and deployment requirements in benchmarks |
Reading fidelity
high
Study strength
low
|
not reported
|
| The authors' code and benchmark materials are available at https://github.com/OpenMOSS/ABC-Bench. Other | positive | availability / openness of code and benchmark |
Reading fidelity
high
Study strength
high
|
not reported
|