8 cumulative citations
View corpus contextReplicating team roles and workflow in an autonomous multi-agent system lifts automated issue resolution to 72.2% on a 500-task benchmark, outperforming single-agent LLM approaches; organizational design and structured communication appear as important as model improvements.
Agyn: A Multi-Agent System for Team-Based Autonomous Software Engineering
A fully automated multi-agent system that models software engineering team roles and workflows resolves 72.2% of SWE-bench 500 tasks, outperforming comparable single-agent LLM baselines.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Large language models have demonstrated strong capabilities in individual software engineering tasks, yet most autonomous systems still treat issue resolution as a monolithic or pipeline-based process. In contrast, real-world software development is organized as a collaborative activity carried out by teams following shared methodologies, with clear role separation, communication, and review. In this work, we present a fully automated multi-agent system that explicitly models software engineering as an organizational process, replicating the structure of an engineering team. Built on top of agyn, an open-source platform for configuring agent teams, our system assigns specialized agents to roles such as coordination, research, implementation, and review, provides them with isolated sandboxes for experimentation, and enables structured communication. The system follows a defined development methodology for working on issues, including analysis, task specification, pull request creation, and iterative review, and operates without any human intervention. Importantly, the system was designed for real production use and was not tuned for SWE-bench. When evaluated post hoc on SWE-bench 500, it resolves 72.2% of tasks, outperforming single-agent baselines using comparable language models. Our results suggest that replicating team structure, methodology, and communication is a powerful paradigm for autonomous software engineering, and that future progress may depend as much on organizational design and agent infrastructure as on model improvements.
Summary
Main Finding
Modeling autonomous software engineering as an explicit, team-like organizational process—using a manager-mediated multi-agent system with role specialization, isolated sandboxes, and GitHub-native workflows—yields strong real-world performance. A production-deployed AGYN-based system (manager, researcher, engineer, reviewer) resolved 72.2% of SWE-bench 500 tasks fully automatically, outperforming a comparable single-agent baseline (mini-SWE-agent) by 7.4%, despite not being tuned for the benchmark.
Key Points
- Organizational decomposition: Software engineering is treated as a coordinated team activity (roles: manager, researcher, engineer, reviewer) rather than a monolithic single-agent task or a fixed pipeline.
- Role specialization and model allocation:
- Reasoning-heavy roles (manager, researcher) use larger, general-purpose models (GPT-5).
- Implementation roles (engineer) use smaller, code-specialized models (GPT-5-Codex) to reduce cost and accelerate iterative edits.
- Isolated execution environments: Each agent has its own sandbox with shell access and package control (Nix), enabling local experiments and avoiding cross-agent contamination.
- Manager-centric coordination: All inter-agent communication is mediated by a manager via a "manage" tool; this improves traceability and aligns with a project-lead style.
- GitHub-native workflow and tooling:
- Separate GitHub identities per agent to mirror distinct contributors.
- Use of gh CLI for compact repository interaction and a custom gh-pr-review extension to read/author inline reviews.
- System performs actions via real pull requests and inline reviews, providing a concrete approve/request-changes acceptance signal.
- Test-driven implementation: Engineer runs tests before and after changes; iterative PRs and reviewer feedback drive convergence.
- Automation design: System purposefully avoids human-style dialog behaviors (e.g., asking for permission) to enable fully autonomous progress.
- Production-first, benchmark-agnostic: The system was designed for real engineering workflows and not tuned for SWE-bench; SWE-bench 500 was used as a post hoc evaluation.
- Open artifacts and platform: AGYN platform is open-source; authors provide forks, opened issues/PRs, and traces to support transparency.
Data & Methods
- Platform: AGYN — open-source multi-agent orchestration platform supporting agent roles, isolated workspaces, tool access, and LLM call tracing.
- Team configuration: Four agent roles configured with role-specific prompts, tools, and execution sandboxes.
- Model configuration: Heterogeneous model assignment (larger models for reasoning and repository understanding; smaller code-focused models for editing/debugging).
- Execution environments:
- Minimal pre-provisioning; agents construct environments using Nix to match repo constraints.
- Large command outputs redirected to files when exceeding thresholds to avoid bloated LLM context.
- Coordination pattern:
- Manager mediates all calls to other agents; number and length of interactions are dynamic (not a fixed pipeline).
- Agents do not share a global prompt or context; outputs are passed via manager tooling.
- GitHub integration:
- Actions performed through gh CLI to reduce metadata bloat.
- Custom gh-pr-review tool for inline reviews and comments.
- Agents act under distinct GitHub accounts and perform PR creation, review, and approval.
- Evaluation:
- Post hoc automated evaluation on SWE-bench 500.
- Success metric: repository test-suite passing after PR merge (i.e., issue resolution).
- Result: 72.2% of tasks resolved; +7.4 percentage points vs mini-SWE-agent baseline under comparable model setups.
- Practical design choices noted:
- Avoided heavy tuning for benchmark behavior.
- Addressed pre-existing test failures as a practical complication (can cause scope drift).
- Chose manager-centric design for traceability and simpler control.
Implications for AI Economics
- Cost-performance tradeoffs via specialization:
- Role-specific model allocation demonstrates that mixing larger reasoning models with smaller code models can increase throughput and reduce average compute cost per resolved issue. This suggests firms can optimize total cost by investing in orchestration and role design rather than only scaling single models.
- Value of organizational design and infrastructure:
- Economic value is captured not only by raw LLM improvements but also by platform and workflow engineering (orchestration, tooling, sandboxes). Investment in agent orchestration platforms (like AGYN), GitHub-native tooling, and developer-like processes can produce outsized productivity gains.
- New market opportunities:
- Commercial opportunities arise for vendors providing agent orchestration, role-definition frameworks, repo-native integrations, and review automation. Buyers may prefer turnkey multi-agent orchestration over raw LLM access.
- Labor and productivity implications:
- Autonomous multi-agent teams can reduce marginal cost and time of routine fixes and small issues, potentially increasing developer productivity. This creates pressure on cost structures for maintenance work but likely increases demand for higher-level oversight, platform engineering, and governance roles.
- Incentives for firms and benchmarking:
- Because organizational design materially affects performance, firms have an economic incentive to optimize workflows, not just model size. Benchmarks that ignore organizational factors can misprice the value of infrastructure investments.
- Externalities and governance costs:
- Deploying autonomous multi-agent teams entails governance, audit, and maintenance costs (credential management, monitoring agent behavior, handling regressions). These overheads should be counted in ROI calculations.
- Measurement and contracting:
- Performance metrics may shift from token-based throughput to outcome-oriented measures (e.g., issues resolved, regression rates, time-to-merge), changing how AI services are priced and contracted.
- Distribution of value capture:
- Providers of orchestration platforms, repo-native tooling, and model-specialization stacks may capture significant value even if LLM providers continue to supply base models—analogous to how DevOps and CI/CD tooling capture value around compilers and runtimes.
- Risk-adjusted deployment costs:
- The need for traceability and review loops (implemented here via reviewer agents and PR workflows) raises the effective cost of safe deployment but also increases trustworthiness—affecting adoption decisions and insurability of automated development.
Overall, the paper implies that economic gains from autonomous software engineering will depend heavily on organizational design, tooling, and orchestration—areas where investments can yield large marginal benefits relative to simply scaling models.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper presents quantitative evaluation on a large benchmark (SWE-bench 500) with a clear performance improvement (72.2% task resolution) relative to single-agent baselines, which provides direct empirical evidence that multi-agent organizational structure can improve automated software engineering outcomes. However, the evidence is limited to a single benchmark and post hoc evaluation, with unclear external validity, potential benchmark-specific tuning, and limited counterfactual or deployment validation that would be required for stronger causal claims.
Methods Rigormedium — The study uses a sizable benchmark and compares against single-agent baselines using comparable language models, and the system was evaluated without human intervention. But the abstract suggests missing details that would raise rigor: potential absence of randomized or real-world deployment experiments, limited transparency about model versions, hyperparameters, ablation studies, sensitivity analyses, and error analysis across task types; these limit confidence in robustness and mechanisms.
SampleEvaluation on SWE-bench 500 (500 software engineering tasks/issues) using the authors' fully automated multi-agent system implemented on agyn; comparisons made to single-agent baselines using comparable large language models; system operates end-to-end without human intervention and was reportedly designed for production use (not tuned specifically for SWE-bench).
Themesproductivity human_ai_collab org_design adoption
GeneralizabilityResults are from a single benchmark (SWE-bench 500) and may not generalize to diverse, large-scale, or proprietary codebases., Unclear how performance transfers across programming languages, repository styles, and development workflows., Performance likely depends on the underlying LLM(s) and compute/configuration; different models or resource limits may change outcomes., Post hoc benchmark evaluation (rather than randomized field trials) may overstate real-world effectiveness in noisy team environments., Unclear robustness to adversarial, ambiguous, or highly novel issues that appear in real production settings.
Claims (10)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Large language models have demonstrated strong capabilities in individual software engineering tasks. Developer Productivity | positive | capability on individual software engineering tasks |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Most autonomous systems still treat issue resolution as a monolithic or pipeline-based process. Other | negative | architecture/style of autonomous systems for issue resolution |
Reading fidelity
medium
Study strength
low
|
not reported
|
| Real-world software development is organized as a collaborative activity carried out by teams following shared methodologies, with clear role separation, communication, and review. Team Performance | positive | organizational structure and practices of software development |
Reading fidelity
high
Study strength
low
|
not reported
|
| We present a fully automated multi-agent system that explicitly models software engineering as an organizational process, replicating the structure of an engineering team. Other | positive | existence and design of an automated multi-agent system |
Reading fidelity
high
Study strength
medium
|
not reported
|
| The system is built on top of agyn, an open-source platform for configuring agent teams, and assigns specialized agents to roles such as coordination, research, implementation, and review, provides them with isolated sandboxes for experimentation, and enables structured communication. Other | positive | system architecture and features (agent roles, sandboxes, communication) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| The system follows a defined development methodology for working on issues, including analysis, task specification, pull request creation, and iterative review, and operates without any human intervention. Task Allocation | positive | ability to execute end-to-end software development workflow autonomously |
Reading fidelity
high
Study strength
medium
|
not reported
|
| The system was designed for real production use and was not tuned for SWE-bench. Other | neutral | design intent / tuning practices |
Reading fidelity
medium
Study strength
low
|
not reported
|
| When evaluated post hoc on SWE-bench 500, it resolves 72.2% of tasks. Developer Productivity | positive | fraction of tasks resolved |
Reading fidelity
high
Study strength
high
|
n=500
72.2% of tasks resolved
|
| The system outperforms single-agent baselines using comparable language models. Developer Productivity | positive | performance relative to single-agent baselines on same benchmark |
Reading fidelity
high
Study strength
medium
|
n=500
|
| Replicating team structure, methodology, and communication is a powerful paradigm for autonomous software engineering, and future progress may depend as much on organizational design and agent infrastructure as on model improvements. Organizational Efficiency | positive | importance of organizational design and agent infrastructure for autonomous software engineering performance |
Reading fidelity
high
Study strength
speculative
|
not reported
|