← Papers

How agent teams are organized matters as much as which model is used: in a deployment‑grounded benchmark, coordination topology altered scores by 30+ points and could double delivery time, with structured pipelines offering the best speed–quality trade‑off and managerial‑heavy modes underperforming.

An Empirical Study of Coordination Mode as the First-Class Citizen in From-Scratch Multi-Agent Coding

Yanyu Ren, Yunfeng Bai, Xizheng Wang, Li Chen, Dan Li · July 30, 2026

arxiv quasi_experimental medium evidence 8/10 relevance Full text usable extracted full text Source PDF

Structured author observations

Linked only from stored provider relations; the raw author line above is never matched by name.

Arxiv

Latest observation: August 4, 2026

Yanyu Ren unresolved corpus identity
Yunfeng Bai unresolved corpus identity
Xizheng Wang unresolved corpus identity
Li Chen unresolved corpus identity
Dan Li unresolved corpus identity

Semantic Scholar

Latest observation: August 25, 2026

Yanyu Ren provider ID
Yu Bai provider ID
Xizheng Wang provider ID
Li Chen provider ID
Dan Li provider ID

In a controlled benchmark across 10 full‑stack projects and multiple LLMs, collaboration topology substantially alters multi‑agent coding outcomes — structured pipelines converge fastest with the highest quality, while heavy managerial oversight harms performance, with topology choices shifting functional scores by over 30 points and sometimes doubling wall‑clock time.

Citation observations

Cumulative provider counts captured on specific dates; providers are never combined.

0 cumulative citations

Semantic Scholar · Observed August 25, 2026

View corpus context

Multi-agent vibe coding promises to accelerate software development, yet existing benchmarks rely on synthetic environments that ignore practical time and monetary costs, conflate reasoning with communication, and reward only superficial completion. We introduce multi-agent from-scratch evaluation benchmark, MSEval, evaluating multi-agent coding on real-world tasks. Grounded in 10 authentic, full-stack projects across 10 domains, MSEval scores performance using hierarchical requirements and deterministic rubrics. Its execution engine, LegoGent, tests 10 collaboration topologies where agents coordinate via periodic sync intervals and deploy through native CI/CD pipelines. Concurrently, the automated grader TAgent dynamically probes implementations to jointly measure functional success, latency, and prefix-cached token cost. Across 100 runs, MSEval reveals that organizational topology rivals model capability in shaping the speed--cost--quality trade-off. For identical tasks and models, varying the topology shifts scores by over 30 points and doubles wall-clock time. Structured pipelines converge fastest with the highest quality, whereas heavy managerial oversight degrades performance. Ultimately, MSEval establishes a rigorous, reproducible standard for measuring how multi-agent teams actually build software. The benchmark is released at https://github.com/robinren03/MSEval.

Summary

Main Finding

Coordination topology matters as much as model capability for multi-agent, from-scratch coding: the way agents are organized (pipeline, swarm, PM-led, etc.) can shift functional scores by >30 points and double wall-clock time for the same task and model. MSEval — a deployment‑grounded benchmark (LegoGent runtime + TAgent grader) — quantifies speed–cost–quality trade-offs across 10 real full‑stack projects and 10 collaboration modes, showing that structured pipelines converge fastest with highest quality while heavy managerial oversight or ill‑bounded parallelism often increases time and token cost without improving outcomes.

Key Points

Benchmark: MSEval — 100-case grid (10 realistic capstone projects × 10 coordination modes). Code and artifacts released.
New tooling:
- LegoGent: activation-driven multi‑agent runtime with bounded periodic sync (every 4 minutes), explicit peer mailbox, CI/CD deployment (GitLab, SonarQube), artifact validators, and mode templates (ownership map, activation schedule, decision rule, handoff artifact).
- TAgent: automated “TA” grader that parses requirement docs into weighted rubrics, discovers UI/API/code surfaces, runs Playwright-based UI probes, reconstructs APIs (OpenAPI/Swagger fallback), inspects code, and returns item-level repair agendas.
Metrics: Functional Completion Score (0–100), best-round wall‑clock time, USD cost (provider token-based pricing), and raw token usage (including prefix-cached reads). Each run allowed up to three refinement rounds.
Collaboration modes evaluated include: feature squads, layer specialists, pipeline handoff, swarming, rotation, PM oversight, QA-first, PR-style/OSS review, adversarial testing, and competitive teams.
Empirical highlights:
- Topology can change scores by >30 points for the same model/task and can double wall-clock time.
- Structured pipeline modes often achieve fastest convergence and highest final quality.
- Heavy managerial oversight (PM-style) frequently increases latency/cost and can reduce net performance.
- Parallelism is beneficial only when side‑effects are isolated; otherwise it causes duplicate work, merge conflicts, and costly repairs.
- Example model trade-offs: Claude Opus 4.8 reached score 97 (110 min, ~$654) while GPT‑5.5 achieved nearly equivalent score close to 96 in 74 min for ~$138 — illustrating model × cost × topology frontiers.
Evaluation is deployment-grounded (builds and serves real endpoints) and iteratively scored (up to 3 rounds) rather than single-pass text inspection.

Data & Methods

Data: 10 authentic university capstone-style full‑stack projects (2000+ lines Python typical) across domains: messaging, asset management, crowdsourcing, requirement tracking, image processing, e-commerce, analytics, RBAC, news search, live teaching.
Experimental grid: 10 projects × 10 collaboration modes → 100 runs per model family; multiple LLMs tested (Claude Opus 4.8, GPT‑5.5, DeepSeek variants, GLM‑5.2; Qwen3.6‑Flash failed to produce results and was omitted).
Agents: teams of 3–4 agents per run, isolated processes with shared workspace guarded by runtime validators.
Runtime: LegoGent uses a periodic sync loop + active mailbox; enforces validated handoffs before deployment; mode templates standardize ownership and handoff artifacts.
Evaluation: TAgent constructs a deterministic rubric per project (30–45 line items normalized to 100 points). It performs:
- UI checks with Playwright and dependency-aware scheduling across browser workers.
- API checks through OpenAPI discovery or code reconstruction.
- Code checks via targeted search and LLM inspection.
- Produces item-level evidence and prioritized repair agendas for subsequent rounds.
Measurement: records per-run functional score (0–100), wall-clock minutes to best round, USD cost computed from token-price tables, and token counts. Hardware: runs on Intel Xeon Gold 6430; isolated ports for parallel evaluations.
Key methodological design choices: only collaboration mode varies per controlled run (same requirement doc, CI/CD path, rubric, and round budget) to isolate topology effects.

Implications for AI Economics

Coordination is an economic variable: organizational design (topology) materially affects compute/token spend, wall-clock time, and delivered value. Treat coordination mode as a first‑order decision when budgeting LLM-enabled software delivery.
Cost–quality trade-offs:
- High‑capability models can buy time savings or higher quality but at steep token/USD cost; lower-cost models + better topology can approach similar functional outcomes at much lower expense.
- Pipeline/structured modes often offer higher ROI when correctness and clean handoffs matter; swarming/parallel modes may save latency only if side effects are isolated.
Managerial overhead has measurable cost: PM-style or heavy oversight can introduce latency and token overhead that reduces throughput and raises marginal cost per delivered feature.
Measurement and procurement: Procurement and product teams should evaluate model+topology combinations, not models alone. Benchmarks like MSEval enable system-level cost-effectiveness comparisons that include deployment and iteration overheads.
Incentives for tooling and marketplaces:
- There is market value in runtimes (like LegoGent) and evaluators (like TAgent) that reduce redundant setup, minimize merge friction, and produce actionable repair guidance — these tools lower marginal cost of agentic development.
- Token-caching, request scheduling, and evaluator-side dependency optimization (as used by TAgent) are direct levers to reduce wasted compute and USD costs.
Research directions: build economic models that jointly optimize model choice, coordination topology, and task allocation for a target cost or delivery time; extend pricing and billing models to incorporate organizational overheads (synchronization latency, handoff repair) in addition to raw token consumption.
Policy/practice: teams deploying LLM-based developer agents should (a) run small experiments varying topology, (b) prefer structured handoffs for complex, highly-interdependent projects, and (c) instrument token and wall‑clock costs to inform operational choices.

If you want, I can: (a) extract a short cost-vs-quality decision rule for choosing a topology given a dollar/time budget, or (b) produce a one‑page checklist for teams to run their own MSEval-style experiments. Which would help you most?

Assessment

Paper Typequasi_experimental Evidence Strengthmedium — The paper provides a well-engineered, deployment-grounded benchmark with 10 realistic projects, 10 collaboration modes, multiple commercial/academic LLMs, and repeated runs that produce rich quality/time/cost metrics; however, it is not field-randomized, lacks human-team baselines, may suffer from selection biases in tasks/models, and does not establish external validity beyond the lab environment. Methods Rigormedium — Strong engineering rigor: explicit control of many experimental variables, automated deployment and grading (CI/CD + TAgent), and multi-dimensional metrics (functional score, latency, token/cost logs). Limitations include potential sample-selection and contamination concerns (university capstones, LLM training overlap), sparse reporting of statistical inference or uncertainty, unclear randomization/seed strategy, and no human baseline or production validation. Sample10 full-scale university capstone-style web projects spanning domains (real-time messaging, asset management, crowdsourcing, requirement tracking, image processing, e-commerce, analytics, RBAC, news search, live teaching), each decomposed into 6–8 modules and a 100-point rubric; 10 collaboration topologies implemented in LegoGent; automated evaluator TAgent; evaluations run across multiple LLMs (e.g., Claude Opus 4.8, GPT-5.5, DeepSeek variants, GLM-5.2, Qwen3.6-Flash) over ~100 team configurations with up to three refinement rounds per run on Intel Xeon servers; measured outcomes: functional completion (0–100), wall-clock minutes, token usage, and USD cost. Themeshuman_ai_collab productivity org_design IdentificationControlled benchmarking experiment that holds task, rubric, deployment path, and model family constant while systematically varying the multi-agent collaboration topology (10 modes) across 10 realistic full-stack projects and repeating runs; outcome differences are attributed to the coordination mode via this controlled variation and repeated trials (up to three refinement rounds per run). GeneralizabilityTasks are university capstone projects and may not capture scale/complexity of industry codebases or long-lived maintenance work, Evaluated LLMs and model versions are a snapshot in time; results may not generalize as models and pricing change rapidly, Automated agent prompts and isolated agent processes may not reflect real human-AI team interactions or mixed human-agent workflows, CI/CD stack and infrastructure choices (GitLab, SonarQube, XDeploy) may influence deployability outcomes relative to other environments, Possible LLM training/data contamination not fully addressed; real-world data and security constraints could alter performance, Limited statistical reporting and possible lack of randomized assignment reduce external causal claims

Claims (8)

Claim	Direction	Outcome	Confidence & Evidence	Details
MSEval evaluates multi-agent coding using 10 authentic full-stack software projects spanning 10 domains and 10 collaboration topologies, producing a 100-case benchmark grid. Organizational Efficiency	positive	Benchmark coverage of multi-agent software-delivery configurations	Reading fidelity high Study strength medium	n=100 0.48
MSEval measures software-delivery performance jointly using functional completion quality, wall-clock latency, and prefix-cached token usage. Organizational Efficiency	positive	Functional completion score, wall-clock time, and token usage	Reading fidelity high Study strength medium	n=100 0.48
For identical tasks and models, changing the organizational topology shifts performance scores by more than 30 points and can double wall-clock time. Team Performance	mixed	Functional completion score and wall-clock time	Reading fidelity high Study strength medium	n=100 over 30 points; doubles wall-clock time 0.48
Pipeline-style structured collaboration converges fastest and achieves the highest quality among the evaluated coordination modes. Organizational Efficiency	positive	Convergence speed and functional completion quality	Reading fidelity high Study strength medium	n=100 0.48
Heavy project-manager oversight degrades multi-agent coding performance. Team Performance	negative	Multi-agent coding performance, including quality and speed–cost trade-off	Reading fidelity high Study strength medium	n=100 0.48
On the instant-messaging project, Claude Opus 4.8 achieved a best functional score of 97, requiring 110 minutes and approximately USD 654. Output Quality	positive	Best functional completion score, coding time, and cost	Reading fidelity high Study strength medium	n=10 97 points; 110 minutes; about $654 0.48
On the instant-messaging project, Qwen3.6-Flash scored 0 across all 10 collaboration modes. Output Quality	negative	Functional completion score	Reading fidelity high Study strength medium	n=10 0 points in all 10 modes 0.48
The benchmark holds the project, model, requirement document, refinement-round budget, deployment path, and grading rubric fixed while varying the collaboration topology. Governance And Regulation	positive	Causal comparability of topology effects on quality, time, and token use	Reading fidelity high Study strength medium	n=100 0.48