Automatically generated multi-agent systems cost up to ten times more but often perform worse than a single-agent Chain‑of‑Thought baseline; expert‑crafted multi‑agent architectures can beat generated ones on a synthetic decomposition task, exposing automated design's inefficient complexity.

The Illusion of Multi-Agent Advantage

Prathyusha Jwalapuram, Hehai Lin, Chuyuan Li, Fangkai Jiao, Sudong Wang, Yifei Ming, Zixuan Ke, Chengwei Qin, Giuseppe Carenini, Shafiq Joty · June 11, 2026

arxiv descriptive medium evidence 7/10 relevance Full text usable extracted full text Source PDF

Automatically generated multi-agent systems are consistently more expensive and underperform Chain-of-Thought with Self-Consistency on standard and interactive reasoning tasks, while expert-designed MAS outperform generated ones on a diagnostic dataset, revealing architectural bloat in current automated MAS design methods.

Prevailing wisdom posits that Multi-Agent Systems (MAS) are superior to Single-Agent Systems (SAS), citing advantages like context protection, parallel processing and distributed decision-making. However, empirical support for this claim relies primarily on comparisons with SAS baselines using benchmarks that prioritize isolated reasoning tasks, which do not adequately assess these advantages. Focusing on automatically generated MAS that are designed for enhanced generalizability over manually-designed counterparts, we perform a rigorous, systematic evaluation against SAS, specifically Chain-of-Thought with Self-Consistency (CoT-SC). Across traditional reasoning datasets and tasks with interactive multi-step workflows (e.g., BrowseComp-Plus), we demonstrate that automatic MAS consistently underperform CoT-SC despite being up to 10x more expensive. To isolate these failures from limitations inherent to task structure, we introduce a diagnostic synthetic dataset tailored for MAS featuring explicit task decomposition, context separation and parallelization potential. We show that expert-architected MAS consistently outperforms automatically generated architectures in both raw performance and cost-efficiency on this dataset, demonstrating that existing evaluation frameworks mask critical architectural gaps and inefficiencies of complex MAS by failing to account for the marginal utility of increased computational cost. Critically, systematic deconstruction of the generated MAS architectures reveals that current automated design paradigms produce architectural bloat that prioritizes superficial complexity which does not translate into functional utility, exposing a fundamental misalignment with multi-agent principles.

Summary

Main Finding

Automatically-generated Multi-Agent Systems (MAS) do not deliver a reliable or cost-effective advantage over strong Single-Agent System (SAS) baselines (specifically Chain-of-Thought with Self-Consistency, CoT-SC). Across standard reasoning benchmarks and a new diagnostic agentic task (SMFR), automated MAS often underperform or only marginally improve accuracy at large inference-cost multipliers (commonly ≈2–10×). Expert-designed MAS architectures can realize substantial gains, showing that MAS can be beneficial when architected correctly — but current automated MAS search/design produces architectural bloat and functional redundancy that negate those benefits.

Key Points

Empirical comparison: Six representative automatic MAS frameworks (DyLAN, MAS-Zero, ADAS, AFlow, MaAS, MAS-Orchestra) were systematically compared to a strong SAS baseline (CoT-SC) across multiple backbone LLMs (GPT-4o, GPT-5, GPT-OSS-120B, Gemini-2.5-Pro) and benchmark suites (HLE-Maths, GPQA-Diamond, SWE-Bench Lite, BrowseComp-Plus).
Cost-aware evaluation: The study explicitly controls for inference budgets and reports accuracy versus total inference cost (including search/validation overhead). When accounting for cost, CoT-SC typically dominates the efficiency frontier.
Quantitative outcomes:
- CoT-SC often outperforms automated MAS at a fraction (~<10%) of the computational cost.
- Significant MAS uplift appears rarely and only with strong backbones (e.g., on HLE-Math for GPT-5/Gemini-2.5-Pro).
- On the diagnostic SMFR dataset: GPT-5 CoT-SC ≈57.0% accuracy; automated MAS rarely beat this economically. Expert-MAS raised GPT-5 to ≈96.5% and GPT-OSS from 26.1% to 36.1% with cost comparable to CoT-SC.
Architectural diagnosis: Deconstruction of automated MAS revealed (i) role redundancy, (ii) collapse into simple CoT-like execution in practice, and (iii) excessive, non-functional complexity (architectural bloat) that fails to realize parallelization, specialization, or context separation benefits.
SMFR diagnostic benchmark: A synthetic multi-hop financial reasoning task was created (588 test instances + 16 validation), explicitly designed to reward context separation, per-subtask parallelism, and role specialization. Even here, automatic MAS underperformed CoT-SC in cost-efficient ways; a deterministic, expert-designed MAS reached near-ceiling performance.

Data & Methods

Benchmarks used: HLE-Maths (mathematical reasoning), GPQA-Diamond (QA), SWE-Bench Lite (code), BrowseComp-Plus (interactive multi-step browsing reasoning), plus the new SMFR diagnostic dataset.
SMFR generation: Procedurally generated "needle-in-a-haystack" stock-price problems using historical US equity prices. Tasks require multi-step extraction and numerical calculations (transaction parsing, P&L derivation, reverse-price calculation, threshold scanning, aggregation).
MAS frameworks evaluated: DyLAN, MAS-Zero, ADAS, AFlow, MaAS, MAS-Orchestra (covering both inference-time dynamic configurators and pre-optimized/learned orchestrators).
LLM backbones: GPT-4o, GPT-5, GPT-OSS-120B, Gemini-2.5-Pro. Results averaged over 3 independent runs (Gemini-2.5-Pro single run due to cost).
SAS baseline: Chain-of-Thought with Self-Consistency (CoT-SC) using 5-sample majority voting.
Expert-MAS: Human-crafted pipeline that enforces context/task separation, uses a Meta-Agent for topology parsing, parallel extractor/calculator sub-agents, and a deterministic Python executor to orchestrate and aggregate results (intended as an empirical upper bound for MAS utility).
Cost accounting: Total tokens and extra validation/search overhead included; comparisons visualize accuracy vs total inference spend.

Implications for AI Economics

Marginal utility of compute and design: Automated MAS commonly increase inference spend dramatically (often up to ~10×) for little or no accuracy gain. From an economic standpoint, current automated MAS represent poor ROI relative to simpler SAS sampling strategies (e.g., CoT-SC). Buyers and deployers should treat MAS claims skeptically unless cost-accuracy trade-offs are presented.
Investment prioritization: Improving backbone model capability (raising the “competence floor”) or investing in expert system architecture/design yields higher marginal returns than adopting off-the-shelf automatic MAS orchestrators. The study suggests diminishing returns (and even negative returns) from spending heavily on orchestration when backbones are mid-tier.
Procurement and pricing: API buyers and platform architects should demand cost-normalized benchmarks. Pricing models that ignore multi-call orchestration overhead mask real user costs; providers might consider metering/or offering bundled orchestration primitives to reflect true deployment economics.
Product strategy and R&D: For product teams, simpler SAS with budget-aware ensembling or CoT-SC are often the economically rational choice. R&D should prioritize (a) improving alignment between automated MAS search objectives and actual agentic principles (parallelization, context modularity), (b) hybrid approaches that combine deterministic orchestration with small, specialized agent calls, and (c) benchmarks like SMFR for evaluating true MAS value.
Labor and expertise: Contrary to expectations that automation will remove the need for human MAS designers, this work shows expert-architected MAS outperform automated designs. There remains economic value in human engineering and domain-specific orchestration expertise.
Policy and claims: Vendors and researchers should avoid unqualified claims about MAS superiority. Economic evaluations and deployment recommendations must include cost-per-performance and concrete evidence that MAS mechanisms (not just greater compute) drive gains.
Modeling implications: Economic models of AI deployment (cost-benefit, TCO, expected utility) should incorporate orchestration inefficiencies and the competence threshold for which coordination complexity delivers positive returns. When projecting productivity gains or cost savings from MAS, include sensitivity to backbone quality and orchestration overhead.

Link / reproducibility note: The paper provides dataset and code resources (https://multi-agent-eval.github.io/) for replication and further economic analysis.

Assessment

Paper Typedescriptive Evidence Strengthmedium — The paper runs systematic, controlled evaluations across standard reasoning benchmarks, interactive multi-step workflows (BrowseComp-Plus), and a bespoke synthetic dataset designed to expose MAS advantages; it also reports cost-normalized comparisons and ablation/deconstruction of generated architectures. However, strength is limited by likely sensitivity to choice of LLMs, specific automatic-generation methods evaluated, baseline tuning, possible narrow set of benchmarks, and unclear robustness across model scales and deployment settings. Methods Rigormedium — Methods include cross-task evaluation, a targeted diagnostic dataset tailored to MAS properties, cost-efficiency analysis, and architecture-level ablations—demonstrating careful experimental design; nonetheless, rigor is constrained by potential omissions such as breadth of automatic-design algorithms tested, transparency of hyperparameter tuning, reproducibility details (model versions, seeds), and the degree to which cost accounting reflects real deployment scenarios. SampleEvaluations use (1) multiple traditional reasoning datasets (unspecified here), (2) interactive multi-step workflow tasks including BrowseComp-Plus, and (3) a synthetic diagnostic dataset explicitly constructed to require task decomposition, context separation and parallelization; compared systems include various automatically-generated MAS architectures, expert-architected MAS, and a Chain-of-Thought with Self-Consistency single-agent baseline, with performance measured alongside compute/cost usage (reports up to 10x higher cost for some MAS). Themesproductivity org_design adoption IdentificationControlled algorithmic experiments comparing automatically generated multi-agent systems (MAS), expert‑designed MAS, and a single‑agent Chain-of-Thought with Self‑Consistency (CoT‑SC) baseline across multiple benchmark suites and a purpose-built synthetic diagnostic dataset; comparisons include performance metrics and cost-normalized evaluations to isolate architectural effects. GeneralizabilityResults may depend on the specific LLM family and model sizes used; other models could yield different trade-offs., Findings are conditional on the particular automated MAS design algorithms evaluated and may not generalize to other automated-design methods or improved search/hyperparameter tuning., Benchmarks chosen (including synthetic dataset) may favor or penalize MAS in ways not representative of real-world deployment tasks., Cost measurements may not fully capture real operational costs (latency, engineering integration, parallel hardware pricing, inference vs. orchestration overhead)., Human-in-the-loop or hybrid MAS designs common in production were not clearly evaluated, limiting applicability to deployed systems., Language, domain, and cultural diversity of tasks are unclear, restricting transferability across applications.

Claims (6)

Claim	Direction	Outcome	Confidence & Evidence	Details
Prevailing wisdom posits that Multi-Agent Systems (MAS) are superior to Single-Agent Systems (SAS), citing advantages like context protection, parallel processing and distributed decision-making. Output Quality	positive	comparative system capability (MAS vs SAS)	Reading fidelity high Study strength speculative	not reported 0.03
Empirical support for MAS superiority relies primarily on comparisons with SAS baselines using benchmarks that prioritize isolated reasoning tasks, which do not adequately assess MAS advantages. Other	negative	adequacy of existing benchmarks for evaluating MAS advantages	Reading fidelity high Study strength medium	not reported 0.18
Across traditional reasoning datasets and tasks with interactive multi-step workflows (e.g., BrowseComp-Plus), automatically generated MAS consistently underperform Chain-of-Thought with Self-Consistency (CoT-SC) despite being up to 10x more expensive. Output Quality	negative	task performance (accuracy/quality) and computational cost	Reading fidelity high Study strength medium	up to 10x more expensive 0.18
Existing evaluation frameworks mask critical architectural gaps and inefficiencies of complex MAS by failing to account for the marginal utility of increased computational cost. Organizational Efficiency	negative	marginal utility (performance gains per unit cost)	Reading fidelity high Study strength medium	not reported 0.18
On a diagnostic synthetic dataset tailored for MAS (explicit task decomposition, context separation and parallelization potential), expert-architected MAS consistently outperforms automatically generated architectures in both raw performance and cost-efficiency. Output Quality	positive	task performance (accuracy/quality) and cost-efficiency	Reading fidelity high Study strength medium	not reported 0.18
Systematic deconstruction of the generated MAS architectures reveals that current automated design paradigms produce architectural bloat that prioritizes superficial complexity which does not translate into functional utility, exposing a fundamental misalignment with multi-agent principles. Other	negative	architectural utility vs complexity (functional utility of design elements)	Reading fidelity medium Study strength speculative	not reported 0.02