11 cumulative citations
View corpus contextSelf-organizing LLM teams systematically underuse their experts and lag behind the best member, suffering performance drops of up to 41% on ML benchmarks; teams average expert and non-expert views—a consensus-seeking bias that grows with size but makes them more robust to adversaries.
Multi-Agent Teams Hold Experts Back
arxiv
quasi_experimental
medium evidence
7/10 relevance
Full text
usable extracted full text
Source
PDF
Self-organizing multi-agent LLM teams routinely fail to match their best member—underperforming expert agents by substantial margins—because they tend toward consensus-averaging that underweights expertise and worsens with team size.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Multi-agent LLM systems are increasingly deployed as autonomous collaborators, where agents interact freely rather than execute fixed, pre-specified workflows. In such settings, effective coordination cannot be fully designed in advance and must instead emerge through interaction. However, most prior work enforces coordination through fixed roles, workflows, or aggregation rules, leaving open the question of how well self-organizing teams perform when coordination is unconstrained. Drawing on organizational psychology, we study whether self-organizing LLM teams achieve strong synergy, where team performance matches or exceeds the best individual member. Across human-inspired and frontier ML benchmarks, we find that -- unlike human teams -- LLM teams consistently fail to match their expert agent's performance, even when explicitly told who the expert is, incurring performance losses of up to 41.1% on ML benchmarks. Decomposing this failure, we show that expert leveraging, rather than identification, is the primary bottleneck. Conversational analysis reveals a tendency toward integrative compromise -- averaging expert and non-expert views rather than appropriately weighting expertise -- which increases with team size and correlates negatively with performance. Interestingly, this consensus-seeking behavior improves robustness to adversarial agents, suggesting a trade-off between alignment and effective expertise utilization. Our findings reveal a significant gap in the ability of self-organizing multi-agent teams to harness the collective expertise of their members.
Summary
Main Finding
Self-organizing multi-agent LLM teams systematically underperform their best individual member: teams fail to achieve “strong synergy” and instead dilute expert knowledge. On frontier ML benchmarks the authors observe relative synergy gaps of 6.3%–41.1%. The primary bottleneck is not identifying who the expert is but leveraging that expert’s knowledge—teams tend to form integrative compromises (averaging expert and non‑expert views) that harm accuracy, and this expertise dilution worsens with team size. Paradoxically, the same consensus-seeking behavior improves robustness to adversarial agents.
Key Points
- Strong synergy test: a team should match or exceed its best member. LLM teams consistently fail this test across tasks.
- Benchmarks: experiments cover classic human-team tasks (NASA Moon Survival, Lost at Sea, Student Body President) and modern ML benchmarks (MMLU Pro, GPQA Diamond, HLE, MATH-500, SimpleQA).
- Synergy gaps: relative underperformance of teams vs best individual of 6.3%–41.1% on ML benchmarks.
- Bottleneck decomposition:
- Identification gap (can the team find who is expert?) is small.
- Expertise leveraging gap (can the team defer appropriately?) is large and primary.
- Mechanism: conversational analysis shows “integrative compromise”—teams average conflicting views rather than weighting higher‑quality expertise—this correlates negatively with performance (p < 0.05).
- Team size: expertise dilution increases with larger teams (performance degrades; p < 0.05).
- Robustness trade-off: consensus/compromise behavior mitigates harm from adversarial team members (teams are more robust to sabotage).
- Experimental protocol: teams of 4 agents, 4 rounds of discussion, final answer = majority vote of post-discussion responses; individual opinions collected beforehand.
- Models used: a mix of frontier models (Anthropic Claude variants, OpenAI GPT-4o/GPT-5, GPT-3.5, etc.) to ensure genuine heterogeneity.
- Metrics: relative synergy gap; for ML benchmarks an “At Least One Correct” upper bound measures potential from perfect per‑problem expertise selection.
- Reproducibility: evaluation harness is open-sourced.
Data & Methods
- Two-pronged design:
- Controlled psychology tasks (intellective tasks) where expertise is induced and can be concentrated or distributed.
- Frontier ML benchmarks sampled (100 problems per benchmark) to test realistic heterogeneous model strengths.
- Expertise distributions:
- Concentrated: one agent holds task‑relevant information.
- Distributed: knowledge partitioned across agents.
- Information conditions:
- No Information, Expert Not Mentioned (control), Reveal Expert (team explicitly told who the expert is, with prompts optimized via GEPA), Best Individual (single agent evaluated alone).
- Full Information baseline (all agents given same expert info) to control for communication noise.
- Models and team configurations:
- Psychology tasks: Claude Haiku 3.5 and GPT-4o-mini in varying mixes.
- ML tasks: diverse set of state-of-the-art models (Claude Opus/Sonnet/Haiku, GPT-5, GPT-4o, GPT-4o-mini, GPT-3.5 Turbo, o3-mini/o4-mini).
- Procedure:
- Collect pre-discussion individual answers.
- Run 4 rounds of open deliberation (no fixed roles or routing).
- Final decision = majority vote of post-discussion answers.
- Analyses:
- Measure relative synergy gaps and decompose into identification vs leveraging gaps.
- Conversational content analysis to quantify compromise vs deference.
- Ablations for team size and adversarial membership to study dilution and robustness.
Implications for AI Economics
- Coordination value and orchestration investment:
- Unconstrained deliberation can reduce the value of specialized high-quality models. Returns to investing in a single high-performing expert model may exceed returns from naïve multi-agent ensembles unless orchestration mechanisms exist to enable proper deference.
- Economic incentives should favor investment in orchestration protocols (role assignment, reputational signals, weighted aggregation, learned routers) that actually leverage expertise rather than rely on open deliberation.
- Market design and pricing:
- Markets for LLM services should price not only model raw accuracy but also the ecosystem (tools, orchestrators, aggregation algorithms) that extract and deploy a model’s comparative advantage within teams.
- Buyers deciding between “one best model” vs. “heterogeneous team” should weigh orchestration costs—without designed coordination, heterogeneous teams can underdeliver.
- Team composition and returns to scale:
- Larger teams can exhibit negative marginal returns due to expertise dilution. Platform and procurement decisions should consider optimal team size; more agents is not necessarily better.
- Trade-offs: robustness vs. performance
- Consensus-seeking improves resistance to adversarial/sabotaging agents; there is an economic trade-off between manipulation-robust designs and performance-maximizing designs. For high-stakes/high-adversary environments (e.g., open-platform moderation), favoring consensus may be rational; for pure accuracy-driven tasks, design should prioritize effective expertise weighting.
- Labor and human+AI teams:
- Human managers or meta-agents that assign roles, enforce deference, or adjudicate outputs may be necessary to realize productivity gains from AI teams. This implies continued demand for coordination skills and investment in toolchains that surface model competence.
- Policy and governance:
- Regulations or procurement standards may need to require transparency about how multi-agent systems aggregate expertise (to avoid deployments that appear more capable than they are).
- Research & product priorities:
- Prioritize mechanisms that enable expertise leveraging: calibrated credibility signals, dynamic role assignment, expertise-weighted voting, learned aggregation (routing/mixture-of-experts at the system level), and reputational systems across tasks.
- Prompt engineering alone (even optimized prompts) is insufficient: structural coordination mechanisms will likely be more economically valuable.
- Practical takeaways for firms:
- For time/cost-limited deployments, using a single best model may outperform an unconstrained pool of heterogeneous models.
- If using agent teams, invest in explicit orchestration (roles, routing, weighted aggregation) and keep teams small unless you have robust aggregation mechanisms.
- Consider the adversarial environment when choosing whether to favor consensus (robustness) or explicit deference (accuracy).
If you want, I can draft concise decision rules or an economic checklist firms can use when choosing between single-model deployments and multi-agent teams, or sketch concrete orchestration mechanisms (weighted voting, reputation tracking, mediator agents) informed by the paper’s findings.
Assessment
Paper Typequasi_experimental
Evidence Strengthmedium — The paper provides consistent, internally controlled experimental evidence across multiple benchmarks and decompositions that support causal claims about how coordination rules affect multi-agent LLM performance; however, evidence is limited to simulated benchmark tasks and specific model/configuration choices, so external validity to deployed systems and real-world economic outcomes is uncertain.
Methods Rigormedium — The authors use multiple benchmarks, systematic manipulations (e.g., expert identification, adversaries), quantitative decompositions, and conversational analyses, which indicate careful empirical work; but the study appears confined to particular LLM families/versions, prompt and hyperparameter settings, and synthetic task environments without field validation or extensive robustness across architectures, limiting methodological completeness.
SampleExperiments run on a set of 'human-inspired' and advanced ML benchmark tasks using teams of autonomous LLM agents of varying sizes and role configurations; conditions include naming an expert, varying team size, and injecting adversarial agents; performance measured against single-expert agent baselines across repeated simulated trials (specific model families, number of trials, and exact benchmarks not specified in the summary).
Themeshuman_ai_collab productivity
IdentificationControlled within-simulation experiments that manipulate team composition, information available to agents (e.g., explicitly naming the expert), presence of adversarial agents, and team size; causal effects inferred by comparing performance across these experimental conditions in benchmark tasks rather than from exogenous field variation.
GeneralizabilityFindings are derived from simulated benchmark tasks rather than real-world organizational settings, Results likely depend on the particular LLM families, model sizes, prompt designs, and hyperparameters used, Team behaviors may change with richer context, memory, tool-use, or different communication protocols not tested, Limited evidence on interactions with human team members or mixed human-AI teams, Unclear scalability to much larger teams, continuous-time interaction, or production deployments
Claims (6)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Across human-inspired and frontier ML benchmarks, LLM teams consistently fail to match their expert agent's performance, even when explicitly told who the expert is, incurring performance losses of up to 41.1% on ML benchmarks. Team Performance | negative | team performance relative to best individual (expert) agent |
Reading fidelity
high
Study strength
medium
|
up to 41.1% performance loss on ML benchmarks
|
| Unlike human teams, LLM teams consistently fail to match their expert agent's performance. Team Performance | negative | ability to achieve team synergy (team performance vs best individual) |
Reading fidelity
medium
Study strength
medium
|
not reported
|
| Expert leveraging, rather than identification, is the primary bottleneck causing LLM teams to underperform relative to their best individual member. Task Allocation | negative | effective use (leveraging) of expert contributions vs. identification accuracy |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Conversational analysis reveals a tendency toward integrative compromise—averaging expert and non-expert views rather than appropriately weighting expertise—which increases with team size and correlates negatively with performance. Team Performance | negative | degree of integrative compromise in conversation; correlation between that behavior and team performance |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Consensus-seeking (integrative compromise) behavior improves robustness to adversarial agents, suggesting a trade-off between alignment (robustness) and effective expertise utilization. Decision Quality | positive | robustness of team outputs to adversarial agents |
Reading fidelity
high
Study strength
medium
|
not reported
|
| Self-organizing multi-agent LLM teams have a significant gap in their ability to harness the collective expertise of their members. Team Performance | negative | ability to utilize collective expertise (overall team effectiveness relative to members) |
Reading fidelity
high
Study strength
medium
|
not reported
|