Swapping LLM teammates barely hurts task success but raises coordination overhead by up to two-thirds; conventions formed in repeated collaboration make agents fungible in outcome but costly in coordination.

Main Finding

LLM agent teammates are largely interchangeable in task outcome but not in coordination efficiency. Swapping a role-matched agent (same base model, role prompt, and experience) between independently formed teams typically changes task score only a little, but raises the communication cost per unit of progress by 16–63%. The swap penalty concentrates in the seat that initiates plans, is often paid by the agent who remains, grows with how long the team has been together, and can be reduced by design choices (decoding temperature, greedy decoding) or by removing stale partner-specific memory.

Key Points

• Swap test: trade role-matched agents (same model + role + experience) between independently formed teams, with a placebo that reproduces the roster-change announcement but returns the same agent.
• Main quantitative pattern:
  • Task score: small losses under swap (example means: Collab-Overcooked level 2: intact 92.5 → swap 90.0; level 6: 64.1 → 58.0; Hanabi: 15.8 → 13.7 of 25).
  • Coordination cost (communication per unit progress): much larger increases under swap (example means: level 2: 3.33 → 3.85 messages/subtask ≈ +16%; level 6: 4.33 → 6.53 ≈ +51%; Hanabi: 0.64 → 1.04 hints/point ≈ +63%).
• Decomposition: define V = value of formation (placebo − naive), Π = partner-specific residue (placebo − swap), and ρ = Π/V. On coordination cost the partner-specific share (ρC) is 2–3× larger than on task score (ρ).
• Swap vs inexperienced replacement: in some settings (Hanabi) a swapped veteran is more costly than an inexperienced fresh replacement, consistent with interference from conventions learned with a former partner.
• Who pays: when the initiator (agenda-setter) is swapped, most extra communication is produced by the agent that stayed (the responder).
• Memory effects:
  • Notebooks: each agent keeps a two-part private notebook (task notes κ and partner notes π). Deleting arriving agent’s partner notes (Swap, cleared) often reduces the penalty; deleting an intact agent’s partner notes (Amnesia) reproduces a substantial portion of the swap residue, indicating partner-specific information was stored in π.
  • Σ = benefit of clearing incoming partner notes can be positive (i.e., outdated partner model can be worse than none).
• Drift and signatures: independently formed teams develop different protocol signatures; the magnitude of swap penalty tracks how far teams drift apart. After a swap, receiving teams often shift their observable signature toward the donor.
• Ablations:
  • Base model, decoding temperature, and formation length move the swap penalty in tandem with the inter-team drift.
  • Greedy decoding reduces both drift and swap penalty.
  • Doubling formation history increases both drift and swap penalty.
• Dynamics: the cost is not purely instantaneous — there is an initial shock at the first post-swap episode and a decay over subsequent episodes (measured in continuation runs).

Data & Methods

• Settings:
  • Three benchmarks: Collab-Overcooked low coupling (levels 1–2), Collab-Overcooked high coupling (levels 5–6), and two-player Hanabi.
  • Held-out evaluation tasks were disjoint from formation tasks.
• Agents:
  • All agents were copies of a single frozen base model (main study: GPT-5.6 Luna, low-cost tier) with role-specific system prompts.
  • Persistent private notebook per agent with two labeled sections: task notes κ (task facts) and partner notes π (partner-specific conventions/agreements). Notebook update budget ≈ 200 tokens after each episode.
• Cohorts and protocol:
  • K = 8 independently formed teams per cohort (same model/prompt but different seeds and task order).
  • Formation: E = 10 episodes per team (five formation tasks run twice).
  • Evaluation: R = 10 episodes per condition; six conditions (Intact, Placebo, Swap, Swap, cleared, Amnesia, Naive). Each condition is an independent fork from the post-formation copy.
  • Swap pairs: teams are paired into four disjoint pairs; role-matched agents are swapped between paired teams.
  • Additional runs: seat-fixed swap arms and a 10-episode continuation after swap to measure recovery.
• Metrics:
  • Task score T: benchmark-native normalization (0–100 for progress completeness in Overcooked; 0–25 in Hanabi).
  • Coordination cost C: communication per unit of progress (messages per completed sub-task in Overcooked; hints per point in Hanabi).
  • Contrasts/quantities: V = Tplacebo − Tnaive; Π = Tplacebo − Tswap; ρ = Π/V; analogues on C (ρC). W = Tplacebo − Tamnesia; Σ = Tswap,cleared − Tswap.
• Protocol signature:
  • Automatically extracted formation-time signature (hand-off templates, field orders, aliases, and hint→action mapping in Hanabi).
  • Compare signatures by Jensen–Shannon divergence d. Measured mean pairwise d across teams and post-swap shifts ∆ toward donor.
• Scale:
  • Main study: ~2,240 episodes across three settings (plus ablations adding ~1,950 episodes for total ≈ 4,190). Agents are not trained; experiments run via API calls.

Implications for AI Economics

• Hidden operational costs of replacement: even when task outcomes remain near-identical, replacing teammates can substantially increase communication overhead (16–63% more communication per unit progress). That translates into higher latency, more API calls, and higher runtime costs in deployed multi-agent services — a nontrivial economic externality of rotating agents or performing zero-downtime swaps.
• Switching costs grow with team familiarity: longer formation/interaction histories increase the partner-specific residue and therefore the cost of replacing agents. Policies that rotate agents frequently (for load balancing, canarying, or hot upgrades) should internalize the rising coordination costs rather than treating replacements as free.
• Design and versioning trade-offs:
  • Homogeneity and shared protocols reduce inter-team drift and lower swap costs; investing in standardized message formats or canonical protocol signatures is economically valuable.
  • Conversely, allowing agents to specialize and develop partner-specific conventions can improve steady-state efficiency but raises the cost of agent turnover — a classic trade-off between specialization (higher steady-state productivity) and flexibility (lower switching costs).
• Practical mitigations and economic levers:
  • Clear stale partner-specific memory when rotating agents (Swap, cleared) — cheap and often reduces mismatch costs.
  • Provide onboarding/rehab formation episodes after swaps to amortize the initial shock; the paper shows a shock + decay dynamic, so investing a small number of formation episodes post-swap can recover efficiency.
  • Use decoding and inference choices as knobs: greedy decoding reduced inter-team drift and swap penalty (implying lower coordination overhead), so inference configuration is an operational lever with economic consequences (but may trade off other quality dimensions).
  • Instrumentation: include coordination-cost metrics (communication per unit progress, extra API tokens/messages) in SLAs, cost forecasts, and upgrade decision rules.
• Strategic implications for multi-agent marketplaces and platforms:
  • Providers and integrators should price in the coordination overhead for heterogeneous team composition and for agent replacement/upgrade strategies.
  • There is value in offering services for “team re-formation” (paid onboarding episodes or protocol alignment passes) or shared memory layers that preserve portable, partner-agnostic task knowledge while minimizing partner-specific leakage.
• Limitations to generalization (policy cautions for economists and practitioners):
  • Experiments are on LLM-based symbolic/communicative benchmarks (Overcooked, Hanabi) with notebooks constrained to ~200 tokens and a single model family. Effects may differ with different architectures, larger private memory, multi-modal environments, or long-lived learned parameters.
  • The study measures short-run post-swap episodes (10 episodes per condition); longer-term adaptation dynamics and cost amortization beyond that horizon were not exhaustively explored.
• Bottom line for economic decision-making: agent interchangeability is not costless. When designing, operating, or pricing LLM-agent systems, account explicitly for coordination/friction costs driven by partner-specific conventions and for the ways system design (protocol standardization, memory management, inference settings) can reduce or increase those costs.