0 cumulative citations
View corpus contextA lightweight CRDT-backed coordination layer cuts catastrophic one-file abandonments and reduces variance in multi-agent coding, producing higher mean code quality at matched compute; pooled results report a ~13.7× reduction in abandonment odds for AgentRoom versus solo runs, implying coordination — not raw parallelism — drives the gains.
AgentRoom: Concurrent Multi-Agent Coding in a CRDT-Backed Shared Workspace
AgentRoom — a CRDT-backed shared filesystem plus an MCP coordination layer — reduces single-agent ‘stub-and-exit’ failures, cuts run-to-run variance, and raises mean code-quality relative to solo and uncoordinated parallel multi-agent coding on multi-file TypeScript tasks (notably at N=2).
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
Concurrent multi-agent coding promises division of labor across modules, robustness through redundancy, and parallel exploration at the natural granularity of multi-file projects. Realtime collaborative editing protocols solve this coordination problem for human teams via Conflict-free Replicated Data Types (CRDTs), but the LLMs underneath generate one token at a time and existing multi-agent coding systems inherit this serial limit: they either sequence agents through phase handoffs or pool independent samples without coordination, and a single agent abandons up to half of hard tasks with a one-file stub-and-exit. AgentRoom is a realtime collaborative editing protocol for concurrent coding agents. Its runtime layer exposes file-level claim, status, and broadcast as MCP tools on a CRDT-merged shared filesystem. Five frontier coding-CLI models ran four backend coding tasks, with cross-language checks in Python DevBench and Rust+axum. For CLI-stable models, AgentRoom with 2 agents abandons fewer tasks than Solo and has less run-to-run variation. At matched-compute, one positive mean LLM-judge contrast puts AgentRoom over parallel-merge. The other contrast, a bundle probe, puts full AgentRoom above each partial case: an ordering rather than a percentage split. Coordination, not parallelism or CRDT-merge, bears the load.
Summary
Main Finding
AgentRoom — a CRDT-backed shared filesystem plus a small Model Context Protocol (MCP) exposing file-level claim, broadcast, state, and read — meaningfully improves multi-agent coding outcomes by adding explicit coordination. At N=2 agents, AgentRoom greatly reduces the lone-agent “stub-and-exit” abandonment failure mode and run-to-run variance, and at matched compute outperforms naive concurrent execution (parallel-merge) and sequential pipelines. Coordination, not just parallelism or CRDT merging, is the principal driver of these gains.
Key Points
- AgentRoom design
- Shared CRDT filesystem (pycrdt / Yrs-based) for immediate character-level merges (∆crdt ≈ 2 s).
- MCP runtime with atomic room_claim(path), room_release, room_broadcast/read (append-only JSONL log), and room_state.
- Advisory 6-step workflow for agents (read state → claim files → write → poll → report).
- Claim enforces mutual exclusion at file granularity to avoid intent-level collisions.
- Core empirical findings
- Abandonment reduction: pooled across 12 model×task strata, solo agents had 13.7× the odds of 1-file abandonment compared to AgentRoom (CMH OR = 13.7, 95% CI [3.9, 48], p < 1e-5).
- Variance reduction: adding a second agent in AgentRoom cut run-to-run standard deviation by ~30–45% on non-failing runs (per-model σ reductions shown).
- Matched-compute six-condition ablation (T4, Sonnet 4.6): monotonic ordering of mean quality (LLM-judge composite, means)
- ChatDev-style sequential: 0.333 (n=6)
- Parallel-merge (concurrent, no coordination): 0.456 (n=12)
- Solo (single agent): 0.544 (n=32)
- Shared-only (CRDT but no collab prompt/MCP): 0.575 (n=11)
- Shared+collab noMCP: 0.588 (n=7)
- AgentRoom: 0.669 (n=14, σ=0.14)
- AgentRoom vs parallel-merge: mean +0.213, Welch t = 3.35, p = 0.003 (matched compute).
- Why coordination matters
- CRDT SEC guarantees eventual byte-level preservation but not semantic compatibility; concurrent, uncoordinated edits commonly produce merged files that break compilation/tests.
- File-level claims eliminate collision probability (theoretical pairwise collision ≈ 0.20 at N=2, K≈5 operating point).
- MCP log surfaces violations and supports cross-agent bug-fix workflows.
- Limitations observed
- Gains concentrated at N=2; quality declines beyond this operating point.
- CRDT merge preserves bytes of conflicting edits (no semantic merge); coordination needed to avoid intent conflicts.
- Experiments focused on Express.js/TypeScript tasks (T1–T5) — broader cross-language generalization limited.
- One model (GPT-5.4-mini) had CLI stability issues under concurrent MCP execution (deployment caveat).
Data & Methods
- Tasks
- Four main Express.js/TypeScript tasks spanning difficulty:
- T1 (JWT auth, ≈6 files), T2 (marketplace, ≥10 files), T4 (double-entry ledger, ≥15 files), T5 (algorithmic trading platform, ≥15 files). T4 is the primary budget-fair benchmark.
- Wall-clock budgets: 300s for T1/T2/T3, 600s for T4, 900s for T5.
- Four main Express.js/TypeScript tasks spanning difficulty:
- Models and deployment
- Five frontier coding/LLM models from three providers: Anthropic Claude Sonnet 4.6 & Haiku 4.5, OpenAI GPT-5.4 & GPT-5.4-mini, Google Gemini 3 Flash. Headline results use the four CLI-stable models (exclude GPT-5.4-mini due to crash).
- Agents invoked via vendor CLIs, non-interactive, vendor-default sampling; no seed control (hosted models).
- Baselines / Ablations
- Solo (single agent), Shared-only (agents share CRDT workspace but no collab prompt or MCP), Parallel-merge (separate workspaces, post-hoc file union), shared+collab-noMCP, ChatDev-style sequential pipeline.
- Merge and coordination primitives
- CRDT: op-based character-level merges with Lamport timestamps; small brace/paren balance checks added.
- MCP: atomic claim(path) → OK / CONFLICT, broadcast/read append-only, state() returns claims + recent log window.
- Scoring & statistics
- Primary continuous scorer: LLM-judge composite (Sonnet 4.6 with fixed rubric: spec coverage 0.35, correctness 0.30, code quality 0.20, test rigor 0.15).
- Cross-validation scorers: regex-based scorer and AST-based scorer (TypeScript compiler API).
- Abandonment labeled via deterministic 1-file-abandonment classifier (<0.3 quality + small file count or early exit).
- Key tests: Cochran-Mantel-Haenszel (pooled OR), Fisher’s exact (per-stratum abandonment), Welch t (mean contrasts), bootstrap 95% CIs (10k resamples).
- Judge robustness: Sonnet correlates with other judges (Anthropic Haiku r≈0.87, Codex r≈0.86) though Codex stricter by ~−0.151 mean.
- Evidence tiers
- Tier I: judge-free binary abandonment outcome (strongest evidence).
- Tier II: continuous LLM-judge quality cells (matched-compute ablation).
- Tier III: exploratory regex/AST cross-model comparisons.
Implications for AI Economics
- Productivity per unit compute improves with coordination
- At matched compute, AgentRoom yields higher expected output quality and fewer catastrophic failures versus naive parallelism. For firms selling AI coding services, this implies better ROI on the same compute budget by investing in coordination infrastructure rather than only adding parallel agents.
- Risk and variance reduction have economic value
- Substantial reduction in abandonment and variance reduces wasted compute and developer oversight costs. For risk-averse buyers or contracting environments (SLAs, deadlines), lower variance can be as valuable as average-quality gains.
- Coordination infrastructure is a leverage point
- The MCP (claims, broadcast, state) is relatively lightweight but produces outsized benefits over CRDT-only approaches. This suggests vendors could charge premium or capture value by offering coordinated multi-agent orchestration rather than raw parallel access to LLMs.
- Diminishing returns and staffing design
- Gains are concentrated at small N (N=2). Economic strategies that simply scale numbers of agents without coordination can amplify failure modes (e.g., parallel-merge amplifies “stub-and-exit”). Allocation policies should prioritize careful orchestration over adding agents.
- Cost models and billing implications
- Billing by compute/run can disincentivize coordination if providers charge per-agent; alternative pricing (outcome-based, subscription to coordinated rooms, or per-project bundles) could align incentives: buyers pay for reliable completion rather than raw agent-minutes.
- Product-market fit for agent orchestration tools
- Tools that provide file-level locks, shared workspaces, and transparent logs map directly to measurable improvements (abandonment and variance). There is a commercial opportunity for middleware (MCP-like services) to improve enterprise adoption.
- Quality assurance and downstream costs
- AgentRoom’s reduction in semantic collisions lowers downstream QA and debugging costs. Firms estimating total cost of AI-assisted development should account for coordination savings as reductions in post-generation QA effort.
- Policy and reproducibility
- Hosted-proprietary models and run-to-run variability complicate reproducibility and auditing. The economic value of deterministic pipelines (or audit logs like MCP broadcast) increases where regulatory or compliance costs are material.
- Research and deployment investment priorities
- From a social-welfare / R&D allocation perspective, investment in coordination primitives (protocols, tooling) yields high marginal returns compared to brute-force parallelism. Future evaluations should include compute-cost-per-quality and total-cost-of-ownership analyses across multiple languages and longer-term maintenance.
Suggested next economic analyses - Compute-cost-per-quality-gain: quantify dollars or compute-hours per incremental LLM-judge point for AgentRoom vs baselines. - Expected cost-savings from reduced QA/bugfix time due to fewer semantic collisions. - Market-design experiments: pricing coordinated-agent services (per-room subscription vs per-run). - Scaling study: incremental marginal benefit and cost beyond N=2 to determine optimal team size under different task classes.
If you want, I can produce a short table mapping observed empirical gains (mean lift, σ reduction, abandonment OR) to an estimated monetary impact under a set of per-run compute and labor-cost assumptions.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper presents controlled, repeatable experimental contrasts with a robust binary outcome (abandonment) that replicates across multiple proprietary LLMs and tasks and a matched-compute ablation showing a mean-quality advantage; however, outcome measurement relies heavily on LLM-based judges and proprietary model behavior, sample sizes are modest in some cells, tasks are limited to multi-file Express.js/TypeScript benchmarks, agents are synthetic (LLMs) rather than human+AI teams, and generalization beyond the specific models/tasks/CRDT implementation is uncertain.
Methods Rigormedium — The design uses principled matched-compute ablations, multiple baselines, stratified pooling, cross-validation of scoring metrics (LLM-judge, regex, AST), and reasonable statistical tests; but some choices (LLM-judge as primary scorer, proprietary closed models, underpowered cells for some contrasts, limited agent counts mostly N=2, and advisory rather than enforced protocol adherence) limit internal and external rigor.
SampleExperimental runs using 4–5 frontier coding LLMs (Claude Sonnet 4.6, Claude Haiku 4.5, OpenAI GPT-5.4 / GPT-5.4-mini, Google Gemini 3 Flash), primarily four multi-file Express.js/TypeScript coding tasks (T1–T5 with emphasis on T4/T5), wall-clock budgets 300–900s depending on task, multiple runs per model/condition with per-cell n reported (varies, e.g. Sonnet T4 n≈14–33 across conditions); analyses use a budget-fair pool (runs >=30s) and both binary abandonment labels and continuous LLM-judge composite quality scores (cross-validated with regex/AST scorers).
Themesproductivity human_ai_collab
IdentificationControlled lab-style experiments comparing AgentRoom to several baselines at matched compute (Solo, Shared-only, Parallel-merge, ChatDev-style). Statistical contrasts use per-stratum comparisons by model/task, Cochran–Mantel–Haenszel pooling for the binary abandonment outcome, Welch t-tests for continuous quality scores, Fisher exact tests for per-stratum counts, and bootstrap CIs for emergence scores; runs are filtered to a budget-fair pool (>=30s) and results are stratified by model and task to reduce confounding by model/task heterogeneity.
GeneralizabilityLimited to TypeScript/Express.js multi-file coding tasks; other languages/runtimes not tested broadly, Results depend on specific proprietary LLM versions and vendor CLIs; behaviour may change with different model families or updates, Agent experiments focus on synthetic LLM agents (N≈2); human–AI team dynamics and larger agent teams are not evaluated, Primary quality measure is an LLM-based judge composite which may introduce scorer bias despite cross-validation, CRDT implementation specifics (char-level Yrs, 2s merge window, brace/paren sanity checks) may influence outcomes and not generalize to other merging semantics
Claims (8)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| Across 12 model-by-task strata, Solo agents had 13.7 times the odds of 1-file task abandonment compared with AgentRoom using two agents. Job Displacement | negative | Odds of 1-file stub-and-exit task abandonment |
Reading fidelity
high
Study strength
high
|
n=225
odds ratio = 13.7; 95% CI [3.9, 48]
|
| AgentRoom with two agents reduced run-to-run quality-score variability by approximately 30–45% relative to Solo across the three powered CLI-stable models. Output Quality | negative | Run-to-run standard deviation of the quality composite |
Reading fidelity
high
Study strength
medium
|
n=140
∼30–45% reduction in σ
|
| On the T4 task using Sonnet 4.6, AgentRoom produced a higher mean LLM-judge quality score than parallel-merge at matched compute. Output Quality | positive | Composite LLM-judge quality score |
Reading fidelity
high
Study strength
medium
|
n=26
+0.213 mean-quality advantage; Welch’s t=3.35, p=0.003
|
| In the six-condition T4 ablation, AgentRoom had the highest mean quality score, exceeding shared-only, shared-plus-collaboration without MCP, Solo, parallel-merge, and the ChatDev-style sequential pipeline. Output Quality | positive | Mean composite quality score |
Reading fidelity
high
Study strength
medium
|
n=82
AgentRoom mean = 0.669; ordering: 0.333 < 0.456 < 0.544 < 0.575 < 0.588 < 0.669
|
| Naive concurrent parallel-merge underperformed the single-agent Solo baseline on the T4 Sonnet task. Output Quality | negative | Composite LLM-judge quality score |
Reading fidelity
high
Study strength
medium
|
n=44
0.456 vs 0.544 mean quality
|
| After excluding catastrophic infrastructure-failure runs, AgentRoom had higher average T4 quality than Solo for Sonnet 4.6. Output Quality | positive | Mean LLM-judge quality score among non-catastrophic runs |
Reading fidelity
high
Study strength
medium
|
n=46
+0.125 mean-quality difference; Welch t=2.30, p=0.022
|
| On T4, the 1-file abandonment rate fell from 32% for Solo to 5% for AgentRoom with two agents. Job Displacement | negative | Rate of 1-file stub-and-exit abandonment |
Reading fidelity
high
Study strength
high
|
32%→5%
|
| The CRDT substrate guarantees preservation of concurrent edits at the byte level but does not guarantee semantic compatibility or compilation correctness. Error Rate | mixed | Semantic compatibility and downstream compilation after concurrent edits |
Reading fidelity
high
Study strength
medium
|
not reported
|