Supervisor power backfires for generative agent teams: allowing a Manager to force revision loops produces hedged, less concise business-intelligence reports and raises token costs by ~51.5% without improving factual specification accuracy; the damage concentrates inside revision loops rather than in first drafts.

Main Finding

A single authority link—the Manager’s ability to reject worker output and force revisions—lowers quality and raises cost in open-ended LLM-team synthesis. In a paired experiment (43 products, 86 runs) that held roles, prompts, models, tools, and data fixed, the flat (non-authoritative Manager) organization outperformed the hierarchical (loop-back authority) organization on Utility (d = 0.42, p = 0.009) and Writing Clarity (d = 0.34, p = 0.030). The hierarchical design consumed ~51.5% more tokens with no specification-accuracy benefit; the quality gap emerges inside revision loops (each Manager-issued loop correlates with a −0.14 drop in Writing Clarity), and hierarchical reports show 53% more epistemic hedging per 1,000 words.

Key Points

• Experimental design
  • Paired within-product test: 43 products × 2 forms = 86 runs. For each product, the same random assignment of model instances to roles was used in both forms.
  • Only treatment difference: Manager has loop-back authority (can reject and force up to two revisions) vs. Manager may only comment (no rejection authority).
• Agents, roles, models
  • Five roles: Researcher, Analyst, Writer, Critic, Manager.
  • Model pool: GPT-5.4, Gemini-3.1-Pro, Qwen-3.5-122B, GLM-5, Mistral-Large-3; Manager drawn from subset (Qwen excluded).
  • Temperature = 0, ReAct agents, shared blackboard orchestration.
• Task and data
  • Business-intelligence reports synthesizing product specs and Amazon customer reviews for gaming laptops (RTX 4060 class).
  • Task is open-ended with partially verifiable ground truth (technical specs checkable; strategic analysis not).
• Evaluation
  • Five-model LLM judge panel (each judge writes qualitative analysis then gives 1–5 scores).
  • Metrics: Writing Clarity (Structure, Coherence, Conciseness), Utility (Actionability, Root Cause, Strategic Depth), Specification accuracy (deterministic extraction & verification via vector thresholds).
  • Mixed-effects models control for writer model, number of reviews, mean rating; product as random intercept.
• Key quantitative results
  • Utility: flat > hierarchical, paired Cohen’s d = 0.42, p = 0.009.
  • Writing Clarity: flat > hierarchical, d = 0.34, p = 0.030.
  • Specification accuracy: no meaningful difference (both at ceiling).
  • Token cost: hierarchical form used ~51.5% more tokens (pipeline overhead), experiment cost small ($47.71 total).
  • Report lengths similar (≈1036 vs 1046 words), indicating padding/hedging rather than verbosity as the judge-perceived issue.
  • Hedging: hierarchical reports contain ~53% more epistemic hedges per 1,000 words.
  • Revision mechanism: Writer’s first draft in hierarchical condition ≈ flat report; quality degrades across Manager-initiated revision loops.
  • Per-loop effect: ~−0.14 Writing Clarity points per Manager loop (hierarchical arm).
• Robustness
  • Effects persist in mixed-effects regressions (hierarchical associated with β ≈ −0.094 for both Writing Clarity and Utility, p ≲ 0.03).
  • Primary tests survive multiple-test correction; multiple sensitivity checks reported (judge leave-one-out, nonparametric tests) in supplement.

Data & Methods

• Design: Paired within-product randomized assignment of models to roles; isolates the single structural primitive (loop-back authority).
• Implementation: LangGraph framework, shared blackboard pattern; tools include get_product_specs and search_reviews (ChromaDB + SBERT embeddings).
• Manager treatment: in hierarchical form, Manager inspects full blackboard after steps and can inject binding directives into a worker’s context; capped at two loop-backs per run. In flat form, Manager explicitly instructed it has no authority to reject work.
• Critic role: separates generation from grounded critique (uses verify_claim tool with embedding-similarity thresholds 0.7/0.5 to label VERIFIED/PARTIAL/UNVERIFIED).
• Judges: five LLMs (same model pool) evaluate each report, each judge provides qualitative analysis before numeric scoring to mitigate known LLM-as-judge biases; Krippendorff’s alpha and per-judge analyses reported in supplement.
• Metrics: paired-sample t-tests for primary outcomes across 43 pairs; Cohen’s d for effect size; linear mixed models across all runs to control for confounds; additional diagnostics (hedging density, lexical measures, draft overlap) to locate mechanism.
• Limitations noted by authors: task class restricted (open-ended business reporting with partial verifiability), loop cap fixed at 2, deterministic generation (temp=0), specific model pool, judge-panel limitations (discussed and sensitivity-checked).

Implications for AI Economics

• Supervision has explicit resource costs and non-monotonic quality effects
  • Supervisory (hierarchical) orchestration increased token usage by ~51.5% without improving checkable accuracy and reduced judged strategic and writing quality. For production systems, that overhead scales linearly with throughput and translates to higher API costs and latency—important for cost-benefit accounting.
• The value of managerial oversight is conditional on verifiability
  • When a Manager can check facts (i.e., verify claims deterministically), oversight can pay for itself. When oversight is restricted to opinion/critique on open-ended synthesis, it can induce sycophancy/hedging and reduce useful output. Platform and product designers should therefore attach supervisory tiers primarily where ground-truth checks or tooling can back critiques.
• Organizational design must treat agent behavioral response to authority as endogenous
  • Classical MAS theory assumed agent behavior under authority fixed; generative LLMs respond to authoritative prompts (sycophancy, Degeneration-of-Thought). Economic models of multi-agent AI services should incorporate these behavioral externalities when designing contracts, pricing orchestration, or advising clients about governance layers.
• Pricing and productization of orchestration services
  • Orchestration services that sell hierarchical workflows should price and advertise the cost and potential quality trade-offs: paywalls for "manager oversight" should reflect extra token/API cost and the potential that authority may reduce exploratory/strategic quality on open-ended tasks.
• Deployment guidance for firms and platforms
  • Prefer flat routing (non-coercive manager) for open-ended synthesis tasks where verification is partial or absent, to preserve exploratory reasoning and concise writing.
  • Use hierarchical loop-back only when directives can be grounded in verifiable checks (automated tools, validators), or when the cost of a false negative/positive is sufficiently large to justify increased tokens and possible quality drift.
  • Consider hybrid designs: separate generation and tool-grounded critique (as done here) and restrict managerial loop-back to verifier-equipped managers.
• Policy and market implications
  • Transparency and standards: Buyers of LLM orchestration should demand disclosure of organizational topologies and expected compute overhead; regulators or industry standards bodies might classify when supervisory layers are likely to degrade generative output.
  • Externalities at scale: even modest token overheads per task accumulate—market-level adoption of hierarchical orchestration for ill-suited tasks can materially increase cloud costs and energy use.
• Directions for economic modeling and further research
  • Model the trade-off between supervision-induced convergence (faster coordination) and supervision-induced degeneracy (sycophancy, hedging) as a function of task verifiability, penalty asymmetry for errors, and verification-cost. Incorporate token/compute cost and latency into objective functions for automated organizations.
  • Test generalizability across other open-ended domains, different loop caps, human-in-the-loop supervision, stochastic generation (nonzero temperature), and alternative judge/evaluation frameworks to map contours where hierarchy is beneficial vs. harmful.

Limitations and cautions - Results are task- and implementation-specific (business-intel reporting, specific model pool, loop cap = 2). Generalize cautiously. - Judge panel is LLM-based (mitigations applied); human evaluation would be a useful complementary check. - The authority link here is purely prompt-based (no material sanctions). Real-world supervised teams with incentives or enforced rewrites may behave differently.

Bottom line: For open-ended synthesis where key claims cannot be deterministically verified, adding an authoritative Manager that forces revisions can reduce judged usefulness and clarity while raising token/computational costs. Supervisory tiers are economically justified mainly when they can perform verifiable checks; otherwise, flat, non-coercive coordination is likely more cost-effective.