Running a zero-shot manager that choreographs worker LLM calls and a shared filesystem can sharply raise coding accuracy for some models—e.g., Qwen3.8-27B jumps +23pp—though it roughly triples inference cost; in some cases orchestration is a cheaper path to high performance than switching to a pricier model.

Main Finding

A zero-shot manager–worker scaffold that coordinates identical LLM calls over a shared filesystem ledger can meaningfully raise coding-task accuracy versus a single-call baseline for many models, without any training or per-benchmark tuning. The accuracy gains are conditional (large for some models, null/negative for others) and come at a substantial token/cost overhead (roughly tripling the bill). In practice, orchestration can be a more cost-effective way to buy accuracy than moving to a larger model in many cases.

Key Points

• Design

  • Zero-shot self-orchestration: a prompt-driven manager loops, curates a task list, spawns fresh worker calls, and updates a shared workspace (plan, tasks, notes, best solution). No learned orchestrator, no per-benchmark tuning; every role is the same underlying model in a fresh context.
  • Shared filesystem (ledger) preserves state across short, role-specific calls and reduces per-call context bloat.
  • Manager is dynamic (decides next steps and when to stop), not a fixed pipeline.

• Core quantitative results (LCB-100, 100 hardest LiveCodeBench problems):

  • Main pinned-backend, reasoning-on, five independent passes at 128k cap:
    • Qwen3.8-27B: single 63.0% → manager 86.4% (∆ +23.4 ± 6.6)
    • GPT-5.6-Luna: single 67.2% → manager 77.8% (∆ +10.6 ± 5.1)
    • GPT-5.6-Terra: single 77.0% → manager 85.0% (∆ +8.0 ± 0.0)
    • Claude Fable 5 (single-only baseline): 87.4%
  • Manager gains shrink as the single-call baseline becomes stronger (largest gains where the model is weakest).
  • Some models did not benefit or worsened in other experiments (e.g., Qwen3.6-35B saw −1 to −9 with reasoning off).

• Cost and token impact

  • Running the manager typically ~triples token usage and cost per 100-problem pass.
  • Example costs per 100-problem pass (rounded):
    • Qwen3.8-27B: single ≈ $20.44 → manager ≈ $51.75
    • GPT-5.6-Luna: single ≈ $0.41 → manager ≈ $1.50
    • GPT-5.6-Terra: single ≈ $3.41 → manager ≈ $11.71
    • Claude Fable 5 single: ≈ $61.11
  • Cost-effectiveness examples:
    • GPT-5.6-Terra with manager (85.0%) nearly matches Fable 5 single (87.4%) at ~1/5 the price ($11.71 vs $61.11).
    • Qwen3.8-27B with manager (86.4%) approaches Fable 5 for less money (≈$51.75 vs $61.11).

• Mechanisms behind gains (from transcript analysis)

  • Context management: short worker calls + shared notes organize state and reduce truncation/cap hits.
  • Problem decomposition: manager-driven task splitting and incremental verification.
  • Rescue of truncated/empty single-call outputs was a major channel for some models (notably Qwen3.8), but not for OpenAI models (which had no cap hits).

• Truncation/rescue specifics

  • OpenAI arms (GPT-5.6 Luna/Terra) had zero cap hits / empty answers in both single and manager conditions — their manager gains come from improved organization/decomposition rather than rescue.
  • Qwen3.8-27B single arm (cap-matched to 128k) had many cap hits (150/500) and empty outputs (35/500); the manager reduced these to 5 cap hits and 0 empties, explaining a large portion of its improvement.

• Experimental framing

  • The paper does not attempt equal-token efficiency comparisons; it asks whether the additional test-time compute from orchestration buys accuracy and how cost-effective that is versus using larger models.

Data & Methods

• Benchmark
  • LiveCodeBench (LCB-100): the 100 latest hard problems from a coding benchmark suite.
• Models evaluated
  • Nine models across several experiments; main pinned-backend comparison focused on 4 arms: Qwen3.8-27B (local vLLM), GPT-5.6-Luna and GPT-5.6-Terra (OpenAI), and Claude Fable 5 (Anthropic).
  • Experiments included both open-weight models (9B → ~2.8T in the broader study) and frontier closed models.
• Experimental conditions
  • Two conditions compared for the same underlying model: single-call (one fresh call per problem) vs manager+workers over shared workspace. Temperature and base model were held constant.
  • Primary reported condition: reasoning-on, five independent passes per arm, output cap 128k (Qwen single run was originally at 250k and was replayed / cap-matched to 128k for fair comparison).
• Metrics and tests
  • pass@1 on LCB-100; per-pass means ± SD across five passes; paired per-problem comparisons (signed permutation / McNemar tests) and run-level tests reported.
  • Token accounting: detailed in-paper token in/out counts used to compute $/pass using quoted list rates and provider-specific rates for open-hosted models.
• Limitations noted by authors
  • Manager condition increases tokens so not an information-efficiency comparison.
  • Gains are model- and setting-dependent; orchestration can be neutral or harmful for some models/flags.
  • Some experimental variability due to routing and backend noise.

Implications for AI Economics

• Orchestration is a lever for cost-effective accuracy improvement
  • For many practical use cases, employing a zero-shot manager scaffold over cheaper (or self-hosted) models may be a better investment than upgrading to a larger/premium model, especially when provider pricing is high.
  • Example: a managed cheaper model can nearly match a frontier model’s accuracy at a fraction of the API spend.
• Pricing, caps, and provider design matter
  • Providers’ output caps and token pricing materially shape the cost-benefit of orchestration. Models or providers that permit long in/output reasoning (fewer caps) reduce the rescue opportunities that orchestration exploits.
  • Open/self-hosted models give further leverage: Qwen3.8’s manager arm was competitive with top-tier closed models at hostable costs, implying enterprises can optimize by choosing orchestration + self-hosting.
• Diminishing returns at the frontier
  • The study shows a nearly flat frontier: the last few percentage points of accuracy can cost disproportionately more. Orchestration often yields larger relative gains for weaker models; for top models, marginal gains are smaller and more expensive per point.
• Product and deployment trade-offs
  • Manager scaffolds increase monetary cost, latency, and system complexity (more calls, state management, monitoring), but can lower model-cost exposure (less need to buy frontier model access).
  • Organizations should weigh:
    • marginal API cost per solved problem,
    • latency and reliability impacts,
    • operational overhead of maintaining a manager/ledger system,
    • whether provider caps/behaviour make orchestration effective (e.g., high cap hits on single calls create rescue potential).
• Strategic implications for model providers and open-source ecosystem
  • Providers that price long-running in/out tokens cheaply or raise caps change the value proposition of third-party orchestration.
  • The competitiveness of self-hosted orchestration strengthens the business case for open weight models + orchestration stacks for cost-sensitive adopters.
• Research & evaluation recommendation
  • Comparing architectures should present token/budgeted-cost alongside accuracy; gains from multi-agent systems may be accounted for in dollars/tokens, not just raw pass@1.

Summary: Zero-shot, ledger-based self-orchestration is a practical, often cost-effective method to boost coding performance of LLMs without model retraining. It is most valuable where single-call baselines are weak or subject to truncation, and it shifts the economic trade-off from model-size upgrades toward engineering investment in orchestration and state management — but it increases token consumption and operational complexity, and benefits are model- and provider-dependent.