0 cumulative citations
View corpus contextScylla benchmarks the dollar cost of getting one correct coding solution and finds that more complex agent architectures frequently raise costs with limited quality gains. Demonstrated on Claude Sonnet 4.5 with same-vendor LLM judges, the framework enables reproducible ablations to pinpoint which components actually matter.
Taming Scylla: Understanding the multi-headed agentic daemon of the coding seas
Scylla is a reproducible, model-agnostic evaluation framework that uses structured ablations and a Cost-of-Pass metric to show that added architectural complexity in agentic coding tools often increases monetary cost without proportionate quality improvements.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
LLM-based tools are automating more software development tasks at a rapid pace, but there is no rigorous way to evaluate how different architectural choices -- prompts, skills, tools, multi-agent setups -- materially affect both capability and cost. This paper introduces Scylla, an evaluation framework for benchmarking agentic coding tools through structured ablation studies that uses seven testing tiers (T0-T6) progressively adding complexity to isolate what directly influences results and how. The key metric is Cost-of-Pass (CoP): the expected dollar cost to get one correct solution, which directly quantifies the trade-off between complexity and efficiency. The framework is model-agnostic, designed to work with any CLI tool; this paper demonstrates it with Claude Sonnet 4.5, using multiple LLM judges (Opus 4.5, Sonnet 4.5, Haiku 4.5) from the same vendor for evaluation consensus, where judges score results using direct tests, human-designed LLM-evaluated rubrics, and qualitative assessment. The result is a reproducible framework that quantifies trade-offs between agent complexity and actual outcomes, suggesting that architectural complexity does not always improve quality.
Summary
Main Finding
Scylla is a reproducible, model‑agnostic evaluation framework that quantifies the dollar trade-off between architectural complexity and delivered coding capability. Using a Cost‑of‑Pass (CoP) metric — the expected dollar cost to obtain one correct solution — the framework’s ablation tiers (T0–T6) show that added agentic complexity (tools, skills, multi‑agent orchestration) often increases cost without reliably improving quality; careful hybrid designs can, however, achieve a Frontier CoP (minimum observed CoP) by selectively combining features rather than maximizing them.
Key Points
- Cost‑of‑Pass (CoP): primary economic metric — expected $ cost to get one correct/pass solution. Directly captures cost versus capability tradeoffs.
- Tiered ablation design (T0–T6): seven progressively complex agent architectures tested independently to isolate contributions of prompts, skills, tools, delegation, hierarchy, and “everything on.”
- T0: baseline prompts (including empty system prompt)
- T1: skills baked into prompts
- T2: external tools / tool schemas
- T3: flat multi‑agent delegation
- T4: hierarchical multi‑agent orchestration
- T5: hybrid optimized combinations
- T6: maximum configuration (all features)
- Dry‑run (Hello World) result: all tiers achieved similar high quality (grade A, scores 0.943–0.983) while costs varied 3.8× ($0.065 for a T5 hybrid up to $0.247 for T6 super). Shows cost differences persist even when quality converges.
- Token efficiency chasm: skills (prompt‑based knowledge) are token‑efficient; tool schemas inflate token usage and thus cost.
- Judges and rubric: outputs judged by three LLMs (Opus 4.5, Sonnet 4.5, Haiku 4.5) using weighted categories — Functional Correctness (35%), Code Quality (20%), Proportionality (15%), Build Pipeline (10%), Overall Quality (20%); Sfinal computed as weighted normalized score; pass threshold = 0.60 (Grade B).
- Architecture: adapter pattern for model‑agnostic CLI evaluation, reproducibility via git worktrees, atomic checkpoints, parallel execution, token extraction and vendor pricing tables to compute costs.
- Hypotheses framed for future work: (H1) some tasks better solved via subagents/tools/skills independent of context needs; (H2) prompt complexity effects vary (KISS vs inverse KISS depending on in‑distribution vs out‑of‑distribution tasks).
Data & Methods
- Framework: Scylla executes structured ablation studies across 113 subtests drawn from 7 tiers; designed for CLI agentic tools (demoed with Claude Code).
- Reproducibility and isolation: each test runs in isolated git worktrees at pinned commits; configuration (CLAUDE.md, skills, tools) injected per workspace; checkpoints saved after runs.
- Adapter layer: BaseAdapter interface allows plugging different agent CLIs; current implementation is ClaudeCodeAdapter which runs the CLI, captures stdout/stderr, parses token usage, and applies vendor pricing.
- Judges: three LLM judges (Anthropic models of varying capability) receive the agent output and score across the 5 rubric categories. Deduction scale calibrates subjective deductions.
- Scoring: Sfinal = sum_i (w_i * P_achieved_i / P_max_i). Grade mapping: S >= .80 = A, >= .60 = B (pass), etc.
- Example test (Test‑001 Hello World): trivial task used as pipe‑cleaner; expected all tiers to pass; showed equivalent quality but divergent costs (T5 frontier cost).
- Cost calculation: token extraction from CLI outputs + vendor pricing tables to compute per‑run $ cost; CoP derived from cost and pass probability.
- Limitations acknowledged in paper:
- Tests target production CLI tools (black‑box vendor stack), not raw model weights — vendor pre/post‑processing and hidden orchestration can affect outcomes.
- Judges are vendor models from the same family — possible correlated biases.
- Demonstration uses a trivial task; broader conclusions need larger, varied workloads.
- Some hooks not language‑triggered (deterministic vendor features) were not evaluated.
Implications for AI Economics
- Use CoP as an operational economic KPI: CoP directly links engineering choices to per‑successful‑unit marginal cost, enabling ROI comparisons across architectures and features.
- Diminishing returns and over‑engineering: adding skills, tools, or multi‑agent orchestration increases fixed and marginal costs (tokens, orchestration loops) and can produce little or no quality gain — firms should avoid “everything on” by default.
- Token efficiency matters economically: architectures that encode expertise as compact prompt skills (lower token footprint) can be materially cheaper per pass than schema‑heavy tool approaches despite similar output quality.
- Product design & procurement: product managers and procurement can use Scylla‑style ablations to decide which agentic features to pay for (or request from vendors) based on CoP reductions, not just capability claims.
- Vendor pricing & competition: transparent CoP benchmarking can pressure vendors to optimize token efficiency and make tradeoff choices visible; vendors that reduce orchestration token overhead gain competitive pricing advantage.
- Cost composition & scaling: orchestration/hierarchy introduce coordination overheads (tokens, repeated runs) that scale differently than per‑query model costs; firms must consider marginal vs fixed cost when scaling agentic systems.
- Policy & governance: CoP and reproducible ablation frameworks provide a quantitative basis for procurement standards, auditability, and cost‑transparency requirements for deployed agentic tools.
- Research & investment prioritization: empirical testing (not intuition) should guide investment in skills, tools, or multi‑agent systems. Focus on hybrid designs that combine the smallest subset of features that achieve Frontier CoP for target workloads.
- Next empirical steps: apply Scylla to realistic, diverse coding tasks; cross‑vendor judge and agent combinations; quantify how CoP varies with task difficulty, OOD tasks (testing H2), and real-world multi‑step pipelines to derive generalized supply curves for agentic coding services.
If you want, I can: - Extract a concise checklist product teams can run to apply CoP benchmarking to their own agentic features, or - Draft a short experimental plan to apply Scylla to a mid‑sized real coding task set (e.g., GitHub issues) to estimate enterprise‑scale CoP.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper provides systematic, controlled comparisons (ablation tiers) and a clear economic metric (Cost-of-Pass), demonstrating meaningful differences across architectures; however, evidence is limited to a single vendor/model family and judge pool, without external human ground-truth or exogenous identification, restricting confidence that results generalize or imply causal effects beyond the tested setup.
Methods Rigormedium — Methodological strengths include a structured ablation design, multiple evaluation modes (direct tests, human-designed rubrics evaluated by LLM judges), and a reproducible CLI-based framework; weaknesses include reliance on same-vendor LLM judges (risking consistent bias), limited description of task sampling and sample size in the demonstration, and no randomized experiment or external human validation reported.
SampleDemonstration uses an agentic coding benchmark run via CLI on Claude Sonnet 4.5 as the primary agent, evaluated by multiple same-vendor LLM judges (Opus 4.5, Sonnet 4.5, Haiku 4.5) using direct automated tests, human-designed rubrics assessed by LLMs, and qualitative review; tasks are coding problems organized into seven increasing-complexity tiers (T0–T6); costs calculated from API/pricing to compute Cost-of-Pass.
Themesproductivity human_ai_collab
IdentificationStructured ablation across seven testing tiers (T0–T6) comparing agentic coding architectures and measuring outcomes with a single summary metric (Cost-of-Pass) to isolate the marginal contribution of design choices; no randomized assignment or exogenous variation for causal identification.
GeneralizabilityResults demonstrated on a single vendor/model family (Claude Sonnet 4.5) — may not generalize to other LLMs or open-source models., All judges are from the same vendor, introducing evaluation bias and reducing external validity., Benchmark focuses on coding tasks executed via CLI; findings may not transfer to non-coding or real-world large codebase workflows., Cost-of-Pass depends on vendor pricing and interface (API/CLI); different pricing models or resource constraints will change absolute cost conclusions., Task selection, problem difficulty distribution, and scale of experiments (sample size) are not shown to cover enterprise diversity, limiting broad applicability.
Claims (6)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| LLM-based tools are automating more software development tasks at a rapid pace. Developer Productivity | positive | degree of automation of software development tasks |
Reading fidelity
high
Study strength
speculative
|
not reported
|
| There is no rigorous way to evaluate how different architectural choices -- prompts, skills, tools, multi-agent setups -- materially affect both capability and cost. Other | negative | availability/rigor of evaluation methods for architectural choices (prompts, skills, tools, multi-agent setups) |
Reading fidelity
high
Study strength
low
|
not reported
|
| This paper introduces Scylla, an evaluation framework for benchmarking agentic coding tools through structured ablation studies that uses seven testing tiers (T0-T6) progressively adding complexity to isolate what directly influences results and how. Other | positive | ability to benchmark agentic coding tools via structured ablation (progressive T0-T6 tiers) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| The key metric is Cost-of-Pass (CoP): the expected dollar cost to get one correct solution, which directly quantifies the trade-off between complexity and efficiency. Other | positive | Cost-of-Pass (expected dollar cost to obtain one correct solution) |
Reading fidelity
high
Study strength
medium
|
not reported
|
| The framework is model-agnostic, designed to work with any CLI tool; this paper demonstrates it with Claude Sonnet 4.5, using multiple LLM judges (Opus 4.5, Sonnet 4.5, Haiku 4.5) from the same vendor for evaluation consensus, where judges score results using direct tests, human-designed LLM-evaluated rubrics, and qualitative assessment. Other | positive | applicability of the framework across models/CLI tools and evaluation process using multiple LLM judges |
Reading fidelity
high
Study strength
medium
|
n=3
|
| The result is a reproducible framework that quantifies trade-offs between agent complexity and actual outcomes, suggesting that architectural complexity does not always improve quality. Output Quality | negative | relationship between architectural complexity and output quality (i.e., whether increased complexity improves quality) |
Reading fidelity
medium
Study strength
medium
|
not reported
|