← Papers

The software that wraps AI agents, not the protocol it uses, determines runtime token costs: across seven scaffoldings and five models a single task’s cost varied hugely by scaffolding (up to 139× for one model), while MCP vs CLI comparisons were inconsistent and sometimes irrelevant because agents often ignored the offered interface; MCP runs also carried a higher share of wasted spending when they failed.

The Scaffolding Matters More Than the Interface: A Controlled Comparison of MCP and CLI Tool Use Across Seven Agent Scaffoldings, Five Language Models, and One Software Task

Marc Alier Forment, María José Casañ Guerrero, Francisco José García-Peñalvo, Juanan Pereira · August 09, 2026

arxiv quasi_experimental medium evidence 7/10 relevance Full text usable extracted full text Source PDF

Structured author observations

Linked only from stored provider relations; the raw author line above is never matched by name.

Arxiv

Latest observation: August 11, 2026

Marc Alier Forment unresolved corpus identity
María José Casañ Guerrero unresolved corpus identity
Francisco José García-Peñalvo unresolved corpus identity
Juanan Pereira unresolved corpus identity

Semantic Scholar

Latest observation: August 25, 2026

M. Forment provider ID
M. Guerrero provider ID
F. García-Peñalvo provider ID
Juanan Pereira provider ID

In a controlled experiment on a six-step git task, the choice of agent scaffolding drove far larger token-cost variation than whether tools were exposed via MCP or a command-line interface, with scaffolding differences producing up to 139× variation for one model and MCP-to-CLI cost ratios spanning 0.43×–29× across paired comparisons.

Citation observations

Cumulative provider counts captured on specific dates; providers are never combined.

0 cumulative citations

Semantic Scholar · Observed August 25, 2026

View corpus context

How much an AI coding agent costs to run can depend more on the agent scaffolding that drives it than on the interface through which it reaches its tools. We set out to measure the cost of tool use over the Model Context Protocol (MCP) against tool use over an ordinary command-line interface (CLI), a difference on which published estimates disagree by more than an order of magnitude while resting on practitioner reports that cannot be reproduced. We ran one fixed software task -- six operations against a private online git repository -- across seven agent scaffoldings and five language models, and we verified completion by inspecting the repository state rather than trusting the agent's self-report. The dominant effect was the scaffolding. Two of the seven ship no MCP support at all; they completed every run using only the CLI, which shows that MCP is unnecessary for this class of work, and they were 5.0x to 28x cheaper than the five scaffoldings that do support MCP, comparing CLI runs alone with no MCP server attached anywhere. The effect was largest for a small 27-billion-parameter model running locally, whose cost varied 139x across scaffoldings while it completed the task under all of them. The comparison we set out to make proved unstable: thirteen strictly paired MCP-to-CLI ratios span 0.43x to 29x, with outliers on both sides. The two interfaces separate on the cost of failure, where 12.9 per cent of the money spent on MCP runs bought no completed work against 2.2 per cent on CLI runs, but not on its frequency: failures were equally common in both, in the original runs and in their repetitions alike. Agents frequently ignored the interface they were assigned, so comparisons that do not verify actual behaviour measure an unknown mixture. The harness, the task, the verification and the complete dataset are released as open source.

Summary

Main Finding

The software harness (agent scaffolding) matters far more for token cost than whether model tool access uses the Model Context Protocol (MCP) or a command-line interface (CLI). Differences in scaffolding implementation produced order‑of‑magnitude cost swings; the MCP-vs-CLI effect is real but unstable and heavily conditional on scaffolding, caching and agent behaviour.

Key Points

Dominant effect: scaffolding choice. For the same task and CLI-only runs, two minimal scaffoldings (pi, Tau) that provide no MCP support were 5.0× to 28× cheaper than the five scaffoldings that do support MCP.
Large scaffolding variability: a single locally run 27B dense model (qwen3.6-27b) showed a 139× cost range across scaffoldings while completing the task under all of them.
MCP vs CLI within-scaffolding comparison is unstable: 13 strictly paired MCP-to-CLI ratios ranged from 0.43× (MCP cheaper) to 29× (MCP much costlier). Outliers exist in both directions.
Cost of failure matters more for MCP: 12.9% of money spent on MCP runs bought no completed work, vs 2.2% for CLI runs. Failures were equally frequent in both arms, but failed runs consumed substantially more tokens (median failed run 181,806 input tokens vs 81,510 for median completed run).
Agents often ignored their assigned interface; some scaffoldings called CLI tools even when MCP was attached. Measuring declared availability without verifying actual tool calls can misattribute costs.
Practicalities: MCP catalogues are sent in full on each request (44 GitHub-related tools in the study’s MCP server), inflating input tokens unless caching or other mitigations apply. Caching and delivery method influence the protocol penalty.
Reproducibility: the authors open-sourced the harness, task, verifier and full dataset (GitHub + Zenodo DOI).

Data & Methods

Fixed task (single-run goal): six operations against a private GitHub repo — locate an issue, create a branch, apply a patch, commit, open a PR, and report file count. Completion was verified by the repository state (four independent checks: branch exists, file patched, PR opened, file count correct). Repo reset before every run.
Seven scaffoldings tested: Claude Code, OpenAI Codex, qwen-code, Hermes, opencode (all MCP-capable), plus two minimal scaffoldings with no MCP (pi, Tau) as controls.
Two experimental arms per MCP-capable scaffolding: MCP arm (official GitHub MCP server attached, 44 tools) vs CLI arm (no MCP server; shell + gh).
Models: five language models (two OpenAI-hosted, one Anthropic-hosted, two open-weight models served locally). Not every scaffolding used every model; local models were also repeated to measure run-to-run variation.
Instrumentation: all requests routed through a local proxy that recorded input tokens, cached tokens, tool calls, and schemas per request. Two subscription cells bypassed the proxy due to credential limits; proxy and self-reported usage agreed where both existed.
Costing: token costs modelled using OpenRouter public per-token rates (snapshot 3 Aug 2026). All cost figures are modelled prices (not actual bills); caching billed as input but cache share reported separately.
Corrections & repetition: authors fixed a configuration credential error (re-ran affected cells) and repeated local-model cells three times to assess variance (90 extra runs produced, 61 completed).
Data/publication: harness, verifiers, and full run-by-run dataset released for reproducibility (GitHub + Zenodo DOI).

Implications for AI Economics

Be skeptical of single-number protocol penalties. The commonly cited MCP token-penalty figures (e.g., "35×") are not universal; the penalty depends strongly on scaffolding implementation, catalogue size, call count, caching, and actual agent behaviour.
Procurement and cost modelling must include scaffolding and implementation choices, not just model and interface. Two deployments using the same model and the same protocol can differ by orders of magnitude in token cost because of scaffold design.
Failure cost is economically significant. Systems that increase token consumption on failed attempts (or make failures more expensive) raise operational spending even if failure frequency is unchanged. Benchmarks and SLAs should track waste rates and cost-per-successful-job, not just success rates.
Small models are more sensitive to interface conventions. If training exposure to MCP-style schemas is sparse, smaller models may perform worse (or require more back-and-forth), amplifying token costs — an important consideration when choosing model size vs. harness complexity.
Design trade-off: general-purpose scaffolding (rich capabilities/MCP) increases flexibility but can raise running costs compared with minimal, task-specific CLI-based agents. "Harvesting" agents into lean, scriptable tools (per the authors’ prior AAWD argument) can provide substantial cost savings for narrowly specified workloads.
Measurement best practices: (1) verify actual in-system behavior (tool calls, repository state) rather than trusting agent reports; (2) route instrumentation to capture all sub-agent calls; (3) publish harnesses and datasets to enable reproducible cost comparisons.
Policy and benchmarking: standardized benchmarks for agent tool use should fix scaffolding implementations, caching policies and verification methods. Otherwise, cross-study comparisons will conflate protocol, scaffolding, and task differences.

If you want, I can extract the full numeric tables (per-scaffolding costs, paired MCP/CLI ratios, failure-cost shares, or the repetition variance) from the published dataset and present them in a concise table for budgeting or modelling.

Assessment

Paper Typequasi_experimental Evidence Strengthmedium — The study is a careful, reproducible controlled experiment with direct measurements (tokens, tool calls, repository verification) and open data and code, which gives strong internal validity for the specific task and scaffoldings tested. Its external validity is limited: a single software task, a selected set of scaffoldings and models (not including largest sparse models or many proprietary stacks), pricing is modelled using one price list, and hosted configurations were not fully repeated due to budget — so conclusions about general MCP vs CLI costs across all workloads and deployments remain provisional. Methods Rigorhigh — Design choices target common sources of bias: fixed task and verifier avoid reliance on agent self-report, a proxy records all requests (preventing under-reporting by delegated sub-agents), MCP/CLI arms were isolated via configuration and credentials, and problematic runs were identified, corrected, and repeated; the full harness, dataset and analysis are published for reproducibility. Remaining issues are a single task, limited repeated runs for hosted models, and some unavoidable credential/measurement exceptions that are transparently documented. SampleExperimental matrix of seven agent scaffoldings (Claude Code, OpenAI Codex, qwen-code, Hermes, opencode, pi, Tau), five language models (two OpenAI-hosted, one Anthropic subscription, two local open-weight models), and one fixed private GitHub repository task consisting of six operations (locate issue, create branch, apply patch, commit, open pull request, report directory file count). Each MCP-capable scaffolding was run in two arms (MCP attached vs CLI only); all requests were proxied and logged; locally served models were repeated three times per configuration to assess variability. Costs are modelled using OpenRouter per-token list rates; repository state was reset between runs and completion was verified via GitHub API on four durable checks. Themesorg_design productivity IdentificationControlled factorial lab-style experiment: the authors hold the software task fixed (six git operations against a private GitHub repo) and systematically vary three factors — agent scaffolding (7 implementations), tool interface (MCP attached vs CLI only) and language model (5 models, including hosted and local open-weight). They isolate arms (MCP vs CLI) via configuration and credential separation, route all requests through a proxy to measure input tokens/caching/tool calls, verify completion by inspecting repository state (four durable conditions) rather than agent self-report, and repeat locally-served model runs to assess run-to-run variation; costs are computed only over successfully completed runs. GeneralizabilitySingle software task (six git operations) — may not generalize to other tasks or workflows with different call patterns or token dynamics, Limited set of scaffoldings (seven) and models (five) — results may differ on other or newer scaffoldings, proprietary agent architectures, or much larger sparse models, Pricing is modelled using a single price list (OpenRouter) and counterfactual pricing for local models, so monetary magnitudes depend on provider-specific rates and discounts, Hosted configurations were not repeated at scale (budget limits), so stability estimates are stronger for local models than for all cells, Private GitHub fixture may not capture real-world complexities (larger repos, network latency, rate limits, different authentication setups)

Claims (11)

Claim	Direction	Outcome	Confidence & Evidence	Details
The agent scaffolding had a larger and more stable effect on tool-use cost than whether tools were accessed through MCP or the CLI. Organizational Efficiency	mixed	Input-token cost of completing the software task	Reading fidelity high Study strength medium	n=7 0.48
For this software task, MCP was unnecessary: the two scaffoldings without MCP support completed every run using only the CLI. Organizational Efficiency	null_result	Successful completion of the six-operation repository task	Reading fidelity high Study strength medium	n=2 completed every run 0.48
The two CLI-only scaffoldings were 5.0× to 28× cheaper than the five scaffoldings that supported MCP, when comparing CLI runs without any MCP server attached. Organizational Efficiency	negative	Modelled cost of completed software-task runs	Reading fidelity high Study strength medium	n=7 5.0× to 28× cheaper 0.48
For the locally run 27-billion-parameter model, cost varied by a factor of 139× across scaffoldings, even though the model completed the task under every scaffolding. Organizational Efficiency	mixed	Modelled input-token cost and task completion across agent scaffoldings	Reading fidelity high Study strength medium	n=7 139× variation 0.48
Strictly paired MCP-to-CLI cost ratios were highly unstable, ranging from 0.43× to 29× across 13 comparisons. Organizational Efficiency	mixed	Ratio of MCP-run cost to CLI-run cost	Reading fidelity high Study strength medium	n=13 0.43× to 29× 0.48
MCP runs had a substantially larger share of spending on failed runs than CLI runs: 12.9% versus 2.2%. Organizational Efficiency	positive	Share of total spending associated with runs that produced no completed work	Reading fidelity high Study strength medium	12.9% of MCP spending versus 2.2% of CLI spending 0.48
MCP and CLI failures occurred with equal frequency in both the original runs and the repetitions. Error Rate	null_result	Frequency of failure to complete the verified repository task	Reading fidelity high Study strength medium	equally common 0.48
Failed runs consumed more than twice as many input tokens as completed runs at the median: 181,806 versus 81,510. Organizational Efficiency	positive	Median input-token consumption by failed versus completed runs	Reading fidelity high Study strength medium	181,806 input tokens versus 81,510 input tokens 0.48
Every failed run cost between 4.7 and 22.9 times as much as a completed run on a scaffolding with no MCP client. Organizational Efficiency	positive	Cost of failed versus completed software-task runs	Reading fidelity high Study strength medium	4.7 to 22.9 times 0.48
The repeated local-model dataset contained 90 runs, of which 70 executed and 61 completed the task. Organizational Efficiency	other	Execution and verified completion of repeated agent runs	Reading fidelity high Study strength high	n=90 70 executed; 61 completed 0.8
Agents frequently used an interface different from the one assigned to them, so comparisons based only on assigned interface can measure an unknown mixture of behaviors. Task Allocation	negative	Conformance between assigned tool interface and actual tool use	Reading fidelity high Study strength medium	not reported 0.48