Shell-only orchestration outperforms typed tool catalogs in simulated enterprise agent tasks: bash-only agents score substantially higher and use fewer tokens/costs than direct typed-tool interfaces, while programmatic tool calling reduces tokens but usually trails bash in quality; where arbitrary execution is feasible use shell, otherwise prefer catalog-restricted programs for compliance.

Main Finding

Bash-only (shell-based) agents outperform typed-tool-only agents on two realistic enterprise benchmarks while using substantially fewer tokens and often lower estimated inference cost. Adding typed tools or persistent synthesized tools to a bash-capable agent yields no consistent quality gain. Programmatic tool calling (PTC) reduces token use versus direct typed calls but generally underperforms bash alone in both task quality and token/cost efficiency. Practical recommendation: use bash-only when arbitrary execution can be safely isolated; use PTC when security/compliance requires a fixed catalog.

Key Points

• Interfaces compared (same model, prompts, stopping rules):
  • Tool-only (typed catalog calls)
  • Bash+Tool (both shell and typed catalog)
  • Bash (shell-only)
  • Bash+Synthesis (shell + persistent agent-authored tools)
  • Programmatic Tool Calling (PTC): programs over a restricted typed catalog
• Benchmarks and models: TheAgentCompany (software-company workflows) and APEX-Agents (investment banking / consulting / law); evaluated with Opus-4.8 and GPT-5.5.
• Core quantitative outcomes (pooled comparisons):
  • Bash improved score vs Tool-only by ~21.8–24.5 percentage points on TheAgentCompany and ~4.8–7.4 pp on APEX-Agents (model-dependent).
  • Bash used 19–72% fewer total tokens than Tool-only (varies by benchmark/model).
  • Adding typed or synthesized tools to bash produced no detectable pooled score improvement.
  • PTC saved tokens relative to Tool-only but typically scored below Bash and had lower tool-call success rates.
• Task-type sensitivity: bash advantage is larger on TheAgentCompany (coding / cross-application workflows) than on APEX-Agents (document-heavy professional analysis), suggesting flexible execution helps more where many small interactions or cross-application orchestration are needed.
• Model differences: Opus-4.8 tends to favor shell-based execution for higher absolute quality (at higher token cost); GPT-5.5 is more token-efficient for direct typed-tool calls.
• Tool synthesis: agents often create reusable tools and invoke them, but reuse does not reliably improve success rates or task scores; synthesized tools shorten invocations but do not deliver consistent quality benefits.
• Shell behavior: agents compose short shell commands heavily (pipes, subshells) and bundle repeated operations into single shell steps, reducing the number and token-cost of explicit tool calls.

Data & Methods

• Experimental design: controlled interface ablation holding model, task text, base prompt, and stopping criteria constant; interface-specific prompts only for synthesis and PTC. Paired task-model comparisons were used.
• Benchmarks:
  • TheAgentCompany: simulated software company environment; authors constructed a 60-tool typed catalog for the benchmark’s 174 tasks; coworker agents simulated and patched for reliability issues.
  • APEX-Agents: professional analysis across domains; evaluated 480 tasks using an Archipelago snapshot with 20 typed application tools; the code-execution server acted as a guarded Unix shell.
• Models: Opus-4.8 and GPT-5.5 at default reasoning levels.
• PTC implementation: Python programs run in a restricted runtime exposing only the typed catalog functions (no shell or other action channels).
• Bash+Synthesis: persistent directory available; prompts encourage packaging reusable tools for later tasks.
• Metrics:
  • Quality: effort-weighted checkpoint rate (TheAgentCompany) and mean rubric score (APEX-Agents); pass rate (fully solved fraction). Bootstrap 95% CIs reported.
  • Resource use: token counts (input, output, cache-read, total), estimated $/task using GitHub Copilot list rates, wall-clock minutes/task.
  • Operational metrics: tool calls per task, tool-call tokens, call success rates, repetition rates, shell command composition, lines of code, tool-synthesis creation and reuse statistics.
• Sample quantitative examples from Table 1 (illustrative):
  • TheAgentCompany (Opus-4.8): Tool-only score 44.9% -> Bash 69.4%; total tokens/task: Tool-only ~945k -> Bash ~264k; estimated $/task: $1.20 -> $0.37.
  • APEX-Agents (GPT-5.5): Tool-only score 39.8% -> Bash 44.6%; tokens/task: Tool-only ~328k -> Bash ~176k; estimated $/task: $0.52 -> $0.40.
• Analysis slices: per-task paired differences by difficulty proxy (Tool-only call count), tool-call composition and failures, reuse & effectiveness of synthesized tools, PTC vs Tool-only token composition.

Implications for AI Economics

• Cost-effectiveness and operating cost:
  • Shell-based orchestration can meaningfully reduce inference-token consumption and (in these experiments) estimated inference cost per task relative to typed-tool-only interfaces—sometimes dramatically (example Opus TheAgentCompany). Savings scale with task volume, so a per-task token/cost reduction can translate to large operational savings for enterprises running many automated workflows.
  • PTC reduces context (uncached input and cache-read) tokens versus direct typed calls, so when catalog constraints are required, PTC is a cost-efficient alternative relative to naïve tool calling—though still often worse than shell-only when shell is allowed.
• Productivity and task quality:
  • The superior quality of bash-only on workflow-heavy tasks suggests higher end-user utility and lower downstream human revision costs in domains requiring cross-application orchestration or many small operations. That implies higher effective labor substitution per dollar spent in these scenarios.
• Deployment trade-offs and compliance:
  • Arbitrary code execution (bash) increases operational risk (security, data exfiltration, compliance). Where such risks are unacceptable, PTC or typed-catalog approaches provide constrained, auditable action surfaces even if they are somewhat less effective than unrestricted shell.
  • Economic decisions must therefore trade direct inference cost and quality against risk-management costs (sandboxing, auditing, human-in-the-loop review) required to permit shell execution.
• Platform design and product strategy:
  • Enterprise automation platforms should expose multiple interface modes (shell, typed tool calls, programmatic/catalognized programs) and let customers choose by risk profile and workload. Providing turnkey isolation/sandboxing for shell execution would make the most cost-effective mode accessible to more customers.
  • Enterprises should consider in-house A/B testing of interface/model combinations on representative workflows to quantify real-world ROI (quality improvements, token/inference cost, human-revision time) before large-scale adoption.
• Model & future-proofing considerations:
  • Recommendations are model- and time-dependent. Providers and buyers should re-evaluate as models evolve: what is optimal for Opus-4.8 and GPT-5.5 (June 2026 snapshot) may change with newer LLM families or fine-tuned tool-use capabilities.
• Limitations affecting economic extrapolation:
  • Results come from two benchmarks and two models; catalog construction, environment patches, and specific guardrails affect generalizability. Security and engineering costs to safely run shell agents are nontrivial and must be included in economic analyses.
• Bottom-line guidance for decision-makers:
  • If you can safely sandbox arbitrary execution: prioritize shell-based agents for higher quality and lower inference costs on cross-application/coding-style workflows.
  • If you must restrict actions for compliance: use PTC (programs over a fixed typed catalog) to regain some token efficiency and chaining benefits while keeping an auditable, constrained action surface.
  • Run targeted A/B tests to quantify your actual task mix, token cost, revision overhead, and security-cost trade-offs before committing to a single interface at scale.