Long-lived, tool-using LLM sessions shift bottlenecks away from inference toward tools, state, and communication; simple, workload-informed changes—task-aware scheduling, co-location, state offloading and caching—cut latency by up to 40%, slash memory footprints >4×, and eliminate a large share of redundant external calls.

Main Finding

Agentic workloads—long-running LLM-driven sessions that orchestrate tools, environments, and persistent state—fundamentally change the cost, latency, and resource profile of AI serving. Model inference is often not the dominant cost: tools, sandboxes, vector DBs, network, and session state frequently dominate latency, memory, and monetary cost. Efficient, economical agent serving therefore requires coordinated management across models, tools, state, and communication rather than model-centric optimization alone.

Key Points

• Scope and tool: The authors introduce AgentSysBench, a benchmark + measurement toolkit with 10 representative agentic applications (RAG, HuggingGPT, DeepResearch, Mini-SWE, Codex, WebAgent, GUIAgent, Claude Code, Openclaw, Pi-AutoR) and a modular serving stack that exposes hardware (H), component-serving mechanisms (C), and deployment architecture (A).
• Measurement scale: Controlled experiments covered 4,641 requests, 64,924 LLM calls, 118,274 tool calls; production traces covered 178,799 sessions in one day.
• Six workload properties that distinguish agentic workloads from conventional LLM inference:
• Heavyweight, stateful execution: non-LLM components dominate latency in 5/10 apps; sandbox working-set memory can peak at ~28 GB per session.
• Cross-stack heterogeneity: components have divergent resource affinities (GPU-bound inference, memory-bound retrieval/vector DB, CPU-bound sandboxes, network-bound services); task latencies on a single component can diverge up to 32×.
• Shifting bottlenecks: dominant bottleneck varies across requests, models, tools, and deployments—averages mask this.
• Long idle-but-live intervals: sessions often hold state idle for minutes to hours between steps.
• LLM control-plane tax: auxiliary LLM calls (tool schemas, observations, safety checks) consume context and tokens, crowding out productive compute and adding cost.
• Cross-request redundancy: production traces show heavy repetition (e.g., 27% of unique search queries recur and account for 67.3% of search API calls; 24% of distinct URLs recur and account for 64% of fetches).
• Actionable interventions and measured gains:
  • Task-aware (disaggregated) serving: reduces latency by 29–40%.
  • Communication-aware placement (agent-aware co-location): up to 4.5× improvement.
  • State offloading (cold state to cheaper storage): reduces live memory by 4.6×.
  • Tool-result caching: eliminates 35.2% of redundant search calls and reduces aggregate search latency by 19.3%.

Data & Methods

• Benchmark design goals: representativeness across request distributions (R), tools/environments (T), models/inference policies (M), and orchestration (O); controllable serving factors (H, C, A); unified component-level instrumentation; and complementing controlled runs with production traces.
• Serving stack: modular, containerized components (LLM engines, embedding services, vector DBs, sandboxes, browsers/GUI, tool servers) that can be co-located or disaggregated to study placement and communication effects.
• Instrumentation: per-LLM-call, per-tool-invocation, and per-state-operation logging of latency, resource usage (CPU/GPU/DRAM), data movement, live session footprint, token accounting, and cost attribution.
• Controlled vs production: controlled experiments isolate variables (models, placement, orchestration) to attribute bottlenecks; production traces reveal phenomena that only emerge at scale (idle sessions, cross-request redundancy, realistic arrival patterns).
• Representative models/orchestration: variety of models (Qwen variants, DS-V4-Pro, Kimi etc.), orchestration patterns (pipeline, plan-exec, ReAct, branching/looping), and arrival processes (one-by-one, Poisson, trace-driven).
• Key quantitative observations (examples):
  • Non-LLM components dominate latency in 5/10 apps.
  • Sandbox per-session working-set memory peaks ≈28 GB.
  • Per-task latency divergence up to 32× across similar-looking tasks.
  • Controlled experiments: 4,641 requests, 64,924 LLM calls, 118,274 tool calls.
  • Production: 178,799 sessions in one day; search-query redundancy statistics noted above.
  • Optimization results: 29–40% latency reduction (task-aware serving), up to 4.5× (placement), 4.6× memory reduction (state offload), 35.2% fewer redundant searches / 19.3% search-latency saved (caching).

Implications for AI Economics

• Cost structure shifts beyond token/GPU billing:
  • Non-LLM costs (CPU for sandboxes, DRAM for vector DBs and session state, network egress for web fetches/external APIs) can dominate. Per-request marginal cost accounting must include these resources, not only GPU time or tokens.
  • Idle-but-live sessions tie up memory/disk for long periods; charging only per-second GPU use underestimates provider costs. New pricing/metrics are needed to cover state residency (e.g., per-GB-hour for active session state or tiered storage offload fees).
• Pricing and product design:
  • Providers should offer differentiated SKUs: GPU-only inference instances are insufficient. Offerings that combine/price CPU, memory, persistent session state, network egress, and caching (or provide managed state-offload + resume primitives) will be valuable.
  • Metering redesign: separate “control-plane” token usage (tool schemas, observation tokens) from “productive” tokens in billing or quota systems to encourage efficient tool interfaces and reduce wasteful context use.
  • Caching as a monetizable feature: caching tool results (search results, fetched pages) offers direct cost reductions for both providers and customers by cutting external API calls and latency. Providers can bundle shared caches or charge lower rates for cacheable workloads.
• Resource procurement and capacity planning:
  • Heterogeneous resource demand implies procurement should not be GPU-centric. Investment in memory-rich servers, fast local storage, CPU capacity for sandboxes, and network topology (co-location) yields better cost-performance for agentic workloads.
  • Bottlenecks shift per-request and over time; static capacity planning based on averages leads to inefficiencies or SLO violations. Providers should invest in adaptive scheduling, multi-resource placement, and dynamic elasticity.
• SLA and latency economics:
  • Head-of-line blocking due to heterogeneity (32× latency divergence) implies that single-queue, homogeneous provisioning raises tail latency and increases required over-provisioning to meet SLAs. Multi-queue, task-aware scheduling reduces required capacity and cost to hit the same SLA.
• External API costs and downstream markets:
  • High redundancy in search/web fetches means substantial external-call cost can be avoided via caching/sharing. This affects both cloud provider economics (less egress, fewer billed third-party calls) and marketplace pricing for search/web APIs (providers could offer tiered caching contracts).
• Business model and feature opportunities:
  • New managed services: session-state offloading + resumable execution, cross-request shared caches, agent-aware co-location, and multi-resource instance types aimed at agentic workloads.
  • Pricing models: subscription tiers for long-lived agent sessions, pay-per-resumable-session (covers memory + state management), or hybrid pricing (time on GPU + GB-hour for maintained state + per-tool-call).
• Environmental/energy implications:
  • Long-lived sessions that keep memory/storage hot and interleave many small CPU/network operations may shift energy consumption patterns (less GPU-bound energy, more system/DRAM/network energy). Efficiency gains from offloading/caching can therefore reduce carbon intensity per user task.
• Strategic implication for customers:
  • Application teams should profile end-to-end agentic workloads (tools, state, external calls) rather than optimize only model selection. Caching frequently repeated queries and compressing/control-budgeting tool interfaces offer immediate, low-risk cost reductions.
• Risk to naive per-token pricing:
  • Token-based billing that ignores control-plane taxes and tool-call costs can misprice services—customers may be charged heavily for control tokens that add little productive value; providers may be under-recovering costs for memory/network-bound work.

Takeaway: agentic workloads change the economic calculus of AI serving. Both cloud providers and application teams should move from GPU/token-centric designs to multi-resource-aware, state-conscious, and cache-enabled architectures and pricing models to capture large, practical efficiency gains and to properly internalize the true marginal costs of agentic AI.