Workflows that let LLMs act as agents repeatedly ping-pong between CPU and GPU, producing bursty, uneven demand that wastes conventional servers; an adaptive server prototype (Agora) reclaims idle CPU/GPU capacity and role-aware scheduling to boost utilization and throughput while protecting agent tail latency.

Main Finding

Agentic AI—workloads where LLMs run in iterative control loops, call external tools, and coordinate across agents—produces highly fragmented, bursty, and heterogeneous CPU/GPU demand that breaks assumptions behind uniform server designs. Optimizing servers for these workloads (via dynamic CPU/GPU harvesting, role-aware core pooling, and consolidation-aware GPU management) materially increases utilization and throughput while preserving tail latency.

Key Points

• Taxonomy (platform-relevant axes)
  • Orchestration: host-orchestrated vs LLM-orchestrated (who decides next actions).
  • Execution structure: sequential vs parallel (how steps are arranged over time).
  • Model composition: homogeneous vs heterogeneous (whether agents share the same model family).
• Host roles expand beyond a simple scheduler:
  • Scheduler: dispatches inference work.
  • Orchestrator: routes messages, advances workflow state.
  • Runner: executes agent logic and invokes tools.
• Fragmentation and heterogeneity
  • Each user request expands into many LLM inferences, tool calls, and orchestration decisions, repeatedly crossing CPU–GPU boundary.
  • Tool execution is a first-class component of end-to-end latency; ~27%+ of requests spend as much or more time in tools than inference.
  • Execution structure causes long low-utilization periods punctuated by sharp bursts (stage boundaries, tool fan-out), producing high peak-to-mean demand.
  • Model composition can unevenly use GPUs (some saturated, others idle), especially with heterogeneous models.
• Architectural mismatches with uniform servers
• Fragmented execution strands CPU and GPU capacity: average utilization low, but short bursts cause near-saturation.
• Homogeneous core pools are inefficient: different host roles have distinct resource profiles.
• High multiplexing on shared cores degrades microarchitectural locality and increases coordination costs.
• Agora prototype (case studies and mechanisms)
  • CPU harvesting: dynamically uses idle cores for co-located throughput work while protecting agent tail latency.
  • GPU harvesting/consolidation: place more agents per GPU, oversubscribe memory, prefetch next-agent state to hide swap latency.
  • Role-aware pooling and affinity scheduling: isolate and right-size control plane vs bursty runner pool, pin tasks to preserve cache/branch locality.
• Quantified improvements (Agora vs static baseline)
  • CPU harvesting: recovers 95% of co-located workload’s standalone throughput; host CPU utilization +30%; agent slowdown <3%.
  • GPU harvesting: frees ~1/3 of GPUs; generation throughput +82%; tail latency cut by 2.5×.
  • Role-aware pooling: tool CPU demand reduced up to 46%; worst-case tool latency improved up to 13%; serving throughput retained at 99%.

Data & Methods

• Two complementary studies:
• Production fleet study (Microsoft Azure) - 24-hour trace of production agentic requests across the hyperscaler. - Instrumented to attribute time, tool activity, and CPU metrics to scheduler, orchestrator, runner roles. - Observed request lifetimes, tool mixes, and distribution of time in LLM vs tool execution.
• Controlled experiments - Representative open-source frameworks exercised points in the taxonomy: - SWE-Agent (single-agent, sequential, tool-heavy coding workflow) - Trae (sequential coding workflow with long tool chains and three model endpoints) - CORAL (four-agent parallel homogeneous evolutionary workflow) - Owl (heterogeneous workforce: coordinator + six text roles + vision/speech models) - Hardware: 96-core AMD EPYC 7V12 CPU, 8 × NVIDIA A100 GPUs. - Software: one vLLM instance per agent role; concurrency varied (1–32 tasks). - Metrics collected: CPU utilization over time, GPU utilization and residency, end-to-end and tail latency, throughput, microarchitectural effects (cache/branch locality), tool latency.
• Analysis focused on temporal patterns (bursts vs slack), CPU/GPU crossing, role-specific loads, and efficacy of the proposed server mechanisms.

Implications for AI Economics

1. Capacity planning and utilization
   • Agentic workloads are forecasted to be among the fastest-growing datacenter consumers. Their bursty, fragmented nature implies naive static overprovisioning is expensive: providers must balance high peak capacity (to meet bursts/SLA) vs low average utilization.
   • Techniques like Agora that reclaim idle CPU/GPU capacity can significantly reduce effective required capacity (e.g., freeing ~1/3 of GPUs), lowering capital expenditure (CAPEX) per unit of delivered agentic throughput.
2. Cost per request and pricing models
   • Traditional per-request or per-inference pricing underestimates the host-side and orchestration costs of agentic workflows (tool calls and orchestrator CPU are material).
   • Providers could shift to pricing that captures workflow complexity (per-workflow pricing, per-stage, or hybrid models that include host orchestration premiums) or offer specialized "agentic-optimized" instance types with different rates.
   • Improved server-side efficiency (consolidation, role-aware scheduling) enables lower marginal costs that providers can monetize (competitive pricing) or use to increase margins.
3. Instance/product differentiation and market opportunity
   • A clear opportunity exists to offer instance families or managed services tuned for agentic AI (e.g., agent-optimized VMs with role-partitioned cores, aggressive GPU consolidation, or hardware featuring asymmetric cores).
   • Vendors who invest in agentic-optimized infrastructure can capture customers needing better cost/latency tradeoffs; differentiation can be a competitive moat.
4. SLA design and tail-risk pricing
   • Agentic workflows are sensitive to tool-induced spikes and tail latency. Providers may need to create (and price) different SLAs: best-effort lower-cost options (with consolidation) vs premium low-tail-latency guarantees (reserved/isolated resources).
   • Customers with strict latency constraints may prefer dedicated or affinity-managed deployments and will pay a premium for isolation.
5. Multi-tenancy and interference externalities
   • Multiplexing many agents on shared cores degrades microarchitectural locality; unmanaged multi-tenancy raises tail latencies and unpredictability.
   • Market designs may include chargebacks for interference or offer isolation options (higher-priced) to avoid cross-tenant degradation.
6. Hardware investment signals
   • Findings motivate investment in heterogeneous server architectures:
     • Mix of efficiency cores (for lightweight orchestration) and high-performance cores (for tool-heavy runners).
     • Hardware support for rapid context switching, offloaded scheduling/context management, and better GPU memory management (to reduce swap latency when oversubscribing).
   • Cloud providers must weigh CAPEX for next-gen servers against software-level improvements—both reduce costs but differ in investment horizon and vendor lock-in.
7. Model composition effects on economics
   • Homogeneous deployments (single base model) enable resident model sharing and batching—lower operating costs. Heterogeneous workflows increase GPU memory footprint and complicate consolidation, raising cost per workflow.
   • Providers can incentivize customers toward homogeneous stacks (discounts, technical guidance) to improve packing efficiency.
8. Operational complexity vs cost savings
   • Techniques like oversubscribing GPU memory and prefetching require sophisticated runtime logic and tuning; they add operational complexity and potential risk (incorrect tuning risks SLA violations).
   • Providers must evaluate trade-offs between cost savings and engineering/operational overhead; managed services that encapsulate that complexity can command premiums.
9. Impacts on pricing & product strategy for cloud providers
   • Short-term: offer software-based optimizations (Agora-like orchestration) as value-added services to reduce customers’ bills and increase utilization.
   • Medium-term: introduce agentic-optimized instance classes and role-aware SLAs.
   • Long-term: hardware differentiation (heterogeneous cores, on-chip support for orchestration) could justify new premium tiers and long-lived capital investments.
10. For customers / firms building agentic applications - Design for homogeneous models when possible to reduce hosting costs. - Prefer frameworks and deployments that expose orchestration semantics to the runtime (host-orchestrated) to enable prefetching and better scheduling. - Consider paying for managed agentic infrastructure to avoid building solution-specific schedulers and capacity planners.

In short: agentic AI shifts cost drivers from pure accelerator compute toward complex host–accelerator interactions and coordination. Operational and architectural innovations (software + targeted hardware) can substantially lower the effective cost of agentic workloads; these techniques create both opportunities (new differentiated products, lower unit costs) and trade-offs (operational complexity, new SLA tiers) for cloud providers and customers alike.