Architecture-aware tuning and AMD's AITER runtime determine LLM inference speed on MI325X clusters: MLA models need specific settings and gain from AITER, whereas GQA models leverage KV-cache offloading; despite model-size differences, all models hit a shared memory-bandwidth ceiling that caps throughput at high concurrency.

Main Finding

Architecture-aware serving optimizations are essential for high-throughput, reliable LLM inference on AMD Instinct MI325X GPUs. Performance and operational characteristics depend more on the model’s active parameters, attention design, quantization format, and compatibility with AMD’s AITER kernels and vLLM features (PagedAttention/KV offload) than on total parameter count. On CDNA‑3 (MI325X) the dominant production bottleneck is memory bandwidth: all tested models hit a common throughput saturation point determined by workload and context length, and MLA/MoE models impose distinct deployment constraints that materially affect throughput, variability, and operability.

Key Points

• Cross-architecture benchmark: four frontier models spanning 235B–1T parameters, three architectural families (Dense+GQA, MoE+GQA, MoE+MLA) evaluated on an 8× MI325X (CDNA‑3) cluster using vLLM v0.14.1.
• Architecture-specific constraints:
  • MLA models require PagedAttention block size = 1 on current ROCm and are incompatible with KV-cache offloading.
  • GQA models benefit from standard block sizes and KV-cache offloading.
  • AITER (AMD AI Tensor Engine) is required for competitive MLA/MoE throughput on ROCm; Triton MLA fallback exists but performs substantially worse.
  • AITER has head-count/time-to-rank constraints and must be disabled for incompatible MLA configurations.
• AITER effects:
  • Controlled A/B on Llama‑3.1‑405B (n=5 restarts per condition) shows AITER yields modest 3–5% throughput gains under high concurrency for GQA but increases measurement variability (coefficient of variation) by 2–16×.
  • Large reported AITER speedups (2–3×) are targeted at MoE/MLA kernels rather than general attention acceleration.
• Active parameters per token correlate with throughput:
  • Llama‑3.1‑405B (dense GQA, 405B) and DeepSeek V3.2 (MoE+MLA, 37B active) reached comparable peak text-only throughput: 15,944 and 15,343 tok/s respectively — despite large differences in total size.
  • Qwen3‑VL‑235B (MoE+GQA, 22B active) achieved 47,873 tok/s on vision workloads (includes image tokens) — highest absolute throughput in study but not directly comparable to text-only numbers.
  • Kimi‑K2.5 (1T, 32B active, MoE+MLA) delivered 7,327 tok/s at 500 concurrent requests (INT4 QAT weights) — demonstrating trillion-parameter inference feasibility on MI325X.
• Saturation and reliability:
  • Throughput saturation common across models and workloads, consistent with a memory-bandwidth bottleneck: ~500 concurrent for short sequences (500 in / 200 out workload), ~100–200 concurrent for long sequences (2048 in / 512 out).
  • All models maintained 100% HTTP success rates up to 1,000 concurrent users, processing 18.9M tokens across 17,406 requests without failures.
• Quantization and kernel compatibility matter:
  • Quantization choices were constrained by architecture/kernel compatibility: FP8 used for Llama‑3.1‑405B and DeepSeek V3.2; BF16 for Qwen3‑VL‑235B (vision encoder incompatibility with FP8 block kernels); INT4 QAT for Kimi‑K2.5 (1T).
  • These choices affect memory usage, achievable batch/context sizes, and effective throughput.

Data & Methods

• Hardware and software:
  • Single-node 8× AMD Instinct MI325X (CDNA‑3, gfx942), each GPU: 256 GB HBM3e, 6.0 TB/s; aggregate: 2 TB HBM3e, 48 TB/s.
  • Inference stack: vLLM v0.14.1, ROCm ecosystem, AMD AITER kernels where applicable; Triton fallback evaluated for MLA.
• Models evaluated:
  • Kimi‑K2.5: 1T total, MoE+MLA, 32B active per token, INT4 QAT weights (deployed on 4× MI325X in a specific experiment).
  • DeepSeek V3.2: 685B total, MoE+MLA, 37B active, FP8.
  • Llama‑3.1‑405B: 405B dense, GQA, FP8.
  • Qwen3‑VL‑235B: 235B total, MoE+GQA, 22B active, BF16 (vision encoder constraints).
• Workloads and metrics:
  • Concurrency scaling from single requests up to 1,000 concurrent users.
  • Stress-test workload: 500-token input / 200-token output (used for saturation sweeps, fine-grained from 500 to 1,000 in steps of 50).
  • Long-sequence workload: 2048 in / 512 out to probe longer-context behavior.
  • Measured tokens/sec (tok/s), per-request latency distributions, HTTP-level success (HTTP 200 + valid response structure), and variability (coefficient of variation).
  • AITER ablation: controlled A/B with n=5 independent server restarts per condition on Llama‑3.1‑405B.
  • KV-cache behavior: PagedAttention block-size tuning and offload enabled/disabled tests per architecture.
• Key observed empirical numbers:
  • Llama‑3.1‑405B peak text-only throughput: 15,944 tok/s.
  • DeepSeek V3.2 peak text-only throughput: 15,343 tok/s.
  • Qwen3‑VL‑235B (vision) peak: 47,873 tok/s.
  • Kimi‑K2.5 (1T, INT4 QAT): 7,327 tok/s at 500 concurrent with 100% reliability.
  • Overall stress test: 18.9M tokens across 17,406 requests, 100% success to 1,000 concurrent.

Implications for AI Economics

• Active-parameter-aware model selection lowers cost per token:
  • Operational cost and throughput scale with active parameters per token, not total parameter count. Choosing MoE/GQA designs with low active parameters (or dense models with favorable attention designs) can yield substantially better dollars-per-token throughput while maintaining quality parity for many tasks.
• Hardware choice and software stack compatibility drive cost and risk:
  • MI325X (CDNA‑3) is viable for frontier inference, including trillion-parameter MoE models, when combined with appropriate quantization and AITER kernels. However, kernel/quantization incompatibilities (e.g., FP8/BF16 constraints, AITER head-count restrictions) can force suboptimal quantization or software fallbacks, increasing cost or reducing throughput.
  • Dependence on AITER for MoE/MLA performance introduces operational vendor-software coupling: upgrades, AITER bugs, or head-count constraints can change throughput or force disabling acceleration—raising SRE and engineering costs and SLO risk.
• Diminishing returns from scale due to memory-bandwidth limits:
  • A common throughput saturation tied to memory bandwidth implies adding more GPUs or larger models will hit a ceiling for a given workload/context length. Economic planning should therefore consider memory-bandwidth rather than just peak FLOPs or GPU memory—e.g., investing in accelerators with higher HBM bandwidth or optimizing workloads (shorter contexts, lower active params) can be more cost-effective than brute-force scaling.
• Quantization and kernel compatibility materially affect deployment economics:
  • Compatibility constraints (e.g., vision encoder incompatibility with FP8 kernels) can force higher-precision formats, increasing memory footprint and reducing effective throughput—raising per-token costs. Conversely, successful INT4 QAT or FP8 deployment (when compatible) can make otherwise infeasible large models cost-effective.
• Performance variability increases operational overhead:
  • AITER’s higher measurement variability (2–16× coefficient of variation) complicates SLO management and capacity planning. Systems needing tight latency/throughput bounds will incur additional engineering and provisioning costs (reservations, overprovisioning, or complex autoscaling).
• Multimodal vs text-only economics differ:
  • Vision-inclusive workloads can yield much higher tok/s totals but are not directly comparable to text-only tokens (image tokens, encoder costs). Product teams should account for token-type heterogeneity when estimating throughput, latency, and cost-per-query.
• Engineering complexity and time-to-production cost:
  • The need for architecture-specific configurations (block size, KV offload, AITER enable/disable, quantization path) increases integration and testing time, raising upfront engineering costs and ongoing maintenance burden. Standardized serving configurations are unlikely to be optimal across diverse model families.
• Practical deployment takeaway:
  • For throughput-sensitive, cost-constrained production, prioritize models with low active params per token or architectures compatible with high-efficiency quantization and vendor kernels available on your chosen hardware. Plan for memory-bandwidth saturation, test AITER/other vendor runtime impacts on both mean performance and variability, and include these effects in cost and SLO models.

If you want, I can: - Estimate cost-per-token under sample cloud pricing scenarios for MI325X-equivalent hardware given the measured tok/s; - Produce a checklist for deployment decisions (model architecture, quantization, kernel/runtime compatibility) to minimize cost and operational risk.