0 cumulative citations
View corpus contextA tailored flash-memory substrate lets large models run on far fewer accelerator packages: FLINT uses dynamic burst coalescing, off-path refresh, and a compact read-only FTL to turn high-bandwidth flash into an effective on-package weight tier, delivering multi-fold throughput and energy gains in simulated LLM inference.
FLINT: Efficiently Leveraging High Bandwidth Flash for Capacity-Scalable LLM Inference Acceleration
FLINT is a workload-driven HBF substrate that co-designs a hardware burst-buffer controller, phantom-plane refresh, and a read-only FTL to enable multi-terabyte near-accelerator flash to serve LLM weights with large throughput and energy advantages in capacity-constrained inference settings.
Citation observations
Cumulative provider counts captured on specific dates; providers are never combined.
LLM inference is increasingly constrained by accelerator memory capacity rather than compute throughput. This constraint is especially acute in single-accelerator and small-node inference systems, where limited on-package memory capacity restricts the size of deployable models. HBF is an emerging 3D-stacked NAND flash technology that provides multi-terabyte near-accelerator capacity, making it a promising capacity tier for storing LLM weights. However, existing HBF-based proposals face three adoption challenges: they (1) rely on coarse-grained static prefetching for LLM weights aiming to hide the microsecond-level read latency of the NAND flash device while maximizing HBF's read throughput, (2) expose NAND flash management tasks (e.g., refresh operations) to the accelerator-visible critical inference path, and (3) miss optimization opportunities to specialize and optimize the flash-management mechanisms to the workload behavior. Our goal is to design an efficient HBF substrate that integrates HBF as a memory-capacity tier alongside HBM while addressing these three challenges. To this end, we propose FLINT, a workload-driven HBF substrate for capacity-scalable LLM inference. FLINT introduces three mechanisms: (1) a hardware burst-buffer controller that dynamically coalesces and pipelines HBF reads aiming to utilize existing NAND flash buffers while sustaining high HBF bandwidth, (2) a phantom-plane refresh mechanism, which removes refresh from the critical inference path by moving refresh-related NAND flash operations outside the read foreground back via low-cost resource duplication, and (3) a read-only FTL, which replaces SSD-class support for arbitrary writes with a compact table that translates logical weight bursts to physical HBF locations.
Summary
Main Finding
FLINT (flash inference tier) is a workload-driven HBF (high-bandwidth flash) substrate that makes on-package NAND flash a practical, high-throughput capacity tier for LLM inference. By (i) dynamically coalescing fine-grained read requests into multi-plane NAND bursts, (ii) removing refresh from the inference-critical path via a phantom-plane refresh design, and (iii) adopting a compact read-only flash-translation layer, FLINT sustains far higher effective HBF throughput than prior designs and enables large LLMs to run with far fewer accelerator packages. In the paper’s evaluation across six production LLMs, FLINT achieves up to 1,205× higher decode throughput (vs. an SSD-spill baseline), 6.2× higher throughput vs. a prior HBM+HBF design, and large energy reductions (e.g., 408× vs. SSD-based spill).
Key Points
-
Problem targeted
- LLM inference is increasingly capacity-bound: model weights exceed on-package HBM capacity, forcing costly multi-accelerator sharding or slow SSD spill.
- HBF (3D-stacked NAND in an HBM form factor) offers multi-terabyte, TB/s-class on-package capacity but has µs-level read latency and requires flash management (refresh, ECC, FTL) that prior proposals did not optimize for inference workloads.
-
FLINT’s three co-designed mechanisms
- Hardware burst-buffer controller - Co-located with the HBF base die. - Dynamically coalesces and pipelines fine-grained accelerator read misses into large coordinated multi-plane NAND bursts. - Drives burst timing from observed progress of the active burst (lookahead window) rather than static compiler prefetch hints. - Uses per-plane page and cache buffers inside HBF dies instead of a separate SRAM staging buffer. - Outcome: keeps HBF channel occupied and aligns burst issuance to runtime demand patterns (including MoE routing, token-dependent behavior, cache-interleaving).
- Phantom-plane refresh - Provision N+1 physical planes for an N-plane logical address space. - One plane is offline at a time (the phantom); refreshes/fork-and-program operations happen to the phantom plane using its independent program circuitry, avoiding contention with foreground reads. - Outcome: refresh (read-disturb/retention mitigation) does not block or throttle decode reads on live planes.
-
Read-only FTL (flash translation layer) - Replace SSD-style page-level, write-oriented FTL with burst-granularity translation tailored to read-only, preloaded weight storage. - Example: 512 GB HBF with 2 MB bursts needs ~256K burst entries (≈512× fewer than page-level FTL); total metadata ≈1.8 MB per stack. - Outcome: simpler, latency-insensitive translation that fits in the HBF base die and eliminates unnecessary write/GC/wear-leveling machinery for inference workloads.
-
Quantitative results (reported)
- Effective HBF throughput: 6.2× higher than a prior HBM+HBF design (H3).
- Decode throughput improvements: 1,205× vs. SSD-equipped GPU system; 2.2× vs. HBM-only multi-GPU system; 6.2× vs. prior HBM+HBF.
- Energy consumption reductions: 408× vs. SSD-spill system; 1.1× vs. HBM-only; 6.8× vs. prior HBM+HBF.
- Service-level objective: FLINT meets a 50 ms time-per-output-token (TPOT) SLO with 3.1× fewer GPU packages than HBM-only (up to 8× reductions reported in some cases).
- Area overhead: small — +3.1% to an HBF die, +3.9 mm² on the HBF base die (7 nm).
-
Additional system-level insights
- FLINT is transparent to existing inference kernels and software stacks — it removes the need for compiler-emitted static prefetch directives.
- Exploits the read-only, burst-like, and predictable locality patterns of LLM weight accesses (including MoE hot-spotting and layer sweeps).
- Uses existing plane-local buffers in NAND (page + cache buffers) instead of extra on-die SRAM staging.
Data & Methods
- Evaluation targets
- Six production LLMs: five mixture-of-experts (MoE) models (DeepSeek-V3, DeepSeek-V4-Pro, Qwen3-235B-A22B, Llama-4 Maverick, Kimi K2) and one dense model (Llama-3.1-405B).
- Workloads include a range of batch sizes and long context operation (examples in paper include context size 128K tokens and batch sizes {1,4,16,64} for some analyses).
- Baselines compared
- HBM+SSD: single-GPU system that spills weights to off-package SSDs when HBM is insufficient.
- HBM-only: multi-GPU system sharded across enough packages to keep all weights in aggregate HBM.
- H3 (prior work): an HBM+HBF hybrid design with an HBM-side SRAM staging buffer and compiler-emitted layer-ahead prefetching.
- Metrics
- Decode throughput (tokens/sec), energy consumption per decode, TPOT (time per output token) SLO compliance, required number of GPU packages to meet SLO, and area overhead on HBF die/base die.
- Flash / HBF modeling details summarized (paper uses realistic HBF/NAND parameters)
- NAND primitive timings: page read ~1–2 µs (SLC), page program ~50 µs, block erase ~3 ms; refresh cost for a block ~55 ms (program + erase work).
- HBF stack organization examples: 16 dies × 32 planes/die × 256–512 blocks/plane × 1024 pages/block × 4KB pages → 512–1024 GB capacities; burst size examples: 2 MB bursts.
- FLINT’s FTL metadata cost example: 512 GB, 2 MB bursts → 256K burst entries; total FTL metadata ≈1.8 MB per stack.
- Evaluation method
- Comparative system-level evaluation against baselines using the described LLMs and HBF microarchitectural parameters. (The paper simulates HBF behavior and integrates it into a system-level inference performance/energy model — see original for implementation/simulation tool specifics and exact modeling assumptions.)
Implications for AI Economics
- Lower capital expenditures (capex) for deploying large LLMs
- FLINT reduces the number of required accelerator packages to meet latency SLOs (e.g., 3.1× fewer GPU packages for a 50 ms TPOT), directly lowering hardware purchase costs for serving large models.
- Enables single-accelerator or small-node deployment of larger models that otherwise required multi-node sharding, expanding deployment options (edge servers, inference appliances, workstations).
- Lower operating expenditures (opex)
- Significant energy reductions (especially vs. SSD-spill deployments) reduce per-token energy cost and overall data-center power/thermal footprint. Even modest improvements vs. HBM-only (reported 1.1× energy reduction) matter at scale.
- Reduced network/interconnect load and synchronization overheads from avoiding multi-accelerator sharding reduce system-level inefficiencies and hidden costs (e.g., NICs, switches, extra software complexity).
- Market and product implications
- Accelerator vendors: a new product tier (HBM + HBF integration with FLINT-like substrate) could differentiate devices for inference-centric workloads, enabling higher-value single-package solutions that serve larger models.
- Cloud providers and inference-serving vendors: fewer GPUs per model instance lowers per-instance cost and allows repacking of capacity across more value-add offerings (e.g., tighter SLO guarantees at lower cost).
- Edge and on-prem deployment: makes larger models more feasible on constrained form factors, potentially unlocking new applications that were previously impractical due to capacity constraints.
- Trade-offs and risks
- Requires adoption of HBF-capable packages and the additional hardware in the HBF base die (small area/complexity overheads, but manufacturing ecosystem and supply chains must adapt).
- FLINT depends on the read-mostly nature of deployed weights; workloads that need frequent on-flash writes or dynamic model updates would require additional mechanisms (or write support), reducing the simplicity advantages.
- Engineering and validation costs to integrate new flash management (phantom-plane refresh, burst FTL) into product-quality controllers and firmware.
- Bottom line for AI economics
- By converting large-capacity, energy-inefficient off-package storage (SSD spill or multi-GPU scaling) into a near-accelerator, high-bandwidth, read-optimized tier, FLINT materially reduces the marginal and fixed costs of serving large LLMs. This can lower the barrier to deploy larger models, reduce per-request costs, and shift some demand from scale-out hardware to vertically integrated packages with HBF — with downstream effects on pricing, capacity planning, and competitive product differentiation.
Assessment
Paper Typedescriptive
Evidence Strengthmedium — The paper provides systematic performance and energy evaluations across six production LLMs and compares FLINT to three sensible baselines, but the results are based on architecture-level modeling and simulation assumptions (HBF device parameters, latency/throughput, and system integration) rather than measurements from a physical prototype or production deployment, leaving room for model and parameter sensitivity.
Methods Rigormedium — The authors evaluate across multiple realistic models (dense and MoE), multiple baselines, and report throughput, energy, and SLO metrics; they also provide microarchitectural design details and area estimates. However, the evaluation appears to rely on simulated/emulated HBF characteristics and assumed device-level timings and costs, with limited discussion of sensitivity to those assumptions or real-world variability, and no hardware prototype validation.
SampleArchitecture-level evaluation using six production LLMs (DeepSeek-V3, DeepSeek-V4-Pro, Qwen3-235B-A22B, Llama-4 Maverick, Kimi K2 — five MoE models — and one dense model, Llama-3.1-405B). Workloads include decode/inference with varying batch sizes and large context windows; compared FLINT against three baselines (HBM+SSD single-GPU spill, HBM-only multi-GPU sharded, and prior HBM+HBF design H3). Hardware and HBF stack parameters are modeled (Gen-1 HBF stacks, burst sizes, NAND timings, plane counts); energy and area estimated from modeled components.
Themesadoption innovation
GeneralizabilityResults depend on assumed HBF device parameters (latency, bandwidth, refresh/program timings) and Gen-1 design points that may change in real products., Workload suite covers several large models but may not represent all model architectures, mixed precisions, or future sparsity/quantization techniques., Evaluation is simulation/model-based without a silicon prototype, so integration, firmware/driver overheads, and production system behaviors could alter gains., Assumes read-only weight workflows (weights preloaded and immutable); workloads or systems that update weights (online fine-tuning) may not benefit., Comparisons use particular GPU/system configurations (e.g., B200-like packages); different accelerators, interconnect topologies, or memory hierarchies could change relative performance.
Claims (12)
| Claim | Direction | Outcome | Confidence & Evidence | Details |
|---|---|---|---|---|
| FLINT improves decode throughput by 1,205× compared with an SSD-equipped GPU system. Task Completion Time | positive | LLM decode throughput |
Reading fidelity
high
Study strength
medium
|
n=6
1,205× higher decode throughput
|
| FLINT improves decode throughput by 2.2× compared with an HBM-only GPU system. Task Completion Time | positive | LLM decode throughput |
Reading fidelity
high
Study strength
medium
|
n=6
2.2× higher decode throughput
|
| FLINT improves decode throughput by 6.2× compared with a prior hybrid HBM+HBF GPU system. Task Completion Time | positive | LLM decode throughput |
Reading fidelity
high
Study strength
medium
|
n=6
6.2× higher decode throughput
|
| FLINT reduces energy consumption by 408× compared with an SSD-equipped GPU system. Organizational Efficiency | positive | Inference energy consumption |
Reading fidelity
high
Study strength
medium
|
n=6
408× lower energy consumption
|
| FLINT reduces energy consumption by 1.1× compared with an HBM-only GPU system. Organizational Efficiency | positive | Inference energy consumption |
Reading fidelity
high
Study strength
medium
|
n=6
1.1× lower energy consumption
|
| FLINT reduces energy consumption by 6.8× compared with a prior hybrid HBM+HBF GPU system. Organizational Efficiency | positive | Inference energy consumption |
Reading fidelity
high
Study strength
medium
|
n=6
6.8× lower energy consumption
|
| FLINT meets a 50 ms time-per-output-token service-level objective with 3.1× fewer GPU packages than the HBM-only GPU system. Organizational Efficiency | positive | GPU packages required to meet a 50 ms TPOT SLO |
Reading fidelity
high
Study strength
medium
|
n=6
3.1× fewer GPU packages
|
| FLINT sustains 6.2× higher effective HBF throughput than prior HBF designs. Firm Productivity | positive | Effective high-bandwidth flash throughput |
Reading fidelity
high
Study strength
medium
|
n=6
6.2× higher effective HBF throughput
|
| The read-only FTL for a 512 GB Gen-1 HBF stack with 2 MB bursts requires 512× fewer entries than a conventional page-level SSD FTL. Organizational Efficiency | positive | FTL metadata size |
Reading fidelity
high
Study strength
high
|
512× fewer entries
|
| The FLINT FTL requires 1.8 MB per HBF stack for its burst translation table, block-granular relocation table, and read counters. Organizational Efficiency | positive | FTL metadata storage overhead |
Reading fidelity
high
Study strength
high
|
1.8 MB per stack
|
| FLINT adds 3.1% area cost to an HBF die. Organizational Efficiency | negative | HBF die area overhead |
Reading fidelity
high
Study strength
medium
|
3.1% area cost
|
| FLINT adds 3.9 mm2 of area to the HBF base die at 7 nm. Organizational Efficiency | negative | HBF base-die area overhead |
Reading fidelity
high
Study strength
medium
|
3.9 mm2 at 7 nm
|