FLINT: Efficiently Exploiting High-Bandwidth Flash for Capacity-Scalable LLM Inference Acceleration

Large language model (LLM) inference is increasingly constrained by accelerator memory capacity rather thancompute throughput. High-bandwidth flash (HBF) is an emerging3D-stacked NAND flash technology that provides multi-terabytenear-accelerator capacity, making it a promising capacity tier forstoring LLM weights. However, existing HBF-based proposals facethree adoption challenges: they (i) rely on coarse-grained staticprefetching, (ii) expose NAND flash refresh to the acceleratorvisible critical path, and (iii) assume solid-state drive (SSD)-classflash-management mechanisms that are poorly matched to aread-only HBF weight tier. Our goal is to design an efficientHBF substrate that integrates HBF as a memory capacity tieralongside high bandwidth memory (HBM) while addressing thesethree challenges. To this end, we propose FLINT (flash inferencetier), a workload-driven HBF substrate for capacity-scalable LLMinference. FLINT introduces three mechanisms: (i) a hardwareburst-buffer controller, which pipelines HBF reads with acceleratorside memory requests using existing NAND flash page/cachebuffers, (ii) a phantom-plane refresh mechanism, which removesrefresh from the LLM decode critical path, and (iii) a read-onlyflash translation layer (FTL), which replaces SSD-class supportfor arbitrary writes with a compact table that translates logicalweight bursts to physical HBF locations. We evaluate FLINT acrossdense and mixture-of-experts (MoE) LLM inference models. Ourevaluation shows that FLINT improves LLM decode throughputby 355×, 3.0×, and 1.17× and reduces energy consumption by294×, 2.5×, and 1.14× compared to an SSD-equipped graphicsprocessing unit (GPU) system, an HBM-only GPU system, and aprior hybrid HBM+HBF GPU system, respectively. FLINT addssmall area cost to an HBF die (3.1%).