UC Berkeley and FuriosaAI Demonstrate High-Bandwidth Flash Memory as Solution for Large Language Model Inference Bottlenecks

Researchers from UC Berkeley and FuriosaAI have evaluated high-bandwidth flash memory as a method to expand memory capacity for serving large language models, addressing bandwidth and capacity constraints that limit performance as models grow larger. The study proposes a hierarchical storage system combining high-bandwidth memory with flash storage and introduces cache-aware scheduling strategies, achieving completion time reductions of 36–87 percent compared to conventional memory-only configurations. The analysis considers trade-offs including write endurance limitations and energy consumption across different system architectures.

The research addresses a fundamental challenge in deploying increasingly sophisticated language models: as these systems grow in size and handle longer conversation contexts, the memory systems designed to store model parameters and attention data become a critical performance constraint. The team's hierarchical approach combines different memory technologies—pairing fast, expensive high-bandwidth memory with larger but slower flash storage—allowing systems to intelligently shift data between tiers based on access patterns. This hybrid strategy, guided by sophisticated scheduling algorithms, achieves substantial speedups while managing practical concerns like flash degradation over time.