# Memory Wall An accelerator's arithmetic units can multiply numbers far faster than memory can deliver numbers to multiply. That gap is the memory wall, and it governs the design of every component above the silicon in this collection. Once a reader sees it, a great deal of otherwise mysterious engineering becomes obvious: why memory is stacked and bonded beside the processor, why interconnect bandwidth is quoted as prominently as arithmetic throughput, why storage systems are built to keep expensive processors from waiting, and why a cluster is judged by how little of its time is spent idle. > [!note] The one-sentence version > Arithmetic has become cheap and moving data has stayed expensive; the whole stack above silicon is an argument about how to keep the arithmetic fed. ## Why it exists Compute throughput has grown faster than memory bandwidth for decades, and far faster than memory latency has fallen. Transistors shrank and multiplied, so arithmetic units multiplied with them; the wires, pins, and circuit boards that carry data obey different physics and improved more slowly. Each generation widened the gap. Model workloads then made the gap acute, because training and inference both consist of streaming very large arrays of numbers through comparatively simple arithmetic, over and over. ## Prerequisites - **Bandwidth:** how many bytes per second a memory system can deliver. - **Latency:** how long one request waits before its data arrives. - **Arithmetic intensity:** the number of arithmetic operations a workload performs for each byte it reads. High intensity work keeps units busy; low intensity work waits on memory. - **Roofline:** the standard mental model in which achievable performance is bounded either by arithmetic throughput or by memory bandwidth, whichever binds first for a given arithmetic intensity. - **Memory hierarchy:** registers, on-die caches, package memory, host memory, local storage, networked storage; each step larger, slower, and cheaper per byte. ## How the wall shapes the stack Read upward and the same motive appears at every layer. - **At the package:** [[wiki/High-Bandwidth Memory|high-bandwidth memory]] exists because board-mounted memory cannot supply an accelerator through a connector; the memory is stacked and placed on the same substrate to shorten every wire and multiply the width of the interface. - **At the rack:** [[wiki/NVLink|accelerator-to-accelerator fabrics]] exist so that a model larger than one device's memory can span many devices while the arithmetic still sees its operands quickly; see [[wiki/Scale-Up and Scale-Out|Scale-Up and Scale-Out]]. - **In serving:** the [[wiki/KV Cache|key-value cache]] turns repeated attention computation into stored state, trading memory capacity for arithmetic; [[wiki/Prefill and Decode|prefill and decode]] are separated precisely because one phase is arithmetic-bound and the other is memory-bound. - **In numerics:** [[wiki/Quantization|quantization]] reduces the bytes each parameter occupies, so the same bandwidth carries more of the model per second; lower precision is a bandwidth strategy before it is an arithmetic one. - **In storage:** [[wiki/Parallel File System|parallel file systems]] and local flash exist to keep training data arriving faster than the accelerators consume it, because an idle accelerator is the most expensive object in the building. ## Stack position - **Upstream:** memory manufacturing and advanced packaging capacity set how much bandwidth a generation can offer at all. - **Downstream:** every architectural choice in compute, interconnect, storage, and serving software. - **Function and vendor:** the constraint is physical and belongs to no vendor; competing architectures are competing answers to it. - **Degraded case:** arithmetic units sit idle, utilization falls, and the capital in the building earns less per hour; the cost of the wall is paid in unused capacity. ## Lineage and trajectory The term entered computer architecture in the mid-1990s as a warning about processor and memory divergence; caches, prefetching, and out-of-order execution postponed it for general-purpose computing. Accelerators met it head-on because their workloads have low arithmetic intensity by nature. The answers have arrived in order: wider interfaces, then stacked memory beside the die, then fabrics that pool memory across devices, then software that economizes on bytes. The trajectory points toward memory moving still closer to logic, including stacking memory directly above compute, and toward numeric formats that shrink the data further. The wall recedes; it has never fallen. ## Relationships - **answered by:** [[wiki/High-Bandwidth Memory|High-Bandwidth Memory]] · [[wiki/NVLink|NVLink]] · [[wiki/UALink|UALink]] · [[wiki/Quantization|Quantization]] · [[wiki/KV Cache|KV Cache]] - **shapes:** [[wiki/Accelerator|Accelerator]] · [[wiki/Prefill and Decode|Prefill and Decode]] · [[wiki/Parallel File System|Parallel File System]] · [[wiki/Utilization|Utilization]] - **constrained by:** [[wiki/Advanced Packaging|Advanced Packaging]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** the divergence of arithmetic throughput and memory bandwidth across processor generations; the roofline model; low arithmetic intensity as a property of transformer training and inference; stacked package memory and accelerator fabrics as the industry's answers. - **Strongly indicated:** memory stacked directly above logic as the next structural step; research and early products exist, and volume production remains to be shown. ## Sources - Computer-architecture literature on the processor-memory performance gap and the roofline model. - Public accelerator specifications documenting package memory capacity and bandwidth per generation, as of October 2026.