# NVMe SSD An NVMe solid-state drive is flash memory attached directly to the processor's expansion lanes and spoken to through a protocol designed for flash rather than inherited from mechanical disks. The result is storage whose latency is measured in tens of microseconds and whose throughput is measured in gigabytes per second, in a package the size of a stick of gum. > [!note] The one-sentence version > Flash spoken to directly over the expansion bus with a protocol built for it, instead of pretending to be a hard disk. ## Why the protocol had to change Storage protocols were built around the physics of a rotating platter: a single command queue was sufficient because the drive could only move one head at a time, and a millisecond of software overhead was invisible against several milliseconds of seek. Flash has no moving parts, serves many requests in parallel across its internal channels, and answers in tens of microseconds. Under the old protocol the software overhead and the single queue became the limit, so a device capable of hundreds of thousands of operations per second delivered a fraction of them. The replacement protocol was designed around flash's actual properties: thousands of parallel queues with deep command depth, a minimal command set, direct attachment to the processor's lanes without an intervening controller, and a driver path short enough that the device rather than the software sets the pace. It is a clean example of a general principle in this collection, which is that an abstraction built for one physical substrate becomes the bottleneck when the substrate changes. ## Prerequisites - **Flash and the flash translation layer:** non-volatile memory erased in large blocks and written in smaller pages, with controller firmware mapping logical addresses onto physical ones and managing wear. - **Queue depth and parallelism:** how many requests can be outstanding at once; flash performance depends on keeping many in flight. - **Endurance:** how much data can be written over the device's life, expressed as drive writes per day; a write-heavy role requires a different device class from a read-heavy one. - **Namespaces:** dividing one physical device into independently managed logical devices, useful for isolating tenants or workloads. - **Form factors:** the card and the newer enterprise form factors designed for serviceability and thermals in dense servers. - **Power loss protection:** capacitors allowing in-flight writes to complete on sudden power loss, which is what distinguishes an enterprise device from a consumer one. ## Its role in an accelerator server Published building blocks in current accelerator servers commonly pair a boot device of roughly a terabyte with several data devices of a few terabytes each, which gives the machine local flash for the operating environment, for the working subset of a dataset, for cached model weights, and for staging checkpoints before they are written to the shared system. That last use is the important one in practice: a checkpoint written first to local flash and then flushed to the parallel file system removes a burst from the shared storage path; see [[wiki/Checkpointing|Checkpointing]]. Local flash is also the physical substrate beneath both the parallel file systems and the disaggregated storage protocols above it, so understanding its characteristics explains the behavior of the layers built on it; see [[wiki/Parallel File System|Parallel File System]] and [[wiki/NVMe over Fabrics|NVMe over Fabrics]]. ## Relationships - **attached by:** [[wiki/PCIe|PCIe]] - **underlies:** [[wiki/Parallel File System|Parallel File System]] · [[wiki/NVMe over Fabrics|NVMe over Fabrics]] · [[wiki/Storage Tiering|Storage Tiering]] - **serves:** [[wiki/Host CPU|Host CPU]] · [[wiki/Checkpointing|Checkpointing]] - **installed in:** [[wiki/Compute Tray|Compute Tray]] - **part of:** [[wiki/Data Fabric and Storage|Data Fabric and Storage]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** the single-queue and software-overhead limits of disk-era storage protocols applied to flash; the replacement protocol's deep parallel queues, minimal command set, and direct lane attachment; flash translation layers and wear management; endurance classes expressed as drive writes per day; namespaces; enterprise form factors; power loss protection distinguishing enterprise devices. - **Established:** published accelerator server building blocks pairing a boot device of roughly a terabyte with several multi-terabyte data devices. - **Strongly indicated:** local flash staging of checkpoints materially relieving shared storage bursts; the engineering reasoning is clear and published comparisons are limited. ## Sources - NVM Express specifications and enterprise flash storage practice, as current October 2026. - Public accelerator server specifications, as of October 2026.