# Data Fabric and Storage An accelerator fleet is only as fast as the data reaching it. The storage layer exists to keep the most expensive processors in the building supplied, continuously, from corpora far too large to hold anywhere near them. It is best understood as one system with several tiers rather than as a set of separate products. > [!note] The one-sentence version > Keep petabytes where they are cheap, keep the working set where it is fast, and move between the two before anybody waits. ## The components - **Local flash** in each server, for the operating environment, a working subset, cached weights, and checkpoint staging; see [[wiki/NVMe SSD|NVMe SSD]]. - **Parallel file systems** of distributed flash, serving every node at aggregate bandwidth with distributed metadata, which is what makes millions of small files tractable; see [[wiki/Parallel File System|Parallel File System]]. - **Object storage** for the durable corpus and archive, at exabyte scale, through an interface that has become a de facto standard; see [[wiki/S3-Compatible Object Storage|S3-Compatible Object Storage]]. - **Disaggregated flash** presented over the network so that capacity can be pooled and attached where needed rather than stranded in a chassis; see [[wiki/NVMe over Fabrics|NVMe over Fabrics]]. - **The transport** that moves it all without consuming host processors; see [[wiki/RDMA|RDMA]] · [[wiki/InfiniBand|InfiniBand]]. - **The tiering policy** that decides what lives where and stages it in advance; see [[wiki/Storage Tiering|Storage Tiering]]. - **Checkpoint handling,** the most demanding write pattern in the facility, as every rank writes simultaneously; see [[wiki/Checkpointing|Checkpointing]]. ## The two design facts worth carrying **Metadata, rather than bandwidth, is usually the hard problem.** Training corpora of millions of small files generate far more lookups than large reads, and a centralized metadata service saturates while the flash behind it sits idle. Distributing metadata is the characteristic engineering achievement of storage built for this work. **Data movement is where utilization quietly leaks.** Accelerators idle between jobs while datasets stage, and idle within jobs when the input pipeline cannot keep pace. Neither failure appears as an error; both appear as a facility that cost what it cost and produced less than it should; see [[wiki/Utilization|Utilization]] and [[wiki/Host CPU|Host CPU]]. ## Relationships - **composed of:** [[wiki/NVMe SSD|NVMe SSD]] · [[wiki/Parallel File System|Parallel File System]] · [[wiki/S3-Compatible Object Storage|S3-Compatible Object Storage]] · [[wiki/NVMe over Fabrics|NVMe over Fabrics]] · [[wiki/Storage Tiering|Storage Tiering]] - **transported by:** [[wiki/RDMA|RDMA]] · [[wiki/InfiniBand|InfiniBand]] · [[wiki/DPU and SmartNIC|DPU and SmartNIC]] - **serves:** [[wiki/Compute and Accelerator Systems|Compute and Accelerator Systems]] · [[wiki/Checkpointing|Checkpointing]] · [[wiki/Retrieval-Augmented Generation|Retrieval-Augmented Generation]] - **determines:** [[wiki/Utilization|Utilization]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** the tier structure of local flash, parallel flash file systems, object storage, and disaggregated flash, moved by direct memory access transports over the cluster fabric; distributed metadata as the decisive capability for datasets of millions of small files; checkpoint writes as the most demanding write pattern; data movement at job boundaries and within input pipelines as a significant source of accelerator idleness. - **Established:** parallel flash platforms used specifically to remove training data bottlenecks through parallel distribution of data and metadata, combined with low-latency fabric to sustain accelerator utilization. ## Sources - High-performance storage architecture practice for AI training facilities, as current October 2026.