# Parallel File System A parallel file system presents a single namespace whose contents are spread across many servers and many flash devices, so that hundreds of clients can read different parts of it simultaneously at the aggregate speed of the whole array rather than the speed of one server. It exists because accelerators consume data faster than any single storage system can deliver, and because an accelerator waiting on data is the most expensive idle object in the building. > [!note] The one-sentence version > Spread the data and the directory information across many servers, so a thousand readers are served at once instead of in turn. ## Why it exists A training run reads an enormous dataset repeatedly, in a random order that changes every epoch, from many processes at once. A conventional file server answers requests through one machine, so its single network connection and its single metadata service become the limit regardless of how much flash sits behind it. Parallel systems remove both bottlenecks by distributing them: data is striped across many storage servers, and the metadata, the directory structure and file attributes, is itself distributed rather than served from one place. That second point is the one that distinguishes a system built for this work. Training datasets frequently contain millions of small files, and the operation that dominates is not reading bytes but looking things up. A system whose metadata service is centralized will saturate on lookups while its flash sits idle, which is why metadata distribution, rather than raw throughput, is the characteristic engineering problem. ## Prerequisites - **Namespace:** the single directory tree clients see, independent of where the data physically resides. - **Striping:** dividing a file across many devices so a single read is served by many in parallel. - **Metadata service:** the component tracking files, directories, and attributes; distributing it is the hard part. - **Client:** software on each compute node that talks to all storage servers directly rather than through a gateway, which is what allows aggregate bandwidth to reach the client. - **Erasure coding and replication:** protecting against device and server loss by storing parity across nodes, or by keeping copies; parity is more space-efficient and costs more on reconstruction. - **Remote direct memory access transport:** moving data into client memory without processor involvement, which is how these systems reach their rated rates; see [[wiki/RDMA|RDMA]]. - **Scratch against persistent tiers:** fast flash holding the working set, with capacity storage behind it; see [[wiki/Storage Tiering|Storage Tiering]]. ## What it serves in this stack Training data delivery at a rate that keeps accelerators busy, which is the primary purpose; see [[wiki/Utilization|Utilization]]. Checkpoint writes, which are bursts of enormous sequential output from every rank at once and are among the most demanding patterns any storage system faces; see [[wiki/Checkpointing|Checkpointing]]. Model weight loading at the start of a serving deployment, where hundreds of nodes read the same large files simultaneously. And shared home and project space for the people using the cluster. ## Position - **Upstream:** flash devices and the cluster fabric; these systems are commonly deployed on the same low-latency fabric as the compute traffic; see [[wiki/InfiniBand|InfiniBand]]. - **Downstream:** every training job's input pipeline and every checkpoint. - **Alternatives:** object storage for capacity and durability rather than low-latency parallel access; local flash for a working set that fits; see [[wiki/S3-Compatible Object Storage|S3-Compatible Object Storage]] and [[wiki/NVMe SSD|NVMe SSD]]. The common production answer uses all three in a tier. - **Degraded case:** the input pipeline becomes the bottleneck and accelerators idle, which converts a storage shortfall directly into wasted capital. ## Relationships - **transported by:** [[wiki/RDMA|RDMA]] · [[wiki/InfiniBand|InfiniBand]] · [[wiki/NVMe over Fabrics|NVMe over Fabrics]] - **built on:** [[wiki/NVMe SSD|NVMe SSD]] - **serves:** [[wiki/Checkpointing|Checkpointing]] · [[wiki/PyTorch|PyTorch]] input pipelines · [[wiki/Host CPU|Host CPU]] - **tiered with:** [[wiki/S3-Compatible Object Storage|S3-Compatible Object Storage]] · [[wiki/Storage Tiering|Storage Tiering]] - **part of:** [[wiki/Data Fabric and Storage|Data Fabric and Storage]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** striping across storage servers and distributed metadata as the two mechanisms removing single-server bottlenecks; metadata lookup rather than byte throughput as the dominant cost for datasets of millions of small files; direct client-to-server access enabling aggregate bandwidth; erasure coding against replication trade-offs; deployment on the low-latency cluster fabric; checkpoint writes as an extreme sequential burst pattern. - **Established:** parallel flash file systems used specifically to remove data bottlenecks for training datasets containing millions of files, through parallel distribution of both data and metadata. - **Strongly indicated:** input pipeline shortfalls as a leading cause of accelerator idleness; operator practice treats it as primary and published quantification varies by workload. ## Sources - Parallel file system architecture documentation and high-performance storage practice, as current October 2026.