# WEKA WEKA is a software-defined distributed file platform that runs across commodity flash servers and presents a single namespace at aggregate bandwidth. Its place in this collection is as a representative of the parallel storage layer: the software that stands between an exabyte corpus and an accelerator fleet that must never wait. > [!note] The one-sentence version > Distribute both the data and the directory information across many flash servers, so a thousand readers are served at once rather than in turn. ## The engineering problem it addresses A training corpus frequently contains millions of small files read in a shuffled order by hundreds of processes at once. The dominant cost in that pattern is not reading bytes but looking things up, and a storage system whose metadata service runs in one place will saturate on lookups while its flash sits idle. Distributing metadata alongside the data is therefore the characteristic requirement of storage built for this work, and it is the property on which platforms in this category are evaluated; see [[wiki/Parallel File System|Parallel File System]]. The second requirement is a transport that does not consume host processors. These platforms move data by remote direct memory access over the cluster fabric, so a client receives data into its memory without its processor copying anything, and in the strongest configurations directly into accelerator memory; see [[wiki/RDMA|RDMA]] and [[wiki/InfiniBand|InfiniBand]]. The third is tiering. Fast flash holds the working set while capacity and durability live in object storage behind it, with movement between them managed rather than manual; see [[wiki/Storage Tiering|Storage Tiering]] and [[wiki/S3-Compatible Object Storage|S3-Compatible Object Storage]]. ## Why this layer earns its cost An accelerator waiting on data is the most expensive idle object in a facility, and the loss is invisible: no error is logged, the hardware reports itself healthy, and the facility simply produces less than it was built to produce; see [[wiki/Utilization|Utilization]]. Storage capable of saturating a fleet's input pipeline, and of absorbing the simultaneous checkpoint write of every rank in a job, is therefore bought as utilization insurance rather than as capacity; see [[wiki/Checkpointing|Checkpointing]]. ## The field This is a competitive category rather than a single-supplier one. It includes long-established open and commercial parallel file systems from the high-performance computing tradition, software-defined platforms of this kind, appliance-based offerings from the large storage vendors, the hyperscale operators' own internal systems, and object storage with caching layers in front of it for workloads that tolerate it. Selection turns on metadata performance under small-file load, checkpoint write behavior, client efficiency, and operational maturity rather than on raw capacity. ## Relationships - **instance of:** [[wiki/Parallel File System|Parallel File System]] - **transported by:** [[wiki/RDMA|RDMA]] · [[wiki/InfiniBand|InfiniBand]] · [[wiki/NVMe over Fabrics|NVMe over Fabrics]] - **built on:** [[wiki/NVMe SSD|NVMe SSD]] - **tiered with:** [[wiki/S3-Compatible Object Storage|S3-Compatible Object Storage]] · [[wiki/Storage Tiering|Storage Tiering]] - **protects:** [[wiki/Utilization|Utilization]] · [[wiki/Checkpointing|Checkpointing]] - **part of:** [[wiki/Data Fabric and Storage|Data Fabric and Storage]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** software-defined distributed file platforms presenting a single namespace across commodity flash at aggregate bandwidth, with parallel distribution of both data and metadata; metadata lookup rather than byte throughput as the dominant cost for datasets of millions of small files; remote direct memory access transport avoiding host processor involvement; tiering between flash working sets and object storage capacity. - **Established:** the platform explicitly positioned to remove data bottlenecks for training datasets of millions of files through parallel data and metadata distribution across flash. - **Established:** the category as competitive, spanning high-performance computing parallel file systems, software-defined platforms, vendor appliances, hyperscale internal systems, and cached object storage. ## Sources - Public distributed storage platform documentation and high-performance storage practice, as current October 2026.