# Slurm Slurm is a workload manager from the high-performance computing tradition. A user submits a job describing what it needs, how many nodes, how many accelerators, how long, and the system queues it, allocates exactly those resources when they are available, runs it, accounts for what it consumed, and releases them. It has managed many of the world's largest machines for two decades and remains where a great deal of large-scale model training actually runs. > [!note] The one-sentence version > Ask for a hundred nodes for six hours; wait your turn; get exactly that, exclusively, and be charged for it. ## Why this model suits training A training run is a single synchronous computation across many machines for a long time. It wants all its resources at once, exclusively, with no neighbors competing for memory bandwidth or network, and it wants them placed close together. Those requirements match the batch model precisely, and they sit awkwardly with a service-oriented control plane whose assumptions are the opposite; see [[wiki/Kubernetes|Kubernetes]]. Three capabilities matter particularly. **Allocation is atomic**: a job receives its full set of nodes or waits, which prevents the waste of half a job holding accelerators while idle. **Placement is topology-aware**: the allocator understands the fabric and prefers contiguous blocks, which determines achieved bandwidth for collectives; see [[wiki/Network Fabric Topology|Network Fabric Topology]]. And **accounting is rigorous**: every job's consumption is recorded against an account, which is what makes a shared facility governable. ## Prerequisites - **Partition:** a named set of nodes with its own limits and policies, used to separate hardware classes, priorities, or user groups. - **Job and job step:** the resource reservation, and the individual program launches that run inside it. - **Fair-share scheduling:** priority computed from an account's recent usage against its entitlement, so that a heavy user yields to a light one without manual intervention. - **Backfill:** running a short job in a gap ahead of a large queued job, provided it finishes before the large job's resources are assembled; this is what keeps utilization high in a queue of mixed sizes. - **Preemption:** suspending or cancelling lower-priority work for higher-priority work, which combined with [[wiki/Checkpointing|checkpointing]] makes low-priority capacity genuinely usable. - **Reservation:** holding resources for a known future need, such as a scheduled large run or maintenance. - **Prolog and epilog:** scripts run before and after a job to prepare and clean a node, which is where health checks and accelerator resets belong. ## Where it meets the rest of the stack The scheduler is the natural place to make a facility's physical constraints visible to its work. A power limit becomes a scheduling input, so that flexible jobs are paused or slowed during a grid event rather than the facility reducing capacity indiscriminately; see [[wiki/Demand Response|Demand Response]]. Node health checks in the prolog keep a degraded machine from joining a collective and slowing every other participant, which is among the highest-value operational practices available; see [[wiki/Collective Communication|Collective Communication]] and [[wiki/Site Reliability Engineering|Site Reliability Engineering]]. ## Relationships - **schedules:** [[wiki/PyTorch|PyTorch]] training runs · [[wiki/OCI Container|OCI Container]] - **allocates:** [[wiki/Accelerator|Accelerator]] · [[wiki/Data Center Rack|Data Center Rack]] - **aware of:** [[wiki/Network Fabric Topology|Network Fabric Topology]] - **depends on:** [[wiki/Checkpointing|Checkpointing]] for preemption - **implements:** [[wiki/Job Scheduling|Job Scheduling]] · [[wiki/Demand Response|Demand Response]] participation - **complements:** [[wiki/Kubernetes|Kubernetes]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** atomic allocation of a job's full resource set; partitions, jobs and job steps, fair-share priority, backfill, preemption, reservations, and prolog and epilog scripts; topology-aware contiguous placement; rigorous per-account usage accounting; two decades of use managing the largest high-performance systems. - **Established:** the suitability of the batch model to long synchronous training runs against service-oriented orchestration. - **Strongly indicated:** node health checks in job prologs materially reducing straggler-induced slowdown; operator practice is consistent and published quantification is limited. ## Sources - Slurm workload manager documentation and high-performance computing scheduling practice, as current October 2026.