# Kubernetes Kubernetes is a control plane for running containerized software on a fleet of machines. An operator declares the state they want, and the system continuously works to make reality match it: starting what is missing, replacing what has failed, and reporting what it cannot achieve. That single idea, declared intent reconciled continuously, is why it became the standard way to run services, and the reason it needed extension before it could run training jobs. > [!note] The one-sentence version > Describe what should be running; the system keeps making the cluster match the description, forever. ## How it works The **API server** holds the declared state in a consistent key-value store. **Controllers** watch that state and compare it with observed reality, acting to close the difference; each controller is a loop doing one such job. The **scheduler** decides which node each new unit of work should run on, given its resource requests and the constraints attached to it. On every node a **kubelet** starts and supervises containers and reports their condition back. A unit of work is a **pod**, one or more containers sharing a network identity and storage, and the smallest thing the system schedules. The architecture's strength is that it is extensible by the same mechanism it uses internally. A **custom resource definition** adds a new kind of object to the API, and a **controller** written for it reconciles that object however the author chooses. This operator pattern is how databases, storage systems, network fabrics, and training frameworks are managed by the same control plane that manages web services. ## What had to be added for accelerator work The system was designed for long-running services that scale by instance count, and a training job has nearly opposite properties. Four gaps were filled. **Gang scheduling.** A distributed training job needs all its workers simultaneously or none; starting half of them wastes the accelerators they hold while they wait. Default behavior schedules pods independently, so batch schedulers were added to place a whole group at once or not at all; see [[wiki/Job Scheduling|Job Scheduling]]. **Topology awareness.** Where workers run matters enormously, since placement determines whether traffic stays inside a scale-up domain; see [[wiki/Network Fabric Topology|Network Fabric Topology]]. Topology-aware placement and device-manager policies were added so that an allocation respects the machine's internal structure. **Device and fabric access.** Accelerators, fabric interfaces, and their driver dependencies are exposed through device plugins, with the host driver and in-image library compatibility that implies; see [[wiki/OCI Container|OCI Container]]. **Queueing and fair sharing.** Research and production users contend for a scarce pool, so queue systems with quotas, priorities, preemption, and borrowing between teams were added above the scheduler. ## Where it fits against the alternative Two orchestration traditions meet in these facilities. This control plane came from running internet services and excels at serving, autoscaling, rolling updates, and heterogeneous long-lived workloads, which makes it the natural home for inference; see [[wiki/Inference Serving|Inference Serving]]. The high-performance computing tradition came from running batch jobs on supercomputers and excels at queueing, allocation, and accounting for large synchronous runs; see [[wiki/Slurm|Slurm]]. Many facilities run both, and the convergence is proceeding from both directions. ## Relationships - **orchestrates:** [[wiki/OCI Container|OCI Container]] · [[wiki/Inference Serving|Inference Serving]] · [[wiki/Inference Endpoint|Inference Endpoint]] - **extended by:** [[wiki/Job Scheduling|Job Scheduling]] - **aware of:** [[wiki/Network Fabric Topology|Network Fabric Topology]] · [[wiki/Accelerator|Accelerator]] - **complements:** [[wiki/Slurm|Slurm]] - **observed through:** [[wiki/Observability|Observability]] - **part of:** [[wiki/Software and Orchestration|Software and Orchestration]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** the declarative reconciliation architecture of API server, consistent store, controllers, scheduler, and node agents; pods as the scheduling unit; custom resources and the operator pattern as the extension mechanism; device plugins exposing accelerators and fabric interfaces; gang scheduling, topology-aware placement, and queueing with quotas and preemption added for batch accelerator workloads. - **Established:** the division of orchestration traditions, with this control plane dominant in serving and batch schedulers dominant in large synchronous training. - **Strongly indicated:** continuing convergence of the two traditions; both ecosystems are adding the other's capabilities and no single outcome is settled. ## Sources - Kubernetes project documentation and batch scheduling extension practice, as current October 2026.