# Site Reliability Engineering Reliability engineering treats operating a system as a software and engineering problem rather than an administrative one. Its central move is to make reliability a measured quantity with an explicit budget, so that the eternal argument between shipping changes and keeping things working becomes an arithmetic question with an agreed answer. > [!note] The one-sentence version > Decide numerically how reliable the service must be, measure it, and spend the difference on change; when the budget is gone, stop changing things. ## The core apparatus - **Service level indicator:** the measured quantity, such as the fraction of requests answered successfully within a latency threshold. - **Service level objective:** the target for that indicator over a window, chosen deliberately rather than aspirationally. - **Error budget:** the permitted shortfall. A 99.9 percent objective permits roughly forty-three minutes of failure a month, and that allowance is a resource to be spent on deploying changes, performing maintenance, and running experiments. Exhausting it freezes change until reliability is recovered, which converts a cultural dispute into a rule everyone accepted in advance. - **Toil reduction:** systematically removing manual repetitive work by automating it, on the principle that an operator's time is best spent eliminating the next interruption. - **Blameless postmortem:** written analysis of an incident focused on the conditions that permitted it rather than the person who touched it last, because systems that punish disclosure stop learning. - **Runbook and on-call:** documented procedures and a rotation with explicit escalation, sized so that the people carrying it can sustain it. ## What is particular about accelerator fleets **Stragglers rather than failures.** A conventional service degrades when something fails. A training cluster degrades when something becomes slightly slow, because a collective completes at the pace of its slowest participant; see [[wiki/Collective Communication|Collective Communication]]. Operations therefore hunts for a device throttling thermally, a link with a rising error rate, a storage path with a degraded device, or a node whose firmware differs by a revision. Detection is statistical: compare every device against its peers and investigate the outliers. **Health checking before admission.** A node is verified before it may join a job, in the scheduler's prologue, because admitting a degraded machine costs the whole run rather than one node; see [[wiki/Slurm|Slurm]]. **Checkpoint hygiene.** Long runs survive through checkpoints, so checkpoint frequency, write performance, and restore testing are first-order reliability concerns; a checkpoint never restored is a hypothesis; see [[wiki/Checkpointing|Checkpointing]]. **Facility coupling.** Thermal and electrical events appear as performance events. An operations team that cannot see cooling and power telemetry alongside job metrics will misdiagnose a throttling cluster as a software regression, which is why facility and compute observability belong in one picture; see [[wiki/Observability|Observability]]. **Uniformity as a reliability strategy.** Identical hardware, firmware, driver versions, and thermal conditions across a fleet reduce the variance that produces stragglers. At scale, sameness is a performance feature. ## Relationships - **measures through:** [[wiki/Observability|Observability]] - **protects:** [[wiki/Inference Serving|Inference Serving]] · [[wiki/Utilization|Utilization]] - **depends on:** [[wiki/Checkpointing|Checkpointing]] · [[wiki/Failure Domain|Failure Domain]] · [[wiki/Mean Time to Repair|Mean Time to Repair]] - **inherits from:** [[wiki/Commissioning|Commissioning]] - **staffed by:** [[wiki/Data Center Workforce|Data Center Workforce]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** service level indicators, objectives, and error budgets as a change-governance mechanism, with a 99.9 percent objective corresponding to roughly forty-three minutes of monthly allowance; toil reduction; blameless postmortem practice; runbooks and sustainable on-call rotation. - **Established:** collective completion gated by the slowest participant, making stragglers rather than outright failures the characteristic degradation of training clusters; pre-admission node health checking; checkpoint restore testing; the coupling of thermal and electrical events to observed performance. - **Strongly indicated:** fleet uniformity in hardware, firmware, and thermal conditions materially reducing straggler incidence; operator practice is consistent and published quantification is limited. ## Sources - Site reliability engineering practice literature and large-scale cluster operations experience, as current October 2026.