# Mean Time to Repair Two numbers describe a component's contribution to reliability: how long it typically runs before failing, and how long it takes to restore once it has. At small scale the first dominates attention. At the scale of a hundred thousand accelerators, where something is always broken, the second is what actually determines availability. > [!note] The one-sentence version > With enough components, failure is continuous and ordinary; the only remaining variable is how quickly each one is put right. ## Why repair time dominates at scale Failures are roughly proportional to component count. A fleet large enough will experience several every day regardless of how good the components are, so the operating question stops being whether something is broken and becomes how much capacity is unavailable at any moment. That quantity is the failure rate multiplied by the repair time, so halving repair time halves unavailable capacity exactly as effectively as halving the failure rate, and it is usually far cheaper to achieve. ## What shortens it **Detection.** Most repair time is often spent before anyone knows there is a problem. Automated detection against peer comparison finds a degrading device long before it fails outright; see [[wiki/Observability|Observability]]. **Localization.** Knowing which physical thing is at fault, which depends entirely on accurate labelling and records; see [[wiki/Structured Cabling|Structured Cabling]]. **Automated drain.** Removing a suspect node from the schedulable pool automatically so that no new work lands on it and running work is not harmed; see [[wiki/Job Scheduling|Job Scheduling]]. **Hot-swap design.** Components replaceable without powering down neighbors: dual-corded power, redundant pumps, blind-mate liquid couplings on trays, pluggable optics. Each is a design decision that converts a repair from an outage into a task; see [[wiki/Compute Tray|Compute Tray]] and [[wiki/Redundant Power Feeds|Redundant Power Feeds]]. **Spares on site.** Inventory sized to observed failure rates, held locally rather than ordered, since a part in a shipping container is a part contributing nothing. **Procedures and practice.** Written procedures, rehearsed, with the right tools staged; a technician improvising at three in the morning is slower and more likely to make a second problem. **Fleet uniformity.** One procedure covering many machines, which is another return on sameness; see [[wiki/Site Reliability Engineering|Site Reliability Engineering]]. ## The design consequence Because repair time is as valuable as failure rate, serviceability deserves to be a stated requirement rather than an emergent property. The best-run facilities specify it: how a tray is replaced, how a pump is isolated, how a transceiver is reached, how a battery module is removed, and how long each should take. Doing that early costs almost nothing and compounds for the facility's whole life, which is the same lesson [[wiki/Commissioning|commissioning]] and [[wiki/Data Center Shell|shell design]] teach from their own directions. ## Relationships - **shortened by:** [[wiki/Observability|Observability]] · [[wiki/Structured Cabling|Structured Cabling]] · [[wiki/Compute Tray|Compute Tray]] · [[wiki/Redundant Power Feeds|Redundant Power Feeds]] - **governs:** [[wiki/Utilization|Utilization]] · [[wiki/Failure Domain|Failure Domain]] exposure - **practiced by:** [[wiki/Site Reliability Engineering|Site Reliability Engineering]] · [[wiki/Data Center Workforce|Data Center Workforce]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** unavailable capacity as the product of failure rate and repair duration, and the equivalence of halving either; failure counts scaling with component count; detection, localization, automated drain, hot-swap design, local spares, rehearsed procedures, and fleet uniformity as the established levers on repair duration. - **Strongly indicated:** detection latency as the largest component of total repair time in large fleets; operator experience is consistent and consolidated measurement is not public. ## Sources - Large-scale fleet operations and reliability engineering practice, as current October 2026.