# PyTorch PyTorch is the framework in which most model research and a large share of production training is written. It matters to an infrastructure collection for a specific reason: it is the layer at which hardware becomes substitutable. A model written here can run on several vendors' accelerators, which makes the framework the most important piece of software in the competitive structure of the industry. > [!note] The one-sentence version > Write the model once in ordinary Python, let the framework handle the derivatives, the compilation and the distribution, and run it on whichever accelerator you bought. ## What it provides - **Tensors on accelerators.** Multi-dimensional arrays with arithmetic executed on the device, with the memory and transfers handled by the framework. - **Automatic differentiation.** Training requires the gradient of a loss with respect to millions or billions of parameters. The framework records the operations performed in the forward pass and walks that record backward applying the chain rule, so the programmer writes only the forward computation. This single facility is why modern model research moves as fast as it does. - **Eager execution with compilation available.** Operations run immediately as written, which makes debugging ordinary, and a compiler can then trace the program into a graph, fuse operations, and generate optimized kernels, recovering much of the performance that a graph-first design would have had from the start. - **Distribution.** Built-in strategies for spreading a model and its training across many accelerators: replicating with gradient synchronization, sharding parameters and optimizer state across devices to fit models larger than one device's memory, and splitting individual layers across devices. These call the collective operations underneath; see [[wiki/Collective Communication|Collective Communication]] and [[wiki/Scale-Up and Scale-Out|Scale-Up and Scale-Out]]. - **Checkpointing.** Saving and restoring distributed model and optimizer state, which is what makes long runs survivable; see [[wiki/Checkpointing|Checkpointing]]. - **Data loading.** Parallel workers that read, decode, and assemble batches on the host while accelerators compute, which is the mechanism whose failure starves accelerators; see [[wiki/Host CPU|Host CPU]]. ## Why framework neutrality is the decisive lever An accelerator vendor's problem is that nobody wants to rewrite their code. A framework that supports multiple backends solves that problem on the vendor's behalf: the researcher writes framework code, the framework dispatches to whichever accelerator is present, and the kernel library underneath becomes an implementation detail. That is why support in this framework is the milestone every alternative accelerator aims at, and why its governance as an open project under a neutral foundation is strategically significant rather than a licensing footnote; see [[wiki/ROCm|ROCm]] and [[wiki/CUDA|CUDA]]. The lesson generalizes across this collection. Open interfaces at the right layer, the socket, the rack, the fabric, the framework, are what convert a single-supplier market into a competitive one, and competitive supply is what lets a country build at scale without asking permission. ## Relationships - **runs on:** [[wiki/CUDA|CUDA]] · [[wiki/ROCm|ROCm]] · [[wiki/Accelerator|Accelerator]] - **calls:** [[wiki/Collective Communication|Collective Communication]] - **reads through:** [[wiki/Parallel File System|Parallel File System]] · [[wiki/Storage Tiering|Storage Tiering]] - **scheduled by:** [[wiki/Slurm|Slurm]] · [[wiki/Kubernetes|Kubernetes]] · [[wiki/Job Scheduling|Job Scheduling]] - **served by:** [[wiki/vLLM|vLLM]] · [[wiki/SGLang|SGLang]] - **part of:** [[wiki/Software and Orchestration|Software and Orchestration]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** device tensors, reverse-mode automatic differentiation over a recorded operation graph, eager execution with optional graph compilation and kernel fusion, distributed strategies including replication with gradient synchronization and parameter and optimizer-state sharding, distributed checkpointing, and parallel host-side data loading. - **Established (datestamped):** as of October 2026, the framework is governed as an open project under a neutral foundation and supports multiple accelerator backends including the leading open alternative stack. - **Strongly indicated:** framework-level backend support as the decisive mechanism for alternative accelerator adoption; vendor strategy converges on it and measured share effects are still accruing. ## Sources - [Helios Is AMD's First AI System To Rival Nvidia Vera Rubin](https://www.cnbc.com/) — CNBC, October 2026; the open accelerator stack's support for this framework alongside leading inference engines. - Public framework documentation for automatic differentiation, compilation, and distributed training, as of October 2026.