# Scale-Up and Scale-Out An AI cluster has two networks, and conflating them is the most common misunderstanding of the whole architecture. Inside a rack, accelerators are joined by a memory fabric that lets one read another's memory directly. Between racks, a packet network moves messages. The first is measured in terabytes per second and nanoseconds; the second in hundreds of gigabits and microseconds. They do different jobs, under different physics, and the boundary between them is a central design decision. > [!note] The one-sentence version > Scale-up makes many accelerators behave as one machine; scale-out makes many machines cooperate on one job. ## Why there are two Training a large model divides the work in several ways at once. **Tensor parallelism** splits a single layer's matrices across devices, so every device needs the others' intermediate results continuously; the traffic is enormous, constant, and latency-sensitive, and it behaves like memory access. **Pipeline parallelism** gives different layers to different devices, passing activations along; the traffic is substantial and more regular. **Data parallelism** gives each device a different slice of the training batch and synchronizes the gradients once per step; the traffic is large but periodic and tolerant of a few microseconds. The first pattern wants a memory fabric: direct reads and writes, no host in the path, bandwidth measured against the accelerator's own memory. The third tolerates a good packet network. So the industry built both, and places the most demanding parallelism inside the scale-up domain and the more forgiving kind across the scale-out network. That mapping is the essential craft of distributing a large training job. ## The scale-up domain A scale-up domain is the set of accelerators that can address each other's memory at full fabric speed. Its size is the single most consequential number in a system's architecture, because the largest tensor-parallel group that can run efficiently is bounded by it. The domain has grown from two devices, to the eight of a conventional baseboard, to the seventy-two of current rack-scale systems; each step moved the boundary outward by roughly an order of magnitude. Implementations include a proprietary fabric with its own switch silicon, a vendor's coherent interconnect extended across a rack, and open consortium efforts aiming at the same semantics across vendors; see [[wiki/NVLink|NVLink]], [[wiki/NVSwitch|NVSwitch]], [[wiki/Infinity Fabric|Infinity Fabric]], and [[wiki/UALink|UALink]]. Inside a rack these links are short, which is why copper remains competitive and why optics have been unnecessary at that reach; see [[wiki/Data Center Optics|Data Center Optics]]. ## The scale-out network Beyond the domain, racks are connected by a lossless, low-latency packet network carrying remote memory operations across it; see [[wiki/RDMA|RDMA]]. Two families compete: a purpose-built low-latency fabric with credit-based flow control, and Ethernet enhanced for the same job through consortium work; see [[wiki/InfiniBand|InfiniBand]] and [[wiki/Ultra Ethernet|Ultra Ethernet]]. Topology, congestion control, and collective algorithms determine how close to the theoretical bandwidth a real job achieves; see [[wiki/Network Fabric Topology|Network Fabric Topology]] and [[wiki/Collective Communication|Collective Communication]]. ## Where the boundary falls, and why it moves Every generation pushes the scale-up domain larger, because the model layers that must be split keep growing and because a larger domain simplifies everything above it. Enlarging it is expensive: the fabric must stay at memory-class bandwidth across a greater distance, which costs power, silicon, and eventually optics. The trajectory is therefore a steady negotiation, and the current frontier is whether a scale-up domain can usefully exceed one rack. That question, more than any single chip specification, is what the next hardware generation is actually arguing about. ## Relationships - **implemented by:** [[wiki/NVLink|NVLink]] · [[wiki/NVSwitch|NVSwitch]] · [[wiki/Infinity Fabric|Infinity Fabric]] · [[wiki/UALink|UALink]] · [[wiki/InfiniBand|InfiniBand]] · [[wiki/Ultra Ethernet|Ultra Ethernet]] - **bounds:** [[wiki/Rack-Scale Computing|Rack-Scale Computing]] · [[wiki/Collective Communication|Collective Communication]] - **shaped by:** [[wiki/Memory Wall|Memory Wall]] · [[wiki/Network Fabric Topology|Network Fabric Topology]] - **transported over:** [[wiki/RDMA|RDMA]] · [[wiki/Data Center Optics|Data Center Optics]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** tensor, pipeline, and data parallelism and their differing communication profiles; the scale-up domain as the bound on efficient tensor-parallel width; the growth of that domain from pairs to eight-device baseboards to seventy-two-accelerator racks; copper's competitiveness at in-rack reach; lossless low-latency requirements on the scale-out fabric. - **Established:** both a purpose-built low-latency fabric family and consortium-enhanced Ethernet in production as scale-out networks. - **Unresolved:** whether scale-up domains usefully exceed a single rack in the next generation; the deciding evidence is shipping systems and their measured training efficiency. ## Sources - [Helios Is AMD's First AI System To Rival Nvidia Vera Rubin](https://www.cnbc.com/) — CNBC, October 2026; seventy-two-accelerator racks, scale-up over a consortium fabric and scale-out over standards-aligned Ethernet, and the copper-to-optics transition timeline. - Public accelerator fabric and cluster networking documentation, as of October 2026.