# Collective Communication Distributed training is a sequence of computations separated by group operations in which every participating device contributes and every device receives a result. Those operations are collectives, and they are where a cluster's networking meets its mathematics. A training run's efficiency is largely the efficiency of its collectives, and the efficiency of a collective is set by its slowest participant. > [!note] The one-sentence version > Every device adds its gradients to everyone else's and gets the total back; how fast that happens, and how evenly, decides what the whole cluster is worth. ## The operations - **All-reduce:** every device contributes a tensor, the tensors are combined element-wise, usually by summation, and every device receives the result. This is the operation data-parallel training performs once per step to synchronize gradients, and it dominates all collective traffic. - **All-gather:** every device contributes a piece and every device receives the concatenation of all pieces. - **Reduce-scatter:** the tensors are combined and each device receives one slice of the result; combined with all-gather it composes an all-reduce, which is how efficient implementations are built. - **Broadcast:** one device's data goes to all others. - **All-to-all:** every device sends a distinct piece to every other device; the heaviest pattern, central to mixture-of-experts architectures where tokens are routed to different experts on different devices. - **Barrier:** a synchronization point every device must reach before any proceeds. ## The algorithms A naive implementation gathers everything to one device and sends the result back, which makes that device's link the bottleneck and scales terribly. Real implementations use topology-aware algorithms. **Ring.** Devices are arranged in a logical ring; each sends a chunk to its neighbor while receiving another, in a reduce-scatter phase followed by an all-gather phase. Total data each device sends is independent of how many devices participate, which is the property that makes it scale, at the cost of latency proportional to the ring's length. **Tree.** Data is reduced up a tree and broadcast back down, with latency proportional to the logarithm of the participant count; better for small messages where latency dominates. **Hierarchical.** The practical answer in real systems: perform the collective within each fast scale-up domain first, then across the slower scale-out network between domains, then distribute back down. This matches the algorithm to the physical bandwidth hierarchy and is why the collective library must know the topology; see [[wiki/Scale-Up and Scale-Out|Scale-Up and Scale-Out]]. **In-network reduction.** Arithmetic performed inside the switches, so the sum is computed as data traverses the fabric rather than at the endpoints; see [[wiki/NVSwitch|NVSwitch]]. ## Why stragglers dominate A collective completes when the last participant has contributed. One device slowed by thermal throttling, a degraded link, a slower storage path, or a noisy neighbor delays every other device in the group, every step, for the duration of the run. On a cluster of thousands of accelerators the probability that something somewhere is slightly slow approaches certainty, which is why operations teams hunt stragglers as a primary activity and why uniformity of hardware, firmware, cooling, and power is prized over peak specifications; see [[wiki/Site Reliability Engineering|Site Reliability Engineering]] and [[wiki/Observability|Observability]]. This is the clearest instance of a principle that runs through the entire collection: at scale, the slowest element sets the pace, so consistency is worth more than brilliance. A thousand accelerators cooled evenly beat a thousand cooled unevenly, and that is a facilities achievement as much as a software one. ## Relationships - **transported by:** [[wiki/RDMA|RDMA]] · [[wiki/InfiniBand|InfiniBand]] · [[wiki/Ultra Ethernet|Ultra Ethernet]] · [[wiki/NVLink|NVLink]] - **accelerated by:** [[wiki/NVSwitch|NVSwitch]] in-network reduction - **shaped by:** [[wiki/Network Fabric Topology|Network Fabric Topology]] · [[wiki/Scale-Up and Scale-Out|Scale-Up and Scale-Out]] - **used by:** [[wiki/PyTorch|PyTorch]] · [[wiki/Job Scheduling|Job Scheduling]] - **degraded by:** [[wiki/Data Center Cooling|Data Center Cooling]] unevenness · [[wiki/Failure Domain|Failure Domain]] events - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** the collective operation set and their roles in data, tensor, pipeline, and expert parallelism; ring algorithms with per-device traffic independent of participant count; tree algorithms with logarithmic latency; hierarchical composition matching the bandwidth hierarchy; in-network reduction; completion gated by the slowest participant. - **Established:** straggler hunting and hardware, firmware, and thermal uniformity as primary operational concerns at cluster scale. - **Strongly indicated:** uniformity contributing more to large-run efficiency than peak per-device specifications; operator practice is consistent and published cross-cluster quantification is limited. ## Sources - Collective communication library documentation and distributed training engineering practice, as current October 2026.