# Congestion Control A cluster network's hard problem is not bandwidth; it is what happens when many senders want the same destination at the same moment. Collective operations create exactly that pattern deliberately and repeatedly. Congestion control is the machinery that keeps the fabric working through it, and it is the single most common place where a cluster network that looked excellent in testing disappoints in production. > [!note] The one-sentence version > Thousands of accelerators finishing a step at once all transmit at once, and whether the fabric survives that is a question of congestion control rather than of link speed. ## The patterns that cause trouble **Incast.** Many senders transmit to one receiver simultaneously, which is the shape of a gather or the final phase of a reduction. The receiver's port becomes a funnel, buffers fill, and without intervention either packets are lost or backpressure propagates outward. **Synchronized bursts.** Collectives are barriers, so all participants proceed at once. The fabric sees an idle period followed by an intense, perfectly correlated burst, which is the worst traffic pattern for any statistically provisioned network and is why training fabrics are provisioned deterministically instead; see [[wiki/Network Fabric Topology|Network Fabric Topology]]. **Head-of-line blocking.** A packet unable to proceed holds up packets behind it that are bound elsewhere, so congestion at one destination degrades traffic that had nothing to do with it. This is the mechanism by which a local hot spot becomes a fabric-wide slowdown. **Victim flows.** The pathological case of the above, where a flow between two uninvolved endpoints is slowed because its path shares a congested link. ## The mechanisms - **Credit-based flow control,** native to purpose-built fabrics, in which a sender may not transmit without confirmed buffer space, so overrun cannot occur; see [[wiki/InfiniBand|InfiniBand]]. - **Priority-based flow control** on Ethernet, which pauses a traffic class at the link level. It prevents loss and can propagate backpressure through the fabric, which is the source of the characteristic failures of poorly tuned lossless Ethernet. - **Explicit congestion notification,** in which switches mark packets as congestion builds and endpoints reduce their sending rate in response, acting before buffers overflow rather than after. - **Congestion control algorithms at the endpoint,** which adjust rate from marks and delay measurements; the quality of this algorithm is much of what distinguishes fabric implementations. - **Adaptive and packet-level routing,** spreading traffic across equal paths by observed load rather than by a hash of flow identifiers, which prevents several large flows from colliding on one path. - **Deterministic provisioning,** meaning a non-blocking fabric, which does not remove congestion at endpoints but removes it from the fabric's interior. ## Why consortium work exists Carrying remote memory operations over Ethernet requires the lossless and low-latency behavior that direct access assumes, and achieving it historically required careful hand-tuning of flow control, buffer, and marking parameters per deployment. That tuning is expert work, it is fragile, and it does not transfer between fabrics. The purpose of the Ultra Ethernet effort is to move these behaviors into the standard itself, with packet spraying, modern congestion control, and selective retransmission defined rather than configured, so that a large cluster fabric can be built from standard parts and behave correctly without an in-house networking research group; see [[wiki/Ultra Ethernet|Ultra Ethernet]] and [[wiki/RDMA|RDMA]]. That is a good example of how this industry actually matures. A capability appears first as expert craft available to a few operators, and becomes infrastructure when it is written into a standard that anyone can buy. ## Relationships - **protects:** [[wiki/Collective Communication|Collective Communication]] · [[wiki/RDMA|RDMA]] - **implemented in:** [[wiki/InfiniBand|InfiniBand]] · [[wiki/Ultra Ethernet|Ultra Ethernet]] · [[wiki/DPU and SmartNIC|DPU and SmartNIC]] - **shaped by:** [[wiki/Network Fabric Topology|Network Fabric Topology]] - **observed through:** [[wiki/Observability|Observability]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** incast, synchronized burst, head-of-line blocking, and victim flow patterns; credit-based flow control; priority-based flow control and its backpressure propagation behavior; explicit congestion notification; endpoint congestion control algorithms; adaptive and packet-level routing against flow-hash routing; non-blocking provisioning removing interior fabric congestion. - **Established:** the historical requirement for per-deployment hand-tuning of lossless Ethernet parameters, and consortium standardization of packet spraying, congestion control, and selective retransmission as the response. - **Strongly indicated:** congestion configuration as the most common cause of production shortfall in cluster fabrics; operator experience is consistent and consolidated data is not public. ## Sources - Ultra Ethernet Consortium and InfiniBand Trade Association specifications, as current October 2026. - Data center congestion management engineering literature, as of October 2026.