# InfiniBand InfiniBand is a network built from the start for one job: moving data between the nodes of a computing cluster with the lowest achievable latency and without dropping anything. It achieved that by refusing several conveniences the internet's protocols provide, and it held the leading position in high-performance cluster networking for two decades, including through the first years of large-scale model training. > [!note] The one-sentence version > A network that never drops a packet because a sender is forbidden to transmit until the receiver has confirmed room for it. ## Why it exists Ethernet and the internet protocols were designed for a world of unreliable links and unknown peers, so they tolerate loss and recover from it: a dropped packet is retransmitted, and congestion is inferred from loss. That is excellent engineering for a global network and poor engineering for a cluster, where loss and retransmission destroy the tail latency that collective operations depend on. A group operation runs at the speed of its slowest participant, so one retransmission delays everyone; see [[wiki/Collective Communication|Collective Communication]]. InfiniBand takes the opposite position. The fabric is small, known, and centrally managed, so it can be made lossless by construction. ## Prerequisites - **Credit-based flow control:** a sender transmits only when it holds credits confirming buffer space at the receiver; without credits it waits. Loss from buffer overrun therefore cannot occur. - **Subnet manager:** a central service that discovers the topology, assigns addresses, computes routing tables, and programs every switch; the fabric is configured rather than self-organizing. - **Remote direct memory access:** writing into a remote node's memory without involving its processor or operating system; see [[wiki/RDMA|RDMA]]. This is InfiniBand's native operating model rather than an addition to it. - **Queue pair:** the paired send and receive queues through which an application posts work directly to the adapter, bypassing the kernel. - **Adaptive routing:** choosing among equal paths based on observed load, which matters greatly on the fat-tree topologies these fabrics use; see [[wiki/Network Fabric Topology|Network Fabric Topology]]. - **In-network reduction:** performing arithmetic inside switches so that collective sums complete as data traverses the fabric. ## How it works An application registers memory with the adapter and posts a work request to a queue pair. The adapter reads the data by direct memory access and sends it; switches forward it along the path the subnet manager computed, and credits ensure no stage overruns. At the far end the adapter writes directly into the registered memory and may signal completion. The processor at the receiving end was never involved, which is where both the latency and the efficiency come from: a large transfer costs the receiving node almost no processor time at all. ## Position, and the live competition - **Upstream:** adapter and switch silicon, and the optics or copper connecting them. - **Downstream:** collective libraries, training frameworks, and parallel storage systems, which commonly run over the same fabric; see [[wiki/Parallel File System|Parallel File System]]. - **Alternatives:** Ethernet, enhanced through consortium work with the loss and congestion behavior clusters need, has become a serious competitor, with the attraction of a vastly larger supplier base and a workforce already trained on it; see [[wiki/Ultra Ethernet|Ultra Ethernet]]. Several very large operators have chosen enhanced Ethernet for new clusters, which is the clearest signal available about where the market is going. - **Degraded case:** on a lossy network, collective operations suffer tail latency that reduces the efficiency of large training runs substantially, which is the whole reason this category exists. ## Lineage and trajectory InfiniBand emerged around 2000 from an effort to replace the server bus, found its market in high-performance computing, and dominated the fastest machines for twenty years. Its principles, losslessness, direct memory access, kernel bypass, and central management, have proved durable enough that the Ethernet world has adopted most of them rather than defeating them. The present trajectory is therefore a convergence rather than a displacement: the cluster-fabric ideas won, and the argument now is about which physical and commercial ecosystem carries them. ## Relationships - **implements:** [[wiki/Scale-Up and Scale-Out|Scale-Up and Scale-Out]] scale-out - **transports:** [[wiki/RDMA|RDMA]] · [[wiki/Collective Communication|Collective Communication]] · [[wiki/Parallel File System|Parallel File System]] - **competes with:** [[wiki/Ultra Ethernet|Ultra Ethernet]] - **arranged in:** [[wiki/Network Fabric Topology|Network Fabric Topology]] - **carried by:** [[wiki/Data Center Optics|Data Center Optics]] · [[wiki/Structured Cabling|Structured Cabling]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** credit-based flow control producing losslessness by construction; centralized subnet management; native remote direct memory access with kernel bypass through queue pairs; adaptive routing on fat-tree topologies; in-network reduction; the fabric's origin around 2000 and its long dominance in high-performance computing. - **Established:** the sensitivity of collective operations to tail latency and therefore to packet loss. - **Established (datestamped):** as of October 2026, several very large operators have selected consortium-enhanced Ethernet for new large-scale clusters. - **Strongly indicated:** convergence of Ethernet toward lossless cluster semantics rather than displacement of those semantics; consortium specifications and shipping products support it and long-run share data is still accruing. ## Sources - InfiniBand Trade Association specifications and high-performance cluster networking practice, as current October 2026. - [Helios Is AMD's First AI System To Rival Nvidia Vera Rubin](https://www.cnbc.com/) — CNBC, October 2026; consortium-aligned Ethernet as the scale-out fabric in a current rack-scale architecture.