# PCIe PCI Express is the bus by which everything inside a server reaches the processor: accelerators, network interfaces, storage devices, and management hardware. It is the most successful interconnect standard in computing, it is present in every machine in the building, and understanding what it does well explains precisely why a separate accelerator fabric had to be invented. > [!note] The one-sentence version > The universal road from the processor to everything else, excellent at reaching devices and too slow and too shared for accelerators talking among themselves. ## How it works PCIe is a point-to-point serial interconnect. A **lane** is a pair of differential wires in each direction; devices connect over one, four, eight, or sixteen lanes, and bandwidth scales with the count. Each **generation** roughly doubles the per-lane rate: the generations in current and imminent server use run through the fourth, fifth, and sixth, with per-lane rates rising accordingly. Devices connect to the processor's **root complex**, either directly or through a **switch** that fans a few processor lanes out to many devices, at the cost of sharing bandwidth among them. Traffic is packetized, credit-flow-controlled, and error-checked, and a device's memory can be mapped into the host's address space so the processor reads and writes it as memory. Two refinements matter in this context. **Peer-to-peer** transfers let two devices on the same hierarchy exchange data without staging through host memory, which is how an accelerator can read directly from a network interface or a flash device. And **Compute Express Link** rides on the same physical layer while adding cache coherence, which allows memory attached to a device to participate in the processor's coherent memory space; it is the standards-based route to pooled and tiered memory, and its adoption is growing. ## What it carries, and what it does not In a current accelerator server PCIe carries the links from the host to each accelerator for control, kernel launch, and data staging; the network interfaces; the boot and data flash devices; and the management hardware. What it does not carry is accelerator-to-accelerator traffic during training. Tensor-parallel work generates sustained exchange between accelerators at hundreds of gigabytes per second, and a bus designed for reaching devices through a shared hierarchy, with the host in the address path, is both too slow and too contended for it. That gap is the entire reason dedicated accelerator fabrics exist; see [[wiki/NVLink|NVLink]], [[wiki/Infinity Fabric|Infinity Fabric]], and [[wiki/UALink|UALink]]. The lesson is general and worth keeping: a bus built to reach a processor is the wrong instrument for peers talking to each other, and recognizing that distinction is what produced the modern rack. ## Why lane budgets shape server design A processor offers a finite number of lanes, and every device wants some. Eight accelerators at sixteen lanes each, several network interfaces at sixteen, and a set of flash devices will exceed one processor's budget, which is why accelerator servers use two processors, PCIe switches, or both, and why the physical layout of a motherboard is largely a negotiation over lanes. The same budget determines whether storage and network traffic can reach accelerators directly, which is the mechanism behind high-throughput data loading; see [[wiki/NVMe over Fabrics|NVMe over Fabrics]]. ## Relationships - **connects:** [[wiki/Host CPU|Host CPU]] · [[wiki/Accelerator|Accelerator]] · [[wiki/NVMe SSD|NVMe SSD]] · [[wiki/DPU and SmartNIC|DPU and SmartNIC]] - **insufficient for:** accelerator peer traffic, answered by [[wiki/NVLink|NVLink]] · [[wiki/Infinity Fabric|Infinity Fabric]] · [[wiki/UALink|UALink]] - **extended by:** Compute Express Link for coherent attached memory - **part of:** [[wiki/Compute Tray|Compute Tray]] · [[wiki/Rack and Cluster Integration|Rack and Cluster Integration]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** lane and generation structure with approximate per-generation doubling of per-lane rate; root complex and switch topology with bandwidth sharing; packetized credit-controlled transport; memory mapping of device memory; peer-to-peer transfers between devices; Compute Express Link adding coherence over the same physical layer. - **Established:** the insufficiency of PCIe for sustained accelerator-to-accelerator training traffic, and dedicated accelerator fabrics as the industry's answer. - **Strongly indicated:** growing adoption of coherent attached memory in production servers; products ship and deployment breadth is still developing. ## Sources - PCI-SIG and Compute Express Link Consortium specifications, as current October 2026. - Public accelerator server architecture documentation, as of October 2026.