# vLLM vLLM is an open-source engine for serving language models. Its contribution was a specific idea borrowed from operating systems, applied to the memory that dominates inference, and the result was a large improvement in throughput that became the industry's baseline rather than one project's advantage. > [!note] The one-sentence version > Manage the attention cache in fixed-size pages, as an operating system manages memory, and the waste that limited serving throughput disappears. ## The idea A serving engine must hold each active sequence's keys and values; see [[wiki/KV Cache|KV Cache]]. Early engines allocated a contiguous block per sequence sized to the maximum length the sequence might reach. Since most sequences finish far short of their maximum, most of that memory was reserved and never used, and because the blocks were contiguous, the free space between them could not be combined for a new sequence. Memory waste from this internal fragmentation was severe, and because cache memory determines batch size, the waste translated directly into lost throughput. The engine's answer was paging. The cache is divided into fixed-size blocks; a sequence is allocated blocks as it grows and holds a table mapping its logical positions to physical blocks. Blocks need not be contiguous, so fragmentation nearly vanishes, and allocation is incremental, so nothing is reserved against a maximum that may never arrive. Attention kernels were written to read through the block table, which is the implementation work that made the scheme viable. Two further benefits follow from paging, and they are the elegant part. **Sharing**: several sequences with a common prefix, the same system prompt, the same few-shot examples, the same document, can point at the same physical blocks, so that prefix is stored once and its prefill is computed once; with copy-on-write for divergence, this is a large saving in the common case of many users sharing a prompt template. **Preemption**: a sequence's blocks can be evicted and restored, which lets the scheduler reclaim memory under pressure rather than refusing work. ## What else it provides Continuous batching at step granularity; see [[wiki/Continuous Batching|Continuous Batching]]. Tensor and pipeline parallelism for models exceeding one device. Quantized weight and cache support; see [[wiki/Quantization|Quantization]]. Chunked prefill and disaggregated prefill and decode; see [[wiki/Prefill and Decode|Prefill and Decode]]. Prefix caching across requests. Speculative decoding, in which a small model proposes tokens a large model verifies in a batch. And a widely adopted request interface, which has made it close to a default serving component rather than a specialist tool. ## Why open engines mattered strategically An open engine that runs well on several vendors' accelerators converts a hardware purchase decision into a genuine choice. The leading open engines support the principal open accelerator stack alongside the incumbent, which means an operator buying alternative silicon inherits a mature serving layer rather than building one; see [[wiki/ROCm|ROCm]] and [[wiki/SGLang|SGLang]]. For a country assembling compute capacity from more than one supply chain, that portability is worth as much as any hardware specification; see [[wiki/National Compute Capacity|National Compute Capacity]]. ## Relationships - **manages:** [[wiki/KV Cache|KV Cache]] - **implements:** [[wiki/Continuous Batching|Continuous Batching]] · [[wiki/Prefill and Decode|Prefill and Decode]] · [[wiki/Quantization|Quantization]] - **runs on:** [[wiki/CUDA|CUDA]] · [[wiki/ROCm|ROCm]] - **peer engine:** [[wiki/SGLang|SGLang]] - **deployed through:** [[wiki/Inference Serving|Inference Serving]] · [[wiki/Kubernetes|Kubernetes]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** contiguous maximum-length cache allocation and the internal fragmentation it produced; paged block allocation with per-sequence block tables removing that fragmentation; attention kernels reading through block tables; prefix block sharing with copy-on-write; block eviction and restoration for preemption; continuous batching, tensor and pipeline parallelism, quantization support, chunked and disaggregated prefill, prefix caching, and speculative decoding. - **Established (datestamped):** as of October 2026, the leading open inference engines support both the incumbent and the principal open accelerator software stacks. - **Strongly indicated:** open engine portability materially improving buyer position in accelerator procurement; the mechanism is clear and measured effects are still accruing. ## Sources - [Helios Is AMD's First AI System To Rival Nvidia Vera Rubin](https://www.cnbc.com/) — CNBC, October 2026; open accelerator stack support for leading inference engines. - Open inference engine documentation and paged attention literature, as current October 2026.