# SGLang SGLang is an open-source serving engine designed around a particular observation: real applications do not send one isolated prompt and read one reply. They send programs, sequences of calls sharing prefixes, branching, looping, calling tools, and constraining output to a required shape. An engine that understands that structure can optimize across the whole interaction rather than one request at a time. > [!note] The one-sentence version > Treat a request as a program with structure and shared context rather than an isolated string, and the engine can reuse and constrain far more than it otherwise could. ## The two central ideas **Aggressive prefix reuse.** Workloads repeat prefixes constantly: a shared system prompt across every user, the same few-shot examples, one document queried many times, a conversation whose history is resent at each turn, a tree of candidate continuations sharing a trunk. The engine organizes cached keys and values in a radix tree keyed by token sequence, so any new request automatically reuses the longest prefix already cached, by any earlier request, without the application declaring anything. Computation for a shared prefix happens once and its cache is stored once; see [[wiki/KV Cache|KV Cache]]. The practical effect in multi-turn and shared-prompt workloads is substantial, and it arrives without application changes, which is the mark of a well-placed abstraction. **Constrained decoding.** When output must satisfy a grammar, valid JSON against a schema, a function call signature, a regular expression, the engine can restrict each step's token choice to those that keep the output valid, by compiling the grammar into a state machine and masking the model's output distribution accordingly. This guarantees well-formed output rather than validating and retrying, and with efficient implementation it costs little per step. For agent and tool-calling systems, where malformed output is a routine failure mode, the guarantee is worth more than a throughput increment; see [[wiki/Inference Serving|Inference Serving]]. ## What else it provides Continuous batching and the standard serving machinery; see [[wiki/Continuous Batching|Continuous Batching]]. Tensor and pipeline parallelism. Quantized weights and cache; see [[wiki/Quantization|Quantization]]. Chunked and disaggregated prefill and decode; see [[wiki/Prefill and Decode|Prefill and Decode]]. Speculative decoding. A front-end language for expressing multi-call generation programs, which is where the project's name and original purpose come from. And support for the principal open accelerator stack alongside the incumbent; see [[wiki/ROCm|ROCm]]. ## Why two engines rather than one The open engines overlap heavily and differ in emphasis, and the competition between them has been productive: paged cache management and radix prefix reuse both began in one project and spread to the other and to vendor stacks. The useful reading is that this part of the stack matured in public, quickly, with ideas moving freely, and the resulting serving layer is far better than any single operator would have built privately. Open development has been the fastest-moving layer in the entire stack, and it is the layer where a new accelerator's prospects are decided; see [[wiki/vLLM|vLLM]]. ## Relationships - **manages:** [[wiki/KV Cache|KV Cache]] through radix prefix reuse - **implements:** [[wiki/Continuous Batching|Continuous Batching]] · [[wiki/Prefill and Decode|Prefill and Decode]] · [[wiki/Quantization|Quantization]] - **runs on:** [[wiki/CUDA|CUDA]] · [[wiki/ROCm|ROCm]] - **peer engine:** [[wiki/vLLM|vLLM]] - **serves:** [[wiki/Inference Serving|Inference Serving]] · [[wiki/Retrieval-Augmented Generation|Retrieval-Augmented Generation]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** radix-tree organization of cached keys and values enabling automatic longest-prefix reuse across requests; grammar-constrained decoding by compiling a grammar to a state machine and masking the output distribution; continuous batching, tensor and pipeline parallelism, quantization, chunked and disaggregated prefill, and speculative decoding; a front-end language for multi-call generation programs. - **Established (datestamped):** as of October 2026, the engine supports both the incumbent and the principal open accelerator software stacks. - **Established:** cross-pollination of paged cache management and radix prefix reuse between the leading open engines and into vendor stacks. ## Sources - [Helios Is AMD's First AI System To Rival Nvidia Vera Rubin](https://www.cnbc.com/) — CNBC, October 2026; open accelerator stack support for leading inference engines. - Open inference engine documentation and structured generation literature, as current October 2026.