# Continuous Batching Batching means serving many requests in a single pass through the model, so that one read of the weights produces tokens for many users at once. Continuous batching makes that practical for interactive serving by admitting new requests and retiring finished ones at every generation step rather than at batch boundaries. The technique is simple to describe and it changed inference economics by a large factor. > [!note] The one-sentence version > Read the weights once and serve everybody in the batch; let people join and leave the batch every single step rather than waiting for the slowest member. ## Why it exists Decode is memory-bandwidth-bound: each step reads the model's weights out of stacked memory and performs comparatively little arithmetic on them; see [[wiki/Prefill and Decode|Prefill and Decode]]. Serving one user, almost all that bandwidth is spent fetching weights to produce a single token, and the accelerator's arithmetic units are nearly idle. Serving thirty-two users in one pass fetches the same weights once and produces thirty-two tokens, so throughput rises by nearly the batch factor at almost no additional bandwidth cost. Batch size, rather than arithmetic throughput, therefore sets decode throughput per device. Static batching cannot deliver this in an interactive service. Requests arrive at unpredictable times and finish after very different numbers of tokens; a fixed batch waits for the slowest member before anyone's results are released, and arriving requests wait for the next batch. Latency becomes unacceptable and devices sit partly idle. ## How it works The engine maintains a running set of active sequences and operates on the step as its unit rather than the batch. At each generation step it performs one forward pass over every active sequence, producing one token for each. A sequence that emits its end token is removed immediately and its cache memory is released. Waiting requests are admitted as soon as there is capacity, which the engine determines from free cache blocks rather than from a fixed count; see [[wiki/KV Cache|KV Cache]]. Prefills for newly admitted requests are interleaved among decode steps, in chunks, so that a long prompt does not stall the batch. The result is a device that stays near its achievable throughput continuously while individual users see latency governed by the current batch size rather than by the slowest request in a cohort. The memory accounting is what makes it work: a request is admitted when its cache can be allocated, so the engine's admission control is really a memory allocator, and paged cache management is what allows that allocator to be efficient. ## What it costs and what it trades Larger batches raise throughput and raise per-user latency, because each step does more work; serving therefore has an explicit knob between tokens per second for the fleet and tokens per second for one person. Operators set it against a latency commitment, and the honest figures quoted for a serving system always state both numbers together, since either alone can be made to look excellent. Long contexts consume cache memory per sequence, so a workload of very long prompts supports a smaller batch than one of short prompts at the same memory, which is why context length and achievable throughput are linked. ## Relationships - **requires:** [[wiki/KV Cache|KV Cache]] - **exploits:** [[wiki/Prefill and Decode|Prefill and Decode]] · [[wiki/Memory Wall|Memory Wall]] - **implemented in:** [[wiki/vLLM|vLLM]] · [[wiki/SGLang|SGLang]] · [[wiki/Inference Serving|Inference Serving]] - **traded against:** latency commitments in [[wiki/Inference Endpoint|Inference Endpoint]] - **raises:** [[wiki/Utilization|Utilization]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** the memory-bandwidth bound on decode and the near-linear throughput gain from batching one weight read across many sequences; step-level admission and retirement; immediate cache release on completion; admission control governed by free cache blocks; chunked prefill interleaving; the explicit trade between fleet throughput and per-user latency; the inverse relationship between context length and achievable batch size at fixed memory. - **Established:** continuous batching implemented in the leading open inference engines and in production serving systems. ## Sources - Open inference engine documentation and transformer serving literature, as current October 2026.