# Prefill and Decode When a language model answers, it does two physically different things in sequence. First it reads the entire prompt at once, every token in parallel, and produces the first token of its reply. Then it writes the remainder one token at a time, each conditioned on everything before it. Those phases are prefill and decode, and the fact that they stress different parts of the hardware is among the deepest facts in how serving is engineered. > [!note] The one-sentence version > Prefill reads the whole question at once and is limited by arithmetic; decode writes the answer one word at a time and is limited by memory bandwidth. ## Why the distinction exists A transformer generates one token at a time, and to choose each one it runs attention against representations of every preceding token. Those representations never change once computed, so a serving engine stores them rather than recomputing; see [[wiki/KV Cache|KV Cache]]. Against that mechanism the two phases fall out naturally. The prompt is fully known, so all its positions can be processed simultaneously. The answer is not, because each token did not exist until the previous one was chosen. The same model on the same device therefore alternates between a highly parallel workload and a strictly serial one. ## Prefill: compute-bound All prompt positions are processed together. Attention runs across the whole prompt under a causal mask, and every layer's matrix multiplications carry sequence-length-sized dimensions. Arithmetic intensity is high, so throughput tracks the accelerator's dense matrix rate and a modern part runs prefill near its compute ceiling. Cost grows roughly linearly with prompt length in the feed-forward work and quadratically in the attention score computation before kernel-level mitigation, which is why very long prompts dominate **time to first token**, the delay a user waits before any text appears. ## Decode: memory-bound Decode has no parallelism across output tokens. Each step is a full forward pass whose matrices are narrow, since each sequence contributes one new token, while the bytes moved stay large: every weight matrix is read out of the accelerator's stacked memory at every step, and the accumulated cache is read back every step too. Arithmetic intensity collapses, so tokens per second tracks memory bandwidth rather than arithmetic throughput; see [[wiki/Memory Wall|Memory Wall]]. Two consequences follow. Memory capacity and bandwidth are the decode resource. And utilization is recovered only by batching many sequences' decode steps together, so that one weight read serves many tokens; see [[wiki/Continuous Batching|Continuous Batching]]. ## The asymmetry's consequences No single accelerator profile serves both phases well: prefill wants compute density, decode wants bandwidth. Mixed on one engine they also contend at the scheduler, because a long prefill run monolithically stalls the inter-token latency of every decode in the batch. Engines mitigate with **chunked prefill**, splitting a prefill so decode steps interleave between chunks. Carried to its conclusion the asymmetry motivates **disaggregated serving**: separate worker pools for each phase, with the newly written cache blocks transferred across the fabric by direct memory access from a prefill worker to a decode worker; see [[wiki/RDMA|RDMA]]. Each pool then scales against its own bottleneck, and heterogeneous accelerators per phase become natural rather than exotic; see [[wiki/Inference Accelerator|Inference Accelerator]] and [[wiki/Inference Request Routing|Inference Request Routing]]. The costs are real: cache migration adds network traffic, placement must respect locality, and the control plane spans two schedulers. Treating inference as one undifferentiated workload degrades every serving number at once: time to first token rises as long prefills stall decodes, throughput per device falls as batch size is capped to protect interactive latency, and fleet procurement mismatches parts to phases. Every latency commitment and every tokens-per-second figure is, underneath, a position taken on this asymmetry. ## Relationships - **uses:** [[wiki/KV Cache|KV Cache]] · [[wiki/Continuous Batching|Continuous Batching]] - **shaped by:** [[wiki/Memory Wall|Memory Wall]] - **implemented in:** [[wiki/vLLM|vLLM]] · [[wiki/SGLang|SGLang]] · [[wiki/Inference Serving|Inference Serving]] - **disaggregated over:** [[wiki/RDMA|RDMA]] - **informs:** [[wiki/Inference Request Routing|Inference Request Routing]] · [[wiki/Inference Accelerator|Inference Accelerator]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** prefill parallel across prompt positions and decode strictly serial across output tokens; cached keys and values read at every decode step; prefill compute-bound and decode memory-bandwidth-bound as engineering consensus demonstrated across hardware generations; prompt-length scaling of prefill cost and its dominance of time to first token; batching as the decode utilization mechanism. - **Established:** chunked prefill and disaggregated prefill and decode with cache transfer, implemented in open serving engines and deployed in production systems. - **Unresolved:** the crossover point at which a given model and accelerator pair tips between the two bounds; it is workload and hardware specific and no universal figure exists. ## Sources - Transformer serving engineering literature and open inference engine documentation, as current October 2026.