# Intelligence Delivery Everything beneath this layer exists to produce capability; this layer delivers it. Weights become a service, requests become answers, and capacity becomes something an application developer can call. As deployed use has grown, this has become where most of the world's compute is actually spent, and where the economics of the whole buildout are settled. > [!note] The one-sentence version > Weights on one side, answers on the other, and a great deal of careful engineering in between deciding how many answers a megawatt produces. ## The mechanisms - **Phase separation.** Prefill reads the prompt in parallel and is arithmetic-bound; decode writes the answer serially and is memory-bound. Nearly every serving design decision follows from that asymmetry; see [[wiki/Prefill and Decode|Prefill and Decode]]. - **Cache management.** Keys and values retained so attention need not be recomputed, managed in pages so memory is neither wasted nor fragmented, and shared across requests with common prefixes; see [[wiki/KV Cache|KV Cache]]. - **Batching.** Many sequences served in one pass through the weights, with admission and retirement at every step, which is what converts a bandwidth-bound workload into a high-throughput one; see [[wiki/Continuous Batching|Continuous Batching]]. - **Numerics.** Fewer bits per value, so the same bandwidth carries more model per second and more sequences fit in memory; see [[wiki/Quantization|Quantization]]. - **Engines.** The runtimes implementing all of the above; see [[wiki/vLLM|vLLM]] · [[wiki/SGLang|SGLang]] · [[wiki/Inference Serving|Inference Serving]]. - **Routing.** Choosing the model and the hardware per request, which is what makes a mixed fleet usable; see [[wiki/Inference Request Routing|Inference Request Routing]]. - **Endpoints.** The standardized interface at which the stack's complexity is discharged; see [[wiki/Inference Endpoint|Inference Endpoint]]. - **Grounding.** Retrieving relevant material and supplying it in the prompt, so answers rest on a specified corpus; see [[wiki/Retrieval-Augmented Generation|Retrieval-Augmented Generation]]. - **Private deployment.** Serving a tenant's own or chosen weights on isolated capacity; see [[wiki/Private Model Hosting|Private Model Hosting]] · [[wiki/Confidential Computing|Confidential Computing]]. ## The figure of merit The useful measure of this layer is answers per unit of energy and capital, at a stated latency, context length, and precision. Any single number quoted without those conditions can be made to look excellent, and insisting on the full set is the most valuable habit available to anyone evaluating these systems. The economic consequence is direct: improvements here, in caching, batching, routing, and numerics, have reduced the energy and hardware required per answer by large factors over a short period, and that reduction is what carries capability from a few laboratories to ordinary American businesses, clinics, schools, and farms. ## Relationships - **composed of:** [[wiki/Prefill and Decode|Prefill and Decode]] · [[wiki/KV Cache|KV Cache]] · [[wiki/Continuous Batching|Continuous Batching]] · [[wiki/Quantization|Quantization]] · [[wiki/Inference Serving|Inference Serving]] · [[wiki/Inference Request Routing|Inference Request Routing]] · [[wiki/Inference Endpoint|Inference Endpoint]] - **runs on:** [[wiki/Inference Accelerator|Inference Accelerator]] · [[wiki/Compute and Accelerator Systems|Compute and Accelerator Systems]] - **orchestrated by:** [[wiki/Software and Orchestration|Software and Orchestration]] · [[wiki/Kubernetes|Kubernetes]] - **distributed through:** [[wiki/Cloud Deployment|Cloud Deployment]] · [[wiki/Edge Inference|Edge Inference]] - **civic dimension:** [[wiki/National Compute Capacity|National Compute Capacity]] · [[wiki/Infrastructure Citizenship|Infrastructure Citizenship]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** the mechanism set of phase separation, paged cache management with prefix sharing, step-level continuous batching, low-precision numerics, routing, standardized endpoints, retrieval grounding, and private deployment; the dependence of any performance figure on stated latency, context length, and precision. - **Established (datestamped):** as of October 2026, managed platforms in the industry provide model routing, hardware selection, multi-region balancing, serverless provisioning, private retrieval-grounded deployments, endpoint management, usage controls, and agent policy enforcement as standard capabilities. - **Strongly indicated:** large reductions in energy and hardware per answer over recent years from serving-layer engineering; the mechanisms are documented and a single aggregate figure is not asserted. ## Sources - Open inference engine documentation and managed inference service practice, as current October 2026.