# Inference Serving Training produces a file of parameters. Serving turns that file into something a million people can ask questions of, reliably, within a promised response time, at a cost per answer that the business can carry. It is the layer at which the entire stack beneath becomes a product, and as deployment has grown it has become where most of the world's compute is actually spent. > [!note] The one-sentence version > Everything between a trained model and a user's answer: workers holding the weights, a queue in front, a promise about latency, and a ledger of what was used. ## What the layer contains - **Model workers.** Processes holding weights in accelerator memory and running an engine; see [[wiki/vLLM|vLLM]] and [[wiki/SGLang|SGLang]]. A worker's capacity is set by memory, batch size, and context length. - **Admission and queueing.** Deciding which request enters which worker and when, with priorities and limits per tenant, which is where a service's fairness is actually implemented. - **Routing.** Selecting among models and hardware classes for a given request; see [[wiki/Inference Request Routing|Inference Request Routing]]. - **Autoscaling.** Adjusting worker count against load. This is harder than ordinary service autoscaling, because a worker must load tens or hundreds of gigabytes of weights before it can serve, so the cold-start interval is long and scaling must anticipate rather than react; warm pools and pre-loaded weights are the standard remedies. - **Model lifecycle.** Loading, versioning, promoting a new version gradually, holding several versions concurrently, and rolling back; models are artifacts under change management like any other deployed software. - **Latency commitments.** Time to first token and inter-token latency as the user-visible promises, which set the batch size ceiling and therefore the achievable throughput; see [[wiki/Continuous Batching|Continuous Batching]]. - **Accounting.** Metering tokens, requests, and accelerator time per tenant, which is how capacity is allocated, limited, and charged. Usage controls at the organizational level are now a standard expectation rather than an advanced feature. - **Policy enforcement.** Rate limits, content and tool-use policy, and the controls applied to autonomous agent workloads whose behavior is less predictable than a single request. ## The two numbers that define a service Throughput for the fleet and latency for the person pull against each other, through batch size. A system reporting only throughput can be running batches so large that users wait; a system reporting only latency can be running devices at a fraction of their capacity. Any serious characterization states both together, along with the context length and precision they were measured at. That discipline, insisting on the full set of conditions behind a performance figure, is the single most useful habit a reader of this material can acquire. ## Why serving became the center of gravity Training a model is a project; serving it is a utility. As deployed use grows, the total compute devoted to answering requests passes the compute devoted to training, and the facility's design question shifts from how large a run can be assembled to how cheaply and reliably an answer can be produced. That shift pulls the whole stack with it: toward memory bandwidth over peak arithmetic, toward low precision, toward hardware specialized for serving, toward capacity sited near users, and toward operations measured in availability rather than in run completion; see [[wiki/Inference Accelerator|Inference Accelerator]] and [[wiki/Edge Inference|Edge Inference]]. ## Relationships - **built on:** [[wiki/vLLM|vLLM]] · [[wiki/SGLang|SGLang]] · [[wiki/Prefill and Decode|Prefill and Decode]] · [[wiki/Continuous Batching|Continuous Batching]] - **orchestrated by:** [[wiki/Kubernetes|Kubernetes]] - **routes through:** [[wiki/Inference Request Routing|Inference Request Routing]] · [[wiki/Inference Endpoint|Inference Endpoint]] - **operated through:** [[wiki/Observability|Observability]] · [[wiki/Site Reliability Engineering|Site Reliability Engineering]] - **part of:** [[wiki/Intelligence Delivery|Intelligence Delivery]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** the serving layer's components of workers, admission and queueing, routing, autoscaling, model lifecycle management, latency commitments, usage accounting, and policy enforcement; long cold-start intervals from weight loading and warm pools as the standard remedy; the inverse relationship between batch size, fleet throughput, and per-user latency. - **Established:** organization-level usage controls and agent policy enforcement as standard expectations in managed serving platforms. - **Strongly indicated:** inference surpassing training as the larger share of deployed compute; the direction is consistent across operator disclosures and the crossover point is not published. ## Sources - Managed inference service documentation and production serving practice, as current October 2026.