# Inference Endpoint An endpoint is the address at which a model can be asked a question. Everything in this collection, the silicon, the power, the cooling, the fabric, the schedulers, the serving engines, exists so that a request arriving at one of these addresses receives a good answer quickly. It is the boundary where infrastructure becomes capability. > [!note] The one-sentence version > One URL, one request shape, and every layer beneath becomes somebody else's concern. ## What it provides - **A standardized request shape.** A widely adopted request and response format for chat-style completion became a de facto standard, so that applications can move between providers and self-hosted deployments with little change. This portability, like the storage interface standard, arrived by adoption rather than by committee and is enormously valuable to buyers; see [[wiki/S3-Compatible Object Storage|S3-Compatible Object Storage]]. - **Streaming.** Tokens delivered as generated rather than after completion, which is what makes a long answer feel responsive and which is why time to first token is the latency figure that matters most. - **Authentication and quota.** Keys or tokens identifying the caller, with rate and spend limits attached; see [[wiki/Inference Serving|Inference Serving]]. - **Model selection.** Naming a model or a capability tier, with the resolution to specific weights and hardware handled behind the boundary; see [[wiki/Inference Request Routing|Inference Request Routing]]. - **Tool and function calling.** A structured protocol by which the model requests an action and receives a result, which is the mechanism beneath agent systems and which depends on reliably well-formed output; see [[wiki/SGLang|SGLang]]. - **Dedicated and shared deployments.** An endpoint served from pooled capacity, against one served from reserved capacity for a single tenant, with the predictable latency and isolation that reservation buys. ## The deployment shapes **Shared multi-tenant.** Requests from many customers share workers; the most efficient use of hardware and the least predictable latency under load. **Dedicated.** Capacity reserved for one tenant, serving a chosen model, with predictable performance and isolation; the common enterprise arrangement, and the one that permits a tenant's own fine-tuned weights to be served; see [[wiki/Private Model Hosting|Private Model Hosting]]. **Serverless and scale-to-zero.** Capacity provisioned on demand, attractive for intermittent use and subject to the cold-start interval that weight loading imposes. **Self-hosted.** The same interface served from a tenant's own facility, which is what makes the endpoint abstraction genuinely portable rather than a lock-in surface. ## Why the boundary matters The endpoint is where the stack's complexity is discharged. An application developer specifies a model and a prompt; on the other side sit wafers, lithography, packaging, substations, chillers, fabrics, schedulers, and engines, none of which the caller need know. That is what an infrastructure achievement looks like when it is finished: the difficulty is absorbed, and what remains visible is a simple request that works. The same was true of the electrical outlet and the dial tone, and it is the reason this capability will reach every American business rather than only the firms able to build it. ## Relationships - **fronts:** [[wiki/Inference Serving|Inference Serving]] · [[wiki/Inference Request Routing|Inference Request Routing]] - **delivered from:** [[wiki/Bare Metal Compute|Bare Metal Compute]] · [[wiki/Neocloud|Neocloud]] · [[wiki/Hyperscaler|Hyperscaler]] · [[wiki/Edge Inference|Edge Inference]] - **isolated through:** [[wiki/Private Model Hosting|Private Model Hosting]] · [[wiki/Confidential Computing|Confidential Computing]] - **part of:** [[wiki/Intelligence Delivery|Intelligence Delivery]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** a widely adopted chat completion request format functioning as a de facto standard across providers and self-hosted deployments; token streaming and time to first token as the governing latency figure; key-based authentication with rate and spend limits; model and tier selection resolved behind the interface; structured tool and function calling; shared, dedicated, serverless, and self-hosted deployment shapes; cold-start intervals imposed by weight loading. - **Established (datestamped):** as of October 2026, managed platforms in the industry offer endpoint management, usage controls, private deployments, and serverless model provisioning as standard capabilities. ## Sources - Managed inference service and model API documentation, as current October 2026.