# Inference Accelerator Training a model and serving it are different industrial problems, and they reward different machines. Training is a long batch job on an enormous dataset, performed once, where raw arithmetic throughput and the ability to couple thousands of devices dominate. Inference is a continuous stream of small requests, each of which must be answered quickly, where memory bandwidth, latency, and energy per answer dominate. An inference accelerator is a processor designed for the second problem. > [!note] The one-sentence version > Training rewards the largest possible arithmetic engine; inference rewards the fastest possible path from memory to an answer, and that difference is now worth its own silicon. ## Why it exists Generating one token of a model's response reads a large share of the model's parameters and performs comparatively little arithmetic on each one. The work is bandwidth-bound by nature; see [[wiki/Memory Wall|Memory Wall]]. A device optimized for peak arithmetic therefore spends much of its time waiting, while a device optimized for streaming parameters out of memory and keeping latency low delivers more answers per second and more answers per watt. As deployed use grows, the total silicon devoted to answering requests grows past the silicon devoted to training, and the economics of the serving machine become the economics of the industry. ## Prerequisites - **Prefill and decode:** the two phases of serving a request; prefill processes the prompt in parallel and is arithmetic-bound, decode generates tokens one at a time and is memory-bound. See [[wiki/Prefill and Decode|Prefill and Decode]]. - **Latency and throughput:** time to first token and tokens per second for one user, against total tokens per second across all users; serving design is always a negotiation between them. - **Batching:** grouping concurrent requests so one pass through the parameters serves many users; see [[wiki/Continuous Batching|Continuous Batching]]. - **Long context:** requests carrying very large prompts, which shift the balance back toward prefill and place heavy demands on cache memory. ## Design strategies **Maximize memory bandwidth per device.** Wide stacked memory, large on-die memory, or both, so that parameters stream to the arithmetic without stalling. **Keep the model resident.** Architectures with very large on-chip memory avoid external memory traffic entirely for models that fit, which collapses latency; the constraint becomes how large a model will fit. **Deterministic scheduling.** Removing caches and dynamic scheduling in favor of a statically planned dataflow makes latency predictable, which matters when a service promises a response time rather than an average. **Specialize the numerics.** Hardware support for low-precision formats multiplies effective bandwidth and arithmetic in the same step; see [[wiki/Quantization|Quantization]]. **Optimize energy per token.** At scale the operating cost of serving is electricity, so joules per answer becomes the figure of merit rather than peak arithmetic. The field is genuinely plural. Merchant GPUs serve most inference today because they are available, programmable, and well supported by software. Custom silicon from large operators serves their own traffic. Several independent architectures target long-context and latency-sensitive serving specifically, including designs with unusually large on-device memory and designs occupying most of a wafer. Dedicated inference cards from established processor vendors target the same workloads at lower power per device. ## Stack position - **Upstream:** the same [[wiki/Foundry|foundry]], [[wiki/Advanced Packaging|packaging]], and memory supply as training silicon, competing for the same capacity. - **Downstream:** [[wiki/Inference Serving|inference serving]] software, endpoints, and the applications above them. - **Function and vendor:** the function is expanding and the suppliers are many; software maturity, rather than peak specifications, decides most deployments. - **Degraded case:** inference runs on training-optimized hardware, which works and costs more energy and capital per answer delivered. ## Lineage and trajectory Specialized inference silicon appeared first at the edge, in phones and cameras, where power budgets left no choice. Data-center inference parts followed, initially as low-power cards for vision and speech, then as architectures built for language models once serving volume justified them. The trajectory follows deployment: as the share of compute devoted to answering requests rises, the pressure toward purpose-built serving hardware rises with it, and the decisive competitive factor is whether the software ecosystem makes a new architecture easy to adopt. Hardware wins markets through compilers and runtimes. ## Relationships - **specialization of:** [[wiki/Accelerator|Accelerator]] - **shaped by:** [[wiki/Memory Wall|Memory Wall]] · [[wiki/Prefill and Decode|Prefill and Decode]] · [[wiki/Quantization|Quantization]] - **served by:** [[wiki/Inference Serving|Inference Serving]] · [[wiki/vLLM|vLLM]] · [[wiki/SGLang|SGLang]] - **selected by:** [[wiki/Inference Request Routing|Inference Request Routing]] - **related:** [[wiki/Edge Inference|Edge Inference]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** the arithmetic-bound and memory-bound division between prefill and decode; batching as the throughput mechanism; dedicated data-center inference parts and independent inference architectures as shipping products; edge neural processing units in phones, laptops, cars, cameras, and robots. - **Strongly indicated:** inference overtaking training as the larger share of deployed compute; the direction is consistent across operator disclosures and the crossover point is not published. - **Unresolved:** which inference architectures hold durable share; vendor performance comparisons remain vendor claims until independently benchmarked. ## Sources - [How Nvidia GPUs Compare To Google's And Amazon's AI Chips](https://www.cnbc.com/) — CNBC, October 2026; edge accelerators, specialist inference architectures, dedicated data-center inference parts. - Public vendor documentation for inference-oriented accelerator families and serving runtimes, as of October 2026.