# Quantization Quantization represents a model's numbers in fewer bits than it was trained in. Sixteen-bit weights become eight-bit or four-bit values, with scaling factors to preserve their range. The immediate effect is less memory and less memory traffic per parameter, and because inference is bandwidth-bound, less traffic means proportionally more tokens per second. > [!note] The one-sentence version > Store each number in half or a quarter of the space, move half or a quarter of the bytes, and generate proportionally more tokens from the same hardware. ## Why it is a bandwidth strategy first Decode reads the entire weight set from stacked memory at every step; see [[wiki/Prefill and Decode|Prefill and Decode]] and [[wiki/Memory Wall|Memory Wall]]. Halving the bytes per weight nearly doubles the step rate, before any arithmetic consideration arises. Three further gains follow. A model that was too large for a given accelerator's memory now fits, which can remove the need to split it across devices and remove the communication that splitting entails. The cache occupies less memory when quantized, so more sequences fit and batch size rises; see [[wiki/Continuous Batching|Continuous Batching]]. And where the hardware has native support for the format, arithmetic throughput rises as well, which is why current accelerators publish headline figures at four-bit precision. ## The formats - **Sixteen-bit floating point,** in half-precision and brain-float variants; the usual training and baseline inference precision. - **Eight-bit floating point,** now widely supported in hardware, in variants trading exponent range against mantissa precision; the current practical default for high-quality inference. - **Eight-bit integer,** the long-standing quantized format, requiring careful scaling. - **Four-bit formats,** both integer and floating point, including block-scaled schemes in which small groups of values share an exponent; these are where the largest gains and the real accuracy questions live, and where current rack-scale systems quote their peak arithmetic figures. - **Mixed precision,** in which sensitive parts of a model, often attention and normalization, retain higher precision while the bulk of the weights are reduced. ## How it is done **Post-training quantization** converts an already-trained model, calibrating scaling factors from a sample of representative data; it is fast, needs no training infrastructure, and is the common path. **Quantization-aware training** simulates the reduced precision during training so the model learns to tolerate it, yielding better accuracy at very low precision at the cost of a training run. **Granularity** matters greatly: scaling per tensor is simplest, per channel better, and per small block best, since a single outlier value otherwise forces a scale that crushes the precision of everything near it. Outlier handling, keeping a small set of extreme values in higher precision, is a key technique behind modern low-bit methods. ## The honest accuracy position At eight bits, with good scaling, quality loss on general tasks is small enough that production serving at this precision is routine. At four bits the position is genuinely mixed: with careful block scaling, outlier handling, and sometimes quantization-aware training, losses are modest on many tasks and more visible on others, particularly tasks requiring precise reasoning or long chains of inference. The responsible practice is to evaluate the quantized model on the actual task rather than relying on an aggregate benchmark, because the degradation is uneven across task types. Stating it plainly: the gains are large and real, and the verification obligation is on the deployer. ## Relationships - **reduces pressure on:** [[wiki/Memory Wall|Memory Wall]] · [[wiki/High-Bandwidth Memory|High-Bandwidth Memory]] · [[wiki/KV Cache|KV Cache]] - **raises:** [[wiki/Continuous Batching|Continuous Batching]] batch size · [[wiki/Utilization|Utilization]] - **supported by:** [[wiki/Accelerator|Accelerator]] · [[wiki/Inference Accelerator|Inference Accelerator]] - **implemented in:** [[wiki/vLLM|vLLM]] · [[wiki/SGLang|SGLang]] - **part of:** [[wiki/Intelligence Delivery|Intelligence Delivery]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** the proportional relationship between bytes per weight and decode step rate; memory savings enabling single-device residency and larger batches; the format families including sixteen-bit float, eight-bit float and integer, and four-bit block-scaled schemes; post-training against quantization-aware approaches; per-tensor, per-channel, and per-block scaling granularity; outlier handling as a key low-bit technique. - **Established:** current rack-scale systems quoting peak arithmetic figures at four-bit precision, with published figures on the order of exaFLOPS per rack. - **Established:** routine production serving at eight-bit precision with small general-task quality loss. - **Unresolved:** four-bit accuracy across task types; results are mixed and task-specific evaluation is required rather than inferred from aggregate benchmarks. ## Sources - [Helios Is AMD's First AI System To Rival Nvidia Vera Rubin](https://www.cnbc.com/) — CNBC, October 2026; peak system specification quoted at four-bit precision for a current rack-scale system. - Quantization research literature and open inference engine documentation, as current October 2026.