# NVIDIA H200 The H200 is the same compute silicon as its predecessor with a larger, faster memory subsystem. That makes it an unusually clean experiment, because one variable changed. The result was a substantial improvement in inference throughput on large models with no change in arithmetic capability at all, which is the most direct evidence available for a claim that runs through this whole collection. > [!note] The one-sentence version > Same arithmetic, more memory bandwidth, considerably more tokens per second; the bottleneck was never the arithmetic. ## What the refresh demonstrates Decode reads the entire weight set out of memory at every generation step and performs comparatively little arithmetic on each value, so tokens per second tracks memory bandwidth rather than arithmetic throughput; see [[wiki/Prefill and Decode|Prefill and Decode]] and [[wiki/Memory Wall|Memory Wall]]. Raising capacity and bandwidth therefore raises serving performance directly, and raising capacity does something further: a model that previously had to be split across devices to fit may now reside on fewer, which removes communication from the critical path entirely and compounds the gain; see [[wiki/Scale-Up and Scale-Out|Scale-Up and Scale-Out]]. Larger memory also enlarges the cache pool available for concurrent sequences, which raises the batch size an engine can admit, which raises throughput again; see [[wiki/KV Cache|KV Cache]] and [[wiki/Continuous Batching|Continuous Batching]]. Three distinct mechanisms, one change. ## Why it matters strategically This part is the reason memory supply is treated as seriously as logic supply in this industry. If capability per device can be advanced substantially by changing only the memory, then the three firms producing stacked memory hold a position comparable to the foundries, and memory generations effectively set accelerator generations; see [[wiki/High-Bandwidth Memory|High-Bandwidth Memory]] and [[wiki/Supply Chain Chokepoint|Supply Chain Chokepoint]]. It is also why memory-focused refreshes appear mid-generation across the industry rather than only at architectural transitions. ## Relationships - **variant of:** [[wiki/NVIDIA H100|NVIDIA H100]] - **mounted on:** [[wiki/NVIDIA HGX|NVIDIA HGX]] - **demonstrates:** [[wiki/Memory Wall|Memory Wall]] · [[wiki/Prefill and Decode|Prefill and Decode]] - **depends on:** [[wiki/High-Bandwidth Memory|High-Bandwidth Memory]] · [[wiki/Advanced Packaging|Advanced Packaging]] - **succeeded by:** [[wiki/NVIDIA B200|NVIDIA B200]] · [[wiki/NVIDIA B300|NVIDIA B300]] - **serves:** [[wiki/Inference Serving|Inference Serving]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** the part as a memory-focused refresh of the preceding compute silicon, with increased stacked memory capacity and bandwidth; the memory-bandwidth bound on decode; larger capacity permitting single-device residency for models that previously required splitting, and enlarging the cache pool to raise admissible batch size. - **Established:** published system building blocks pairing this part with dual host processors and two terabytes of system memory. - **Strongly indicated:** memory generation rather than logic generation setting the pace of inference capability improvement; the mechanism is established and a single aggregate figure is not asserted. ## Sources - Public accelerator specification documentation, as of October 2026.