# Model Router A model router decides which model answers each request. The premise is empirical and well supported: the distribution of real requests is heavily weighted toward work that a smaller, faster, cheaper model handles perfectly well, with a minority genuinely requiring the strongest model available. Routing captures that distribution instead of paying frontier prices for every answer. > [!note] The one-sentence version > Send the easy questions to the small model and the hard ones to the large model, and the same quality costs a fraction as much. ## The methods **Explicit tier selection.** The application names a capability tier and the service resolves it to specific weights. Simple, predictable, and dependent on the application developer judging correctly. **Classifier routing.** A small, fast model or a learned classifier examines the request and predicts which tier will satisfy it. Cheap to run and imperfect; the design question is how it errs, since routing a hard question to a small model produces a poor answer while the reverse merely costs money. **Cascade.** The small model answers first, a verifier or confidence signal evaluates the result, and the request escalates to a larger model when the answer is judged inadequate. This bounds the quality risk and pays a latency penalty on escalated requests, which is why it suits asynchronous work better than conversation. **Speculative decoding** is the same idea inside one request rather than across requests: a small model proposes tokens and the large model verifies them in a batch, so the expensive model does less sequential work for identical output; see [[wiki/Prefill and Decode|Prefill and Decode]]. **Policy routing.** Direction by tenant, jurisdiction, data sensitivity, or contractual commitment rather than by difficulty, which is how residency and privacy requirements are enforced at the routing layer; see [[wiki/Private Model Hosting|Private Model Hosting]]. ## What routing requires to work well Cache locality awareness, so a conversation returns to the worker already holding its history rather than recomputing the prefix; see [[wiki/KV Cache|KV Cache]]. Current load and queue depth, so the choice reflects the fleet as it is. An honest evaluation of routing quality on the actual task distribution, because aggregate benchmarks conceal the cases where a small model fails. And observability, since a router silently degrading answer quality to save money is the characteristic failure of this layer and is invisible without measurement; see [[wiki/Observability|Observability]]. ## Why it belongs in an infrastructure collection Routing is where model economics meets facility economics. A router that moves half of a service's traffic to a model a fraction of the size reduces the accelerators, the power, and the capital required to serve that traffic proportionally, which is a facilities-scale outcome produced by a software decision; see [[wiki/Capital Intensity|Capital Intensity]] and [[wiki/Utilization|Utilization]]. It is also one of the mechanisms steadily lowering the price of a useful answer, and therefore one of the mechanisms widening the set of American institutions that can afford to use this capability at all; see [[wiki/National Compute Capacity|National Compute Capacity]]. ## Relationships - **part of:** [[wiki/Inference Request Routing|Inference Request Routing]] · [[wiki/Inference Serving|Inference Serving]] - **paired with:** [[wiki/Accelerator Selection|Accelerator Selection]] - **uses:** [[wiki/KV Cache|KV Cache]] · [[wiki/Observability|Observability]] - **enforces:** [[wiki/Private Model Hosting|Private Model Hosting]] policy - **affects:** [[wiki/Capital Intensity|Capital Intensity]] · [[wiki/Utilization|Utilization]] - **broadens:** [[wiki/Intelligence as a Utility|Intelligence as a Utility]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** explicit tier selection, classifier routing, cascade with escalation, and policy routing as deployed methods; speculative decoding as the intra-request analogue; the asymmetry in which under-routing degrades quality while over-routing costs money; cache-locality and load awareness as routing inputs. - **Established (datestamped):** as of October 2026, managed inference services in the industry provide automated model selection as a standard capability. - **Strongly indicated:** the request distribution being heavily weighted toward work that smaller models handle adequately; operator practice and published routing results support it and the proportion is workload-specific. - **Unresolved:** routing quality degradation in production, which requires task-level measurement rather than aggregate benchmarks and is rarely published. ## Sources - Open inference engine documentation and managed inference service practice, as current October 2026.