# Inference Request Routing A facility holding several models on several kinds of accelerator must decide, for each arriving request, which model will answer it and which hardware will run it. Routing is that decision, made in milliseconds, against a changing picture of load, cache state, latency commitments, and available capacity. It is the layer at which a mixed fleet becomes a single usable service. > [!note] The one-sentence version > Decide per request which model and which silicon should answer, and a warehouse of mismatched hardware behaves like one coherent service. ## Why it exists Two kinds of diversity make routing necessary. **Model diversity**: a facility serves models of different sizes and specializations, and many requests are answered perfectly well by a smaller, faster, cheaper model, while a minority genuinely require the largest one. **Hardware diversity**: a fleet contains several accelerator generations and types with different memory capacities, bandwidths, and arithmetic profiles, and different workload shapes suit different ones; see [[wiki/Inference Accelerator|Inference Accelerator]]. Without routing, both forms of diversity are liabilities: operators must either run everything on the largest model and the newest hardware, which is wasteful, or expose the complexity to the application, which pushes infrastructure decisions onto people who should not be making them. Routing turns both into assets. ## What the decision considers - **Request shape.** Prompt length, expected output length, and whether the work is prefill-heavy or decode-heavy, which indicates whether a compute-dense or bandwidth-rich part suits it; see [[wiki/Prefill and Decode|Prefill and Decode]]. - **Required capability.** Whether the task needs the strongest available model, which can be decided by explicit tier selection, by a classifier, or by attempting a smaller model and escalating. - **Cache locality.** Whether a worker already holds the relevant prefix, since routing a conversation back to the worker holding its history avoids recomputing it; see [[wiki/KV Cache|KV Cache]]. Cache-aware routing is one of the highest-value optimizations in the layer. - **Current load and queue depth,** so that the choice reflects the fleet as it is rather than as it was configured. - **Latency commitment** attached to the tenant or request class. - **Resource efficiency,** meaning the throughput per unit of accelerator time and energy a given placement achieves. - **Region and locality,** since the round trip to the user is part of the experience and capacity may be pooled across sites; see [[wiki/Fiber Route Diversity|Fiber Route Diversity]]. ## Why this is where accelerator diversity pays Buying from more than one accelerator vendor is attractive for supply resilience and price, and it is only practical if something makes the resulting mixed fleet usable. Routing is that something: it absorbs the differences and presents one interface, so a tenant asks for a capability rather than for a device. The strategic consequence is large. Diversity of supply becomes an operational detail rather than an application burden, which is precisely the condition under which a buyer can insist on a second source, and under which a country can assemble compute capacity from more than one supply chain; see [[wiki/Supply Chain Chokepoint|Supply Chain Chokepoint]] and [[wiki/National Compute Capacity|National Compute Capacity]]. ## Relationships - **selects among:** [[wiki/Accelerator|Accelerator]] · [[wiki/Inference Accelerator|Inference Accelerator]] - **informed by:** [[wiki/Prefill and Decode|Prefill and Decode]] · [[wiki/KV Cache|KV Cache]] - **part of:** [[wiki/Inference Serving|Inference Serving]] · [[wiki/Intelligence Delivery|Intelligence Delivery]] - **presented through:** [[wiki/Inference Endpoint|Inference Endpoint]] - **strategic dimension:** [[wiki/Supply Chain Chokepoint|Supply Chain Chokepoint]] · [[wiki/National Compute Capacity|National Compute Capacity]] - **source articles:** [[articles/Vertically Integrating an AI Superpower from AI Factory to Citizen|Vertically Integrating an AI Superpower from AI Factory to Citizen]] · [[articles/The Gigawatt Is the New Measure of War|The Gigawatt Is the New Measure of War]] - **collection route:** [[collections/American Infrastructure|American Infrastructure]] · [[collections/American|American Collection]] ## Evidence posture - **Established:** routing decisions considering request shape, required capability, cache locality, current load, latency commitments, resource efficiency, and region; cache-aware routing avoiding prefix recomputation; escalation from smaller to larger models as a capability-selection method. - **Established (datestamped):** as of October 2026, managed inference services in the industry provide automated model selection, hardware selection, and multi-region load balancing as standard features. - **Strongly indicated:** routing as the enabling condition for economically viable multi-vendor accelerator fleets; the mechanism is clear and published operator comparisons are limited. ## Sources - Managed inference service documentation and production serving practice, as current October 2026.