An LLM request router sits in front of several replicas of the same model and chooses which replica should receive each prompt. Cache-aware routing makes that placement decision before execution by considering reusable prompt state at each replica, not only how busy it looks. The surprising part is that replicas stop being interchangeable after they begin serving traffic: one may already hold reusable computation for the prompt, while another may have a shorter queue.
Prefix-aware routing became a managed endpoint option in September 2026, a sign that cache locality has moved from an engine detail into a deployment decision.
So where should the next request go: to the warm replica or the idle one? The compact answer is to send it where the saved prompt computation is worth more than the extra work already waiting there. A useful router therefore needs two views at once: what can be reused and what must still be processed.
To see why, start with the work a cached prefix actually removes. Then we will keep one support prompt fixed while four routing policies make different placement decisions. That comparison will expose the useful rule, its failure modes, and the state a production router has to keep current.
What a warm prefix actually saves
Before routing makes sense, we need one piece of transformer serving machinery.
An incoming prompt is a sequence of tokens. During prefill, the model processes those input tokens through every transformer layer. In each attention layer it derives a key vector and a value vector for every position. Those vectors are the processed state that later tokens attend to. The first generated token can appear only after this prompt work has finished.
During decode, the model generates one new token, appends its new key and value state, and repeats. Decode can reuse the state for all earlier positions instead of rebuilding it. The stored collection is called the KV cache.
Now consider a support assistant whose prompt begins with 3,200 tokens of policies, tool descriptions, and response rules. Only the short customer question at the end changes. If the engine has already processed the same beginning, it can reuse the matching KV state and prefill only the new suffix.
Prefix caching does not mean that the model remembers a similar idea. The match is structural. Serving engines divide the prompt into token blocks, identify a block using its tokens and the blocks before it, and reuse only an unbroken matching prefix. If block three changes, later blocks no longer describe the same model state, even if much of their text looks familiar.
That distinction gives prefix caching an honest boundary. It skips model computation already performed for the input prefix. It does not skip the generation of new output tokens, because those tokens do not exist yet. A workload with long repeated inputs and short answers can gain a lot; a workload with unrelated prompts or very long answers may gain little.
Within one replica, the rule is simple: reuse every valid prefix block that is still resident, prefill the remaining input, then decode normally.
