KV-Cache-Aware Load Balancing & Request Routing

Going deeper than Article 7's routing example — into how a fleet actually implements cache-aware and disaggregated routing.

Article 7 §4 showed an L7 balancer picking a node from cache utilization and queue depth, not geography. Article 6 explained why a matching prefix makes the second call cheaper. This deep-dive connects those dots: how the balancer knows cache state, how prefix affinity works, and what changes when prefill and decode live on different pools (the disaggregation Article 7 §2 introduced). Coming from 7b, this is the fleet-level placement decision before GPUs start exchanging activations.

1. How the balancer knows cache state

A cache-aware LB is only as good as its signals. Engines expose (or sidecars scrape) approximate prefix-cache hit potential, KV memory occupancy, and waiting/running queue depth — the same family of metrics Article 8 will treat as "buffer occupancy" and "queue depth on a switch." Those are the inputs behind Article 7's Node-C pick (warm cache, short queue) versus Node-B (long queue, wrong prefix). The control plane is usually eventual: a node that just filled its cache may still look attractive for a few hundred milliseconds. Design for stale-but-useful signals, not perfect global knowledge.

2. Prefix affinity and sticky sessions

Common pattern: hash the prompt prefix (or a session / document ID) to a preferred replica so follow-up turns land where the KV blocks already live. That is sticky routing with a purpose — unlike classic session stickiness that only preserves app state.

• Helps — multi-turn chat, RAG over a hot document set, system prompts shared across many users: high prefix hit rate, lower prefill cost, better TTFT.
• Hurts — bursty affinity piles load onto one hot node while others sit idle; a sticky node that is queue-saturated can be slower than a cold node with an empty cache. Good routers blend affinity with a load cap (send elsewhere when queue or cache pressure crosses a threshold).

3. Disaggregated handoff and capacity planning

When prefill and decode are separate pools, the LB (or a scheduler) places the prompt on a compute-heavy prefill node, then ships the KV blocks to a memory-optimized decode node over the cluster fabric. That handoff is an east-west bulk transfer on the critical path to first streamed tokens after prefill — size it like any RDMA/TCP payload proportional to context length × layers × precision, and remember 7b: fabric congestion here looks like TTFT, not like a generic network alert.

Versus a stateless web LB: you are no longer spreading identical backends. You are placing stateful working sets. Capacity plans need headroom for cache locality (extra replicas of popular prefixes), transfer bandwidth for disagg, and explicit escape hatches when affinity and fairness fight.