Understanding the KV Cache
The other major VRAM consumer, and the primary physical limit on context window size.
In Article 5, prefill filled the cabinet and decode only appended one new card per step. This article is about that cabinet itself: how big it gets in VRAM, why that size is the real ceiling on context and concurrency, and — same structure, longer horizon — how platforms reuse a matching prefix across requests so the second call is cheaper and faster.
Quick refresh: Q, K, V
- Token — a chunk of text (a word or word fragment) the model processes.
- Embedding — a vector (thousands of numbers) representing a token's core semantic meaning.
- Query (Q) — what the current token is looking for in prior context.
- Key (K) — how a prior token advertises itself so Queries can find it.
- Value (V) — the content a prior token contributes once selected.
Think of a sentence as a group project where every word is a person in the room. The Key is like a nametag: "cat" wears a tag that says "I am a furry four-legged pet." The Value is the actual knowledge in their backpack — the details they bring to the sentence. When a new word like "sat" enters, it holds up a Query: "I need a surface or an animal." The model compares that Query against everyone's Key nametag; "cat" matches well, "the" barely matches. Because "cat" matched, the model reaches into "cat"'s backpack and takes its Value to help write the next part of the sentence.
Splitting a word into two roles lets it advertise itself differently from how it contributes. A
token can advertise with one set of features (the Key) so other tokens can find it, while holding
different detailed content (the Value) to share once selected. Both are produced by static weight
matrices (W_K, W_V) learned during pre-training — transforming one
embedding through them is expensive per layer, across dozens of layers. That cost is why you never
want to redo it for tokens you already processed.
Within one generation: why the cache exists
Generation is sequential: producing token #2 requires looking back at token #1; token #3 requires looking back at #1 and #2; and so on. Without caching, the model recomputes K and V for every preceding token, every step.
Generating "The cat sat" (assume 1 second per token's K/V, across all layers):
| Step | Without KV cache | With KV cache |
|---|---|---|
| 1 ("The") | 1s | 1s (calculate, then save) |
| 2 ("cat") | recompute "The" (1s) + "cat" (1s) = 2s | reuse cached "The" (0s) + "cat" (1s) = 1s |
| 3 ("sat") | recompute "The" + "cat" (2s) + "sat" (1s) = 3s | reuse cached (0s) + "sat" (1s) = 1s |
| Total | 6s | 3s |
By token 1,000, the uncached model would recompute the Key for "The" a thousand times — quadratic
growth in compute time. The KV cache applies memoization: once K and V are calculated for a token,
they're saved into GPU memory and reused. Reading a vector from VRAM is far cheaper than
recomputing it through W_K/W_V. That is reuse within one
generation — the cabinet from Article 5.
The KV cache and the context window
Those saved vectors are not free. Every token added to the context requires its K and V to be stored in VRAM across all layers. For a standard 70B-class model, that is roughly 0.5 MB per token. Memory grows linearly with context: a 4,000-token context needs about 2 GB of VRAM per user; a 100,000-token context needs about 50 GB.
Two different VRAM limits, often blurred together:
- Capacity — how large a context (or how many concurrent conversations) you can hold. Even if a model's architecture supports a 1-million-token window, serving it still means allocating gigabytes purely for that request's KV cache.
- Bandwidth — how fast the GPU can read that growing cache on every decode step. Decode is memory-bound because each new token must scan the cabinet.
GPU compute decides how fast a model thinks. Capacity and bandwidth decide how large a context you can actually serve.
Common misconception
Misconception: large context windows are constrained because the attention math becomes too difficult to compute. Reality: compute is rarely the bottleneck for long context during inference. You run out of VRAM first (capacity), and even before that, every decode step slows as the GPU re-reads a larger and larger cache (bandwidth).
Because the cache grows with the context window, modern infrastructure leans on techniques like vLLM's PagedAttention, sliding window attention, or KV cache quantization (compressing 16-bit keys/values to 4-bit) to reduce memory usage — allowing platforms to support 100k+ token context windows without running out of GPU memory. More on these levers in Article 9.
Across requests: prompt caching
The same K/V entries can be reused across requests when a later prompt shares a prefix with an earlier one — that is why sending the same system prompt twice is often cheaper and faster. Same cabinet, longer horizon: within a generation you avoid recomputing past tokens; across requests you avoid recomputing a matching prefix.
One catch: later positions' Keys and Values are computed from hidden states that already mixed in earlier tokens (layer by layer). Change text early in a prompt, and every downstream cache entry is invalid — the prefix no longer matches.
Uncached structure (variable data first): "User query: [A] | System
Instructions: [B]". Changing [A] at the start invalidates everything after it —
0% cache hit. Cached structure (static data first): "System Instructions:
[B] | User query: [A]". The static prefix [B] matches the saved cache every
time; only [A] needs new computation — 80%+ cache hit.
Why this matters for how you build
- Tail-load variable content — put static elements (system instructions, tool definitions, RAG documents) at the top of prompt templates, and dynamic, user-specific input at the very end.
- Optimize agent loops — in multi-turn agentic applications, a consistent prefix lets long execution loops run substantially faster and cheaper.
That's the compute side fully mapped: VRAM, loading, prefill/decode, KV cache. Next, Article 7 zooms back out to home turf — what the network is actually doing while all of that happens.