Prefill and Decode

Weights are resident. Every request now runs the two-phase cycle that uses them.

Article 4 left the model sitting in VRAM. Prefill and Decode do not reload it — they reuse those weights on every request. The cycle has two stages with opposite resource profiles, the split Article 2 flagged as the engine's core scheduling problem.

Prefill (compute-bound)

The model processes the entire prompt across all transformer layers in one parallel forward pass — every prompt token at once, heavy GPU matrix multiplication. That work builds the prompt's internal representation and produces the first output token. Prefill ends when that token is ready to stream. What the user experiences: the initial lag before anything appears (TTFT).

Decode (memory-bound)

After the first token, the model generates the rest one token at a time. Each step still needs the full weight matrices, plus a growing record of past context (the KV cache) pulled from VRAM — but it only computes new work for the single token being born. Little new math relative to bytes moved: the bottleneck is memory bandwidth, not raw FLOPs. What the user experiences: tokens streaming one by one; TPOT is the gap between them.

Same resident weights. Opposite bottlenecks. That is the split Article 2 called out — and why a naive "run the model" loop cannot schedule mixed prefill and decode traffic well.

What prefill writes for decode: Q, K, V

Attention is how tokens look at each other. Every token builds three vectors:

• Query (Q) — "what previous context do I need?"
• Key (K) — "here's what I contain" (the nametag others match against)
• Value (V) — "here's the payload to mix in if you matched me" (not relevance — relevance is the Q·K score)

Prefill computes K and V for every prompt token and files them so decode can consult that record instead of re-reading the whole prompt from scratch. Without that cache, generating token #1,000 would recompute K and V for all 999 predecessors every step.

Walking through an example: "Can you tell me a joke?"

Phase 1 — Prefill (indexing the prompt)

1. Tokenization — ["Can", " you", " tell", " me", " a", " joke", "?"]
2. Parallel forward pass — all 7 tokens processed simultaneously.
3. Q, K, V per token — e.g. "joke": Key advertises "humor / entertainment noun"; Value carries the semantic payload (setups, punchlines); its Query looks at neighbors the same way every other token does.

How tokens interact

1. Match Query to Key — "tell" sends its Query and checks it against every other token's Key. The highest dot-product match: "tell" ↔ "joke" (a strong semantic relationship). That Q·K score is relevance.
2. Softmax scoring — raw scores become attention weights that sum to 100%. Example: 70% to "joke," 20% to "you," 10% spread across the rest.
3. Mix Values — "tell" blends 70% of "joke"'s Value (payload) with 20% of "you"'s Value into its own updated representation.

When prefill finishes: each prompt token's Keys and Values are filed in the KV cache — they describe content, and content does not change once computed. Queries were only needed to build those representations; they are not what gets cached. The first output token is ready to stream — TTFT ends here. For our joke request, that might be "Why".

Phase 2 — Decode (answering, one card at a time)

The prompt's K/V stay in the cabinet. For output token 1 ("Why"), the model computes Q_Why, K_Why, V_Why; Q_Why looks back at the cached Keys of the whole prompt, and K_Why, V_Why get appended. For token 2 ("did"), Q_did looks at the prompt's Keys plus K_Why — and so on, toward something like "Why did the …". Each step: new Q/K/V for one token, consult the growing cabinet, append one card.