What Is VRAM and Its Role in Inferencing?

The resource every later concept — quantization, model loading, the KV cache — competes for.

Every LLM you run — a 7B model on a laptop or a 70B model behind an API — lives or dies by one resource: VRAM. It's the single biggest constraint in AI inferencing, and understanding exactly what fills it up is the difference between a model that runs smoothly and one that silently grinds to a crawl.

What VRAM actually is

VRAM is the high-speed memory that sits directly on the GPU. Unlike system RAM, it's built for the massive, parallel, rapid-fire data access that GPU computation demands — exactly what LLM inference needs at every step of generating a response.

For inference to run fast, everything the model needs right now has to physically live in VRAM. If it doesn't, the GPU has to fetch it from somewhere much slower, and that's where performance falls apart.

At a high level, VRAM usage breaks down into three pieces:

• Model weights — the model's parameters. Fixed in size once you've picked a model and precision (the blue slice).
• CUDA / runtime overhead — baseline memory the GPU driver and runtime need just to operate: kernels, buffers, framework overhead (the gray slice).
• KV cache — the one piece that isn't fixed. It grows as context and concurrent conversations grow (the amber slice).

The KV cache: the model's working memory

The KV cache is the model's short-term working memory. As it processes and generates tokens, it stores Key (K) and Value (V) vectors for every token that has come before, so it doesn't redo that work from scratch on every next token. That's what makes fast, fluent generation possible — and why the amber slice in the diagram keeps growing.

Concrete example: running Llama 3.1 8B with a 32K-token context window can add 4 GB or more of KV cache on top of the base model's own footprint. On a lot of consumer and prosumer GPUs, that's a meaningful chunk of total available VRAM — gone before generation even starts.

What happens when you run out

If weights, overhead, and KV cache together exceed available VRAM, the system doesn't just stop — it starts offloading the overflow to system RAM. That keeps things technically running, but every time the GPU needs data that's been pushed out, it has to reach across a far slower bus to get it. Generation speed craters. This is the core tension in local and self-hosted inference: you're constantly balancing model capability, context length, and the VRAM you actually have — and the KV cache is the variable most likely to tip that balance without warning.

Levers you'll pull later

You don't need the how-to yet. Three levers show up across the rest of the portal — each shrinks or manages a slice of the same formula:

• Model quantization — shrinks the weights slice. Formats like Q4_K_M cut footprint so more of the GPU is left for KV. Details in Article 3; then Article 4 loads those smaller weights into VRAM.
• KV cache quantization — shrinks the growing amber slice (e.g. F16 → Q8/Q4) so the same context window fits. Deep dive with the rest of the cache story in Article 6.
• Flash Attention (and friends) — cut how much intermediate memory attention needs as context grows. You'll meet them again when we talk prefill/decode and optimization (Article 5, Article 9).