Pre-Model Loading Steps — Quantization & Precision

The decision that sizes the weights slice of your VRAM budget — made before Load ever runs.

Article 1 framed VRAM as a fixed pool:

Article 2 put Load on the map as getting those weights into VRAM. Before that step spends your budget, you choose how big the weights slice will be — by picking a precision and a quantization format. That choice is what this article is about.

You'll see it on every Ollama model card: variants like Q4_K_M or Q8_0 for the same underlying model. Same architecture, different VRAM bill.

Concrete payoff: a 70B-parameter model at FP16 needs about 2 bytes per parameter — roughly 140 GB of weights alone. The same model at 4-bit (~0.5 bytes/param) lands near 35 GB. One GPU instead of a multi-GPU shard — or a lot more headroom left for the KV cache from Article 1.

Precision vs. quantization

Precision is the data format used to store weights (and the live values that flow through the network during a forward pass — activations): how many bits, and how those bits are structured — FP16, BF16, INT8, INT4, and so on.

Quantization is the process of converting from a higher-precision format to a lower-precision one. It maps a large, continuous set of floating-point values (like FP16) into a smaller, discrete set of lower-bit values (like INT8 or INT4). Converting an FP16 model (2 bytes/param) down to INT4 (0.5 bytes/param) is quantization in action. The goal: shrink the VRAM footprint and speed up execution while minimizing accuracy loss.

Decoding "Q4_K_M"

Q4_K_M tells you both the bit depth and the compression strategy: Q4 = 4-bit quantization, K-quants = the technique, M = the medium-size variant. K-quants (from llama.cpp) process weights in super-blocks rather than uniform blocks, and selectively keep critical layers — attention matrices, feed-forward layers — at higher or mixed precision instead of quantizing everything uniformly. That's why Q4_K_M is widely considered the "sweet spot" for running local LLMs: most of the VRAM win of 4-bit, with quality protected where it matters most.

What you gain — and what you trade

Key advantages

• Massive VRAM & storage reduction — the fixed weights slice shrinks; more of the pool is left for KV cache and concurrent requests.
• Higher throughput, lower latency — smaller weights ease the memory-bandwidth bottleneck during autoregressive generation (the decode phase from Article 2), increasing tokens/sec.
• Lower infrastructure cost — fewer or smaller GPUs directly reduce hosting cost and energy use.

The practical trade-off

• Accuracy degradation — quantization rounds precise weights into a smaller discrete set. Extreme quantization (sub-4-bit) can degrade reasoning or increase hallucination on complex tasks. Formats like Q4_K_M exist to soften that hit.

The twist: why dequantize at all?

Here's the puzzle. If the whole point is to store weights as 4-bit integers, why does the engine convert them back toward FP16/BF16 right before the math?

Because most local and many production setups use weight-only quantization: weights stay compressed in VRAM, but compute still runs in higher precision against the live activation vectors. The win isn't "do all math in 4-bit forever" — it's moving far less data across the VRAM bus on every token. That matches Article 2's point that decode is memory-bandwidth-bound: shaving bytes off every weight fetch pays off even if you expand them again in fast on-chip memory (SRAM / registers — the GPU's tiny working scratchpad next to the compute units).

1. Storage in VRAM (quantized) — weights sit compressed (e.g., 4-bit / Q4_K_M), with small per-block scale factors stored alongside them.
2. Transfer to compute units — as tokens are generated, weight blocks move from VRAM into fast local SRAM/registers.
3. Dequantization — before the matrix multiply, the engine expands 4-bit integers back toward 16-bit floats using those scale factors.
4. Execution — Tensor Cores multiply the restored weights against the (FP16/BF16) activation vectors from the current forward pass.

• Hardware alignment — most Tensor Cores natively compute mixed-precision FP16/BF16; expanding quantized weights lets them meet standard activations.
• Preserving accuracy — activations shift with context; keeping the multiply in higher precision limits compounding rounding error.
• Bandwidth is the prize — VRAM → registers is the slow hop. On-the-fly dequant in registers is comparatively cheap. That's why weight-only quantization still wins for decode even though the math isn't "native INT4 end to end."

Going deeper (optional)

Most readers of this portal pick a pre-quantized file (GGUF on Ollama / llama.cpp, AWQ/GPTQ in many servers). Producing those files — and pushing quantization into the compute path itself — is a separate story:

• PTQ vs. QAT — Post-Training Quantization fits scale factors on a calibration dataset after training; Quantization-Aware Training bakes quantization into fine-tuning. Both cost compute up front so you don't have to invent a quant at Load time.
• Weight+activation quantization — goes further than weight-only: activations are quantized too, so more of the multiply can stay on integer Tensor Cores. Bigger speedups on compute-bound work (prefill), with more risk of precision noise on long-context generation.