Loading the Model
The quantized file you sized in Article 3 makes the physical trip into VRAM.
Article 3 chose how big the weights slice of your VRAM budget would be. This article is the next box on the Article 2 lifecycle: Load — getting those weights resident where the GPU can use them.
Important timing note: Load is usually a startup / cold-start job, not something that runs on every prompt. Once weights live in VRAM, Prefill and Decode (Articles 5–6) reuse them for each request. The LM Studio progress bar or Ollama pull-and-run sequence is this residency step: a transfer across the memory hierarchy (SSD → system RAM → GPU VRAM), plus allocation and layout decisions by the inference engine.
Quantization still pays here. A ~35 GB Q4 file (the 70B example from Article 3) crosses these buses once; a ~140 GB FP16 twin takes about four times as long and needs four times the VRAM on arrival.
What is a model file?
A model file (or artifact) is the stored representation of a trained model: the finalized weights learned during training, plus structural metadata describing how they fit together. Modern LLM files have two key components:
- Weights files (
.safetensors,.gguf,.bin) — multi-gigabyte matrices of floating-point or quantized integer values (the formats from Article 3). - Architecture config (
config.json, or metadata inside GGUF) — attention heads, layer count, hidden sizes, vocabulary limits. The engine reads this first to know how much memory to reserve and how to lay the tensors out.
The path: two hops into VRAM
The engine can't "run" the file like an executable. It treats the file as a blueprint and a raw data source, then moves weight bytes across two distinct hops — each with its own bus and protocol stack:
Phase 1: SSD → System RAM
- Physical interface — PCIe Gen 4/5, typically x4 lanes to the NVMe drive.
- Protocol — NVMe, with commands encapsulated into PCIe Transaction Layer Packets.
- Transfer — DMA: the NVMe controller writes weight bytes into host RAM without burning CPU cores on the copy.
- Bandwidth / latency — Gen 4 x4 ≈ 7.8 GB/s, Gen 5 x4 ≈ 15.7 GB/s; latency ≈ 10–100 µs.
Phase 2: System RAM → GPU VRAM
- Physical interface — PCIe x16, or NVIDIA NVLink / NVSwitch on GPU-to-GPU or host-to-GPU topologies.
- Protocol — PCIe Memory Write TLPs, or CUDA driver calls (
cudaMemcpyHostToDevice,cudaMallocManaged). - Transfer — host RAM is often "pinned" (page-locked) so it can't be swapped; the GPU's DMA engine reads from pinned RAM into HBM/GDDR without CPU intervention.
- Bandwidth / latency — PCIe Gen4 x16 ≈ 31.5 GB/s, Gen5 x16 ≈ 63 GB/s; NVLink C2C (Grace Hopper) up to ~900 GB/s bidirectional; latency ≈ 1–5 µs.
Same mental model as any storage-to-compute pipeline you've tuned: find the slowest hop, then decide whether to avoid crossing it (lazy loading) or speed it up (Gen5, NVLink). Decode is memory-bandwidth bound inside the GPU — but none of that matters if weights are still stranded on SSD or thrashing through host RAM on every layer.
What the engine does during Load
On top of the buses, the inference engine owns three jobs (the "Loading and memory layout" work from Article 2):
- Read the blueprint — parse architecture config / GGUF metadata to know layer shapes and how much VRAM (and host RAM) to reserve, including room for KV cache later.
- Transfer and place weights — move tensors along the path above into the planned layout (single GPU, multi-GPU shard, or partial offload).
- Ready for inference — kernels and memory layout are set so the next request can Prefill without another full model copy from disk.
Bulk load vs. lazy load (mmap)
How aggressively you walk that path is a cold-start vs. residency trade-off:
| Without lazy loading (bulk) | With lazy loading (mmap) | |
|---|---|---|
| Process | Allocates and copies immediately via a blocking read() / explicit H2D copy | Maps disk offsets into virtual memory; pages fault in on demand |
| Startup | Wait for gigabytes to cross SSD → RAM → (often) VRAM before the first token | Near-instant process start — weight bytes arrive as layers are first touched |
| Resource impact | Reserves host/device memory for the whole working set up front | OS page cache can evict cold pages under pressure and reuse warm pages across runs |
Bulk loading maximizes steady-state speed once everything is in HBM. mmap optimizes cold start and host RAM pressure — at the risk of page faults during early generation if layers aren't warm yet.
When the model doesn't fit one GPU
If the weights slice (plus overhead and a usable KV cache) exceeds a single device — the failure mode from Article 1 — Load has to place tensors somewhere else:
- Layer offloading — keep some layers in system RAM (or even on disk) and move them into VRAM when needed. The model runs; decode slows whenever a layer crosses PCIe again.
- Multi-GPU parallelism — split layers or tensors across devices (tensor / pipeline parallelism from Article 2), using NVLink or PCIe between GPUs.
Both are still "Load" decisions: where each weight lives when the engine claims the model is ready.
Quantized weights on this path
Load does not undo Article 3. For typical weight-only formats (GGUF, many AWQ/GPTQ serves), the engine places compressed weights into VRAM along with their scale factors. The on-the-fly expand to FP16/BF16 still happens later, in on-chip SRAM/registers, right before the matrix multiply — not as a "inflate the whole model during Load" step. Some stacks can also run native low-precision matmuls; either way, the bytes that crossed PCIe and now occupy the weights slice stay close to the quantized size you chose.
Load is residency. The model file leaves disk, crosses two buses, and ends up as a planned layout in VRAM (and maybe host RAM). Quantization decided how large that shipment was; Load decides how and where it lands. After that, every request reuses the same resident weights — Prefill and Decode don't reload the model.
Weights are sitting in VRAM. Next, Article 5 covers the first thing that happens to them on every request — the two-phase prefill/decode cycle.