The Life of an Inference Request

A 10,000-foot map — for engineers who already think in packets, buffers, and tail latency, not tokens.

I spent the last two years automating networks and pointing agentic AI at that automation — then realized the interesting box wasn’t the one sending packets, it was the one answering prompts.

Training a frontier model can cost on the order of $100M+, but that bill lands rarely. Serving those models (AI Inferencing) to millions of users, continuously, is where roughly 85% of LLM operating cost actually lives. Inference is the long game: latency, VRAM, KV cache, and the network path in between.

The AI Inferencing Pipeline is a complex system that involves multiple components working together to serve AI models to users.

AI Inferencing Pipeline Overview

In Client request arrives Load Weights land in VRAM Prefill Prompt processed in parallel Decode Tokens generated one at a time Response Tokens stream back to client Edge & Security Stack WAF · LB · AI guardrails DPU / SmartNIC offload → network hop interconnect VRAM — shared resource under every stage model weights + CUDA / runtime overhead + KV cache → read the VRAM article first — it's the resource everything else competes for
Click any stage to open its article. VRAM sits underneath the compute stages because nearly every one draws from the same fixed pool; the Edge & Security Stack sits above the ingress hop for the same reason, on the network side.
Stage 1
In

Request is tokenized and admitted — the edge and admission layers, before the engine ever sees it.

Stage 2
Load

Weights cross the PCIe/NVLink bus from disk into VRAM — the resource everything downstream competes for.

Stage 3
Prefill

The whole prompt is processed in one parallel pass. Compute-bound. This is what sets TTFT.

Stage 4
Decode

One token at a time, reading the growing KV cache from VRAM at every step. Memory-bandwidth-bound.

Stage 5
Response

Tokens stream back over the network, same as any other payload leaving the data center.

How this guide is organized

Nine articles, color-coded by phase in the sidebar:

  • Foundation + Map — VRAM (the resource) and the full request lifecycle (the map), so everything after has a place to attach to.
  • Setup + Mechanics — the decisions made before and during a forward pass: quantization, model loading, prefill/decode, and the KV cache.
  • Bridge + Operate + Optimize — zooming back out to your home turf: the network's role, observability, and performance tuning. Optional deep-dives (7a–7c) cover the edge/security stack, interconnect fabric, and KV-cache-aware routing; Article 10 covers testing & benchmarking.

Read it in order the first time through — each article assumes the vocabulary from the one before it.

Start with Article 1: What Is VRAM? →