LLM Request Lifecycle and Inference Engines

A detailed end-to-end map of a request — now that Article 1 gave you VRAM as the resource everything below competes for.

Article 1 gave you the pool: weights + overhead + KV cache, all fighting for VRAM. A request lifecycle diagram looks simple — it is not. The piece of software that actually manages that competition — the inference engine — doesn't even get its own box in most diagrams. It's hiding across several stages: it owns the amber core, and it touches the layers above and below without owning them. This article pulls that engine into the open.

What the engine actually does

Walk the amber band left to right. Every inference engine solves the same handful of problems — and almost all of them are ways of managing the VRAM budget from Article 1.

1. Loading and memory layout

Before any request arrives, the engine gets model weights into VRAM and lays out memory: sharding across GPUs when a model doesn't fit on one device (tensor or pipeline parallelism — wire-level in Article 7b), and reserving the right split between weights and the KV cache. Get this wrong and you either crash on the first large request or waste GPU memory you can't get back. How big those weights are is the precision choice in Article 3; the load hop itself is Article 4.

2. Prefill and decode

Once a request is admitted, generation has two phases with opposite resource profiles — prefill is compute-bound, decode is memory-bandwidth-bound. Two phases, two completely different bottlenecks, running back-to-back for every request. Multiply that by hundreds of concurrent users, each at a different point in their own prefill/decode cycle, and you have a scheduling problem that a plain "load model, run forward pass" script has no chance of handling efficiently. That scheduling problem is the inference engine's entire reason for being — and the focus of Article 5.

3. KV cache management

The KV cache grows with every token generated, for every active request — the expanding amber slice of VRAM from Article 1. Engines like vLLM introduced paged attention — treating the cache like virtual memory pages instead of one giant contiguous block — so memory can be allocated and freed in small chunks instead of being wasted on worst-case reservations. Article 6 is where that slice gets its own deep dive.

4. Continuous batching

A cross-cutting multiplier on Prefill and Decode: instead of waiting for a fixed batch of requests to all finish before starting the next batch (leaving the GPU idle while it waits on the slowest request), the engine continuously slots new requests in and removes finished ones on every iteration. This alone is often the single biggest throughput improvement over a naive serving loop.

5. Speculative decoding and other latency tricks

Optional optimizations on top of the core loop: using a small draft model to guess several tokens ahead, then having the big model verify them in parallel — cutting the number of expensive full forward passes needed per output token.

Names you'll hear

Once those jobs are clear, the product names make more sense. The rest of this article focuses on production engines; local ones solve the same problems at smaller scale.

Production & high-throughput engines

• vLLM — uses PagedAttention to manage memory efficiently and boost multi-user serving speeds.
• TGI (Text Generation Inference) — built by Hugging Face for secure, high-volume production deployments.
• TensorRT-LLM — NVIDIA's toolkit with custom kernels and operator fusion for enterprise GPUs.
• SGLang — optimized for fast multi-turn generation and complex agentic workflows.

Local & consumer hardware engines

• llama.cpp — pure C/C++ engine designed to run models on everyday hardware and Apple Silicon with low memory footprints.
• Ollama — a user-friendly wrapper around llama.cpp that simplifies downloading and running local models.
• MLX — tailored specifically for Apple Silicon hardware acceleration.
• ExLlamaV2 — fast loader for high-speed generation using custom quantization on consumer GPUs.

Where the engine sits in the larger path

The lifecycle diagram above was time — left to right as a request moves. Zoom out and the same system is a stack of layers. Queue maps to request management; Stream maps to the output layer. Layers the lifecycle hid — accelerators, orchestration, observability — show up clearly here. The engine still owns the amber core; it touches the layers above and below without owning them. "Serving" is the surface the engine exposes once it's confident it can handle real traffic reliably.

1. Entry / Edge — where the Client lands: API gateway (auth, rate limiting, request routing). Load balancing here is often request-aware, not round-robin, since prefill-heavy vs. decode-heavy requests cost very differently. The inspection / guardrail perimeter is the optional deep-dive in Article 7a; the network overview is Article 7.
2. Request management — the Queue from the lifecycle diagram: admission, priority, queuing. The engine doesn't decide who gets served first; it executes what's handed to it. Cache-aware placement of which engine replica gets the request is Article 7c.
3. The inference engine — Load · Prefill · Decode · KV, plus continuous batching, sitting on one or more accelerators (the amber band above).
4. Accelerator & hardware topology — what the engine runs on. The engine issues the parallelism strategy (tensor/pipeline sharding) but doesn't manage the physical interconnect or memory hierarchy itself — that fabric story is Article 7b.
5. Orchestration — decides how many copies of the engine exist and where. The engine has no concept of "spin up another replica."
6. Output layer — Stream in the lifecycle diagram: takes the engine's raw tokens and handles streaming plus safety filtering. Guardrails also apply to input tokens.
7. Observability (wraps everything above) — TTFT/TPOT, tokens/sec, GPU utilization, queue depth, and model-quality monitoring (drift, output sampling, guardrail trigger rates). Mapped metric-by-metric in Article 8. The engine doesn't monitor or version itself.