Optimizing AI Inferencing

Every prior concept, synthesized into one ratio to optimize against.

Why optimize inference?

Pre-training a frontier model is enormously expensive, but it happens infrequently. Inference — serving millions of users continuously — is widely cited as the larger share of total LLM operational cost over a model's lifetime. Unoptimized inference can also mean multi-second latency per request; optimization work routinely gets that down to milliseconds while cutting data center power draw.

The technical reason optimization is not one generic speedup: decoder-only LLMs have two phases with opposite bottlenecks. Prefill processes the entire input prompt in one parallel pass — every token's K/V can be computed simultaneously, so the GPU's compute cores are fully saturated. Compute-bound. Decode generates one token at a time: each step reads the entire model's weights plus the entire growing KV cache from VRAM just to produce a single new token. The actual math per step is tiny; the GPU spends almost all its time waiting on memory bandwidth. Memory-bandwidth-bound. A technique that helps one phase can be irrelevant — or counterproductive — for the other.

A precision correction on KV caching

What caching eliminates is the redundant re-computation of K/V for prior tokens at every step. It does not eliminate the attention comparison itself — for a new token, the model still has to scan every cached card to decide what to pay attention to. If you're generating word 501, it still scans all 500 cards; that scan is the actual "thinking" step, not busywork. So "caching speeds things up" specifically means it cuts the pointless re-preparation — not the comparison step, which was always going to grow as the conversation gets longer.

Smaller piles mean more piles fit in the leftover space — more concurrent conversations on the same GPU. And because the GPU pays the same slow weight-transfer cost whether it's serving one conversation or fifty at once, batching that fixed cost across more conversations is the direct route to a cheaper answer per word.

Which lever moves which Article 8 metric?

Architectural optimizations (decided at model-selection time)

GQA — Grouped Query Attention

Multi-head attention runs several "readers" (heads) over the same conversation, each noticing a different pattern. Originally, each reader kept its own full stack of index cards — 32 heads meant 32 separate KV cache piles per conversation, even though a lot of that content ends up similar across readers. GQA's fix: group readers (e.g., 8 groups of 4) so each group shares one pile. Every reader still asks its own questions, but they pull from a shared stack. Result: 8 piles instead of 32, per conversation — a large VRAM saving for a small compromise in per-reader distinctness.

MoE — Mixture of Experts

Attacks a different axis: total parameter capacity vs. active compute per token. A model can have hundreds of billions of total parameters but route each token through only a couple of experts, so compute and memory-bandwidth cost per token stays close to a much smaller dense model while capability scales up. This is a win on both prefill and decode, but it doesn't touch KV cache size — MoE and GQA solve orthogonal problems and are often combined. Actionable takeaway: model choice is itself a cost lever — a GQA + MoE architecture (Mixtral, DeepSeek-class models) gives more capability per dollar of active compute.

FlashAttention / FlashDecoding — exact rewrites, not approximations

Inside the GPU there are two storage speeds: the big warehouse (VRAM/HBM — huge, slow to reach) and a tiny workbench right next to the compute (SRAM — extremely fast, very small).

FlashAttention's trick: do the work in small batches, right on the workbench, and never write the giant attention table down at all. A tiny example — 4 words compared against each other — produces a 4×4 grid of scores (16 numbers), no big deal. Scale that to a realistic 8,000-word prompt: 8,000 × 8,000 = 64 million numbers, for one attention head, in one layer — multiply by 32+ heads and 32+ layers and you're talking billions of transient scores. That table is scratch work, used once and discarded — but computing it naively means writing all of it to slow VRAM and reading it back, just to throw it away. FlashAttention instead processes small chunks (e.g., 128×128) entirely on the fast workbench, folding each chunk's contribution into the running answer and discarding it — the full grid never exists as one object, so it never makes that slow round trip.

FlashDecoding solves a different problem. During decode there's no giant grid — generating word 8,001 in an 8,000-word conversation means 1 new word compared against 8,000 existing cards: 8,000 comparisons, not 64 million. The problem here is wasted parallelism, not a wasted grid: a GPU has thousands of cores, but reading through 8,000 cards sequentially with one worker leaves the rest idle. FlashDecoding's fix: split the pile across many workers (say, 8 workers × 1,000 cards each), have them read and compare in parallel, then combine the partial results with a simple weighted sum — which is safe precisely because addition doesn't care what order it happens in. Sequence was already locked into the cards back at prefill time; FlashDecoding only reorganizes who does the reading.

Continuous / in-flight batching

The highest-ROI server config on this list — usually a toggle, not a model change. Static batching wastes GPU cycles waiting for the longest sequence in a batch to finish. Continuous batching evicts a finished sequence and slots in a new one at every iteration, keeping the batch full — a decode-phase utilization fix that is close to a free lunch.

Speculative decoding

A small, fast draft model predicts several tokens ahead; the large target model verifies them all in one parallel forward pass instead of one sequential pass per token, achieving 2-3x speedups without losing output quality. Statistically lossless — unverified guesses are rejected and the first mismatch corrected — with faster token throughput cutting inference cost.

Quantization — largest footprint / throughput lever

Covered in depth in Article 3; here it is as a serving lever. FP16 → FP8 roughly halves weight VRAM footprint with minimal accuracy loss on modern hardware (Hopper+) and often ~2x throughput — the biggest single change to leftover VRAM for concurrent conversations. FP4 (Blackwell) pushes further, but with real accuracy risk that needs validation per use case — benchmark on your own eval set, not just the headline number.

Paged KV cache (and KV quantization)

Article 6 introduced PagedAttention and KV quantization as the levers that attack the cabinet itself. Two different problems, often combined:

• Paging — fragmentation, not footprint. Sequences get variable-length blocks instead of one giant contiguous reservation, so you actually use the VRAM you have instead of losing it to allocation gaps.
• KV quantization (INT8/FP8) — shrinks each index card so more conversations fit in the same leftover space, at some risk to attention quality that you validate like weight quant.

Parallelism, engine sizing, and serving stack

These are the same multi-GPU and disaggregation ideas from Article 7, now as cost/latency levers — they only pay off when the interconnect can keep up with the shard traffic.

• Tensor parallelism — splits individual matrix ops across GPUs in a node; reduces per-token latency but requires a fast interconnect (NVLink) and has diminishing/negative returns if you over-shard.
• Pipeline parallelism — splits layers across nodes; needed when a model doesn't fit in one node, but introduces "bubbles" (idle time waiting on pipeline stages) that continuous batching partially masks.
• Disaggregated prefill/decode — the newest and most aggressive lever: splitting the compute-bound and memory-bound phases onto differently-provisioned nodes, right-sizing hardware for each phase's actual bottleneck — and depending on the fabric to move KV cache between pools in real time.
• Engine sizing (ISL/OSL histograms, LISO/LILO profiling) — TensorRT-LLM-specific: compile for your actual traffic distribution instead of worst-case max context, avoiding paying for VRAM headroom you never use.