Testing & Benchmarking AI Inference Pipelines

Starts with a question network testing already knows to ask first: what does the traffic actually look like — and what shape does it take?

Everything in Articles 1-9 describes what to optimize. This is about how to prove it worked — and proving it starts with recognizing that "send some prompts and measure throughput" is the LLM equivalent of testing a network with one traffic pattern and calling it done. Prompts aren't uniform transactions. They're dynamic workload profiles, and the shape of that workload — what Keysight calls the shape of prompts — is what actually determines how the inference stack behaves.

The geometry of prompt shapes

A prompt can be described as a vector across three dominant axes: Compute/Context, Memory, and Latency Sensitivity. What looks like "a simple request" at the application layer is a distinct resource-distribution profile at the infrastructure layer — and that profile is what actually determines GPU utilization, KV-cache growth, network fabric pressure, and where TTFT/TPOT/TTLT land.

• Prefill-dominant (compute/context-heavy) — large context ingestion, long documents, RAG analysis, legal or multi-document review. The GPU spikes hard on ingest, the KV cache expands rapidly, and TTFT becomes a function of context size and parallelization efficiency. Stack concurrency on top and the spike compounds across sessions, quickly exposing GPU-to-memory imbalances.
• Decode-dominant (latency-sensitive) — chat, streaming assistants, code generation: moderate prefill, extended decode. TPOT dominates the user's actual experience and TTLT grows with output length; the system isn't compute-starved, but users still feel degradation if token cadence gets uneven.
• Memory-heavy (KV-cache-driven) — long-running conversations and agent chains shift pressure onto memory instead of compute. KV-cache residency becomes the gating resource, and a GPU can look underutilized while memory is actually saturated.

Two more wrinkles worth knowing: concurrency and burst reshape geometry in real time — a prompt that looks balanced in isolation can distort badly under load, since short prompts under burst become latency-heavy purely from queuing. And RAG / multi-phase prompts don't sit still on the chart at all — retrieval latency, a prefill compute surge, and a decode streaming phase happen in sequence, so the shape morphs across the request's lifecycle rather than holding one geometry throughout.

Why this demands a new benchmarking approach

Real inference traffic is heterogeneous — compute-heavy prefill bursts, memory-intensive long-context sessions, latency-critical interactive exchanges, and multi-phase RAG workflows, all mixed together. Benchmarks built around peak throughput or average latency collapse that multidimensional variability into one number and lose the thing that actually matters: which resource breaks first, under which shape of traffic.

What a shape-aware test plan asks

The proof loop closes when you point familiar network-test methods at the three axes above — not at a single tokens/sec headline:

• Mixed shapes, not one prompt — run prefill-dominant, decode-dominant, and memory-heavy profiles (plus a RAG multi-phase mix) the way you mix traffic classes on a switch.
• Percentiles from Article 8 — watch TTFT for prefill-heavy shapes, TPOT variance for decode-heavy ones, and cache % / queue depth as leading indicators before the user-visible cliff.
• Soak and ramp-hold-drain — fragmentation and cache-eviction bugs often appear only after hours; concurrency ramps show when geometry distorts under burst.