Inference Stack Observability
Monitoring — your comfort zone, applied to a new stack. Mapped, metric by metric, to networking analogs.
In Article 7, load balancers routed on KV-cache utilization and queue depth — not geography. Those decisions are only as good as the signals below. (If you took the optional deep-dives — perimeter in 7a, fabric in 7b, placement in 7c — you are joining the same map with those layers already named.) Engines and exporters typically scrape these from the inference server (vLLM, TensorRT-LLM, TGI, and similar) plus GPU/fabric agents; this page is the glossary of what to watch and what each number means in networking terms.
1. Latency metrics — the "RTT equivalents"
| Metric | Networking analog | What it tells you |
|---|---|---|
| TTFT | RTT / first-byte latency | What a user perceives as "responsiveness." |
| TPOT | Inter-packet gap / jitter | Steady TPOT = smooth streaming; spiky TPOT is as noticeable to a user as jitter on a voice call. |
| TTLT | Full transfer completion | TTFT + all TPOT steps combined — full response completion time. |
| E2E latency | Total session time | Total request time, but on its own it hides whether the problem was prefill (slow first response) or decode (slow, uneven streaming). |
2. Throughput metrics — the "bandwidth equivalents"
| Metric | Networking analog | What it tells you |
|---|---|---|
| Tokens/sec (aggregate) | Cluster-level bandwidth | Total capacity, not experience. |
| Tokens/sec per request | Per-flow throughput | Correlates with actual user experience — the way per-flow throughput matters more than aggregate link utilization when diagnosing one slow session. |
| Requests/sec at target latency | SLA-conditioned throughput curve | Same shape as a load test reporting "max sustained throughput while p99 stays under X ms." |
| Concurrent prompts | Active-flow / connection concurrency | Classic ramp-hold-drain load-test profile. |
3. Utilization & saturation — the "buffer occupancy equivalents"
| Metric | Networking analog | What it tells you |
|---|---|---|
| Cache utilization % | Buffer occupancy | Climbing over the whole inference window with no relief is the pattern that precedes a collapse. |
| Scheduler state (running vs. waiting) | Queue depth on a switch | The gap is the admission queue backing up — a leading indicator that TTFT is about to climb, before GPU utilization even looks saturated. |
| GPU power & Tensor Core utilization % | Link utilization | Low tensor utilization with a maxed-out cache and a growing queue means the bottleneck isn't compute. |
| Prefill / decode time | — | Time to ingest and generate output — feeds TTFT directly. |
4. Where your L2-L7 background gives you an edge
Most AI teams treat inference as purely a compute problem. At multi-GPU/multi-node scale, it isn't:
- Inter-GPU interconnect metrics (NVLink/NVSwitch, or InfiniBand/RoCE across nodes) — GPUs exchange activations and KV cache shards constantly during distributed inference. Congestion, ECN marking, or PFC pause frames on that fabric directly inflate TTFT/TPOT, the same way they'd inflate RTT in any RDMA environment.
- Tail latency amplification — in a sharded/tensor-parallel setup, one slow network hop delays the entire token-generation step for every user sharing that GPU group. Textbook "one congested link degrades every flow behind it" — the same tail-latency-under-load testing already done for data center fabrics, with an LLM sitting on top instead of a generic RDMA workload.
- Packet loss / retransmission on the inference fabric — nearly invisible to an ML engineer watching only GPU dashboards, but it shows up as unexplained TPOT variance. This is where L2-L7 validation tooling has more diagnostic power than anything in the standard LLM-observability stack.
5. Quality / business metrics (usually ML/product-owned)
Guardrail trigger rate, output drift, cost-per-1K-tokens — real signals, usually owned by ML/product. Infra still cares when cost-per-token or guardrail rate spikes after a deploy: that is often the first hint that a serving or routing change moved the wrong lever.
These are the metrics that make inference production-grade and debuggable — and the mapping above is deliberate: nearly every one has a direct analog to something already lived in the networking domain.
You can now see a problem in the metrics. Article 9 is the capstone: every optimization lever in this portal, organized around the same two bottlenecks — prefill and decode — that have shown up in every article so far.