Bridge · Article 7b · Optional deep-dive

Model / Tensor Parallelism & the Interconnect Fabric

Why multi-GPU inference turns every forward pass into a synchronized exchange across the wire — and what happens on that wire.

Article 7 §2 flagged model parallelism and disaggregated serving as fabric problems. This deep-dive is the wire-level view: once a request is admitted (optionally after 7a's perimeter), a large model may not fit on one GPU — so each token's forward pass becomes a synchronized exchange across shards. Solo readers: you only need the idea that "sharding puts the network on the critical path of every token."

GPU 0shard of model GPU 1shard of model GPU 2shard of model GPU 3shard of model NVLink / NVSwitch sub-ms exchanges during a single forward pass across sharded GPUs InfiniBand / RoCEv2 (node-to-node) same idea, across servers instead of within one — this is where your L2-L7 test background applies directly
When a model does not fit on one GPU, tensor/pipeline parallelism turns every forward pass into a synchronized exchange across the interconnect fabric.

1. Why the fabric is on the token critical path

Tensor parallelism (TP) splits individual matrix ops across GPUs in a group — each layer needs an all-reduce (or all-gather) of activations before the next layer can proceed. Pipeline parallelism (PP) splits layers across GPUs/stages — activations move forward (and gradients would move back in training) at stage boundaries. Inference still pays the forward transfers. Either way, a slow hop delays every user sharing that GPU group for that token step — textbook tail-latency amplification.

2. Two topology tiers, same idea

TierTypical fabricWhat moves
Intra-nodeNVLink / NVSwitchTP shards inside one server — highest bandwidth, lowest latency
Inter-nodeInfiniBand or RoCEv2Same collectives across servers when the model (or PP stages) span nodes

RoCEv2 and IB are the networking home turf: lossless (or carefully loss-managed) fabrics, PFC/ECN behavior, and congestion that shows up as unexplained TPOT variance if you only watch GPU dashboards. Article 8 maps those fabric symptoms to the metrics side.

3. Over-sharding and negative returns

More GPUs is not free latency. Each extra TP rank adds collective traffic. Past a point, communication time exceeds the compute you saved by splitting the matmul — TTFT/TPOT get worse. That is the same diminishing-returns curve as striping a flow across too many links when serialization and sync dominate. Capacity planning here is "smallest TP that fits weights + KV with headroom," not "max GPUs available."

Network lens

The L2–L7 validation methodology used for RDMA fabrics in data-center testing applies directly: catch interconnect bottlenecks (congestion, pause storms, asymmetric paths) before they show up as inflated TTFT/TPOT in production. The workload on top is an LLM collective schedule instead of a generic RDMA benchmark — the failure modes on the wire rhyme.

Up next

You've seen GPUs talk to each other inside (and across) a node. Article 7c covers how a fleet decides which node gets a request in the first place, using the KV-cache-aware routing logic first introduced in Article 7.