Tensor, Pipeline, and Expert Parallelism

At some point in the life of any inference stack, the model outgrows a single GPU. Llama 3 70B in FP16 needs 140 GB of weight memory. An H100 has 80 GB. So on a single device, you can't even load the thing. And even if you could, you'd be stuck running one copy of the model's compute, which caps what you can serve. The whole game of serving big models is taking that model and splitting it across many devices in a way that (a) makes it fit and (b) keeps every GPU busy.

There are three orthogonal ways to do that. This post is about the handles you need to hold all three in your head at once: tensor parallelism, pipeline parallelism, and expert parallelism. They slice the same model along three different axes. Real systems combine them. Once you see the three axes as separate dimensions, everything about how Llama 405B or DeepSeek V3 gets served stops being mysterious.

Before we start, a tiny bit of vocab from earlier arcs. A transformer layer is mostly a stack of matrix multiplies. An all-reduce is a collective operation where every GPU contributes a partial result and, at the end, every GPU has the summed result. It's the bread and butter of model parallelism, and its cost is set by how fast the interconnect is between the GPUs. If your interconnect is slow, all-reduce is slow. If your interconnect is slow and you do an all-reduce on every layer, you have a bad time.

The model doesn't fit, and even when it does, throughput suffers

Let me be concrete. Here's what the memory looks like for serving a 70B model in FP16 on one H100:

140 GB of weights on an 80 GB card. Even if you heroically quantized to INT4 (sub-4x compression) to squeeze the weights under 80 GB, you'd have almost nothing left for KV cache and you'd be serving one request at a time. At that point you're running a demo, not a serving stack.

So you split. But once you're splitting, you have choices, and those choices have big consequences for latency and throughput.

Three orthogonal axes

I like to think of the model as a loaf of bread. You can slice the loaf three different ways, and you can do all three at the same time.

Tensor parallelism (TP) slices horizontally through every weight matrix. Each GPU gets a horizontal slab (or equivalently, a set of columns) of every single layer. Every layer now needs an all-reduce across the TP group to stitch partial outputs back together. This is comm-heavy, but if the interconnect is fast it's also latency-friendly.

Pipeline parallelism (PP) slices vertically between layers. GPU 0 gets layers 0-39, GPU 1 gets layers 40-79. The activations are passed forward GPU-to-GPU once per token (tiny compared to a weight matrix), so PP can stretch over slower links. But it introduces "bubbles," idle time while later stages wait for earlier ones.

Expert parallelism (EP) only applies to Mixture-of-Experts models. Different experts sit on different GPUs. Every MoE layer does a pair of all-to-all collectives to ship tokens to the GPU that owns their routed expert and then ship the outputs back.

The hero at the top of the page cycles through these three slicing modes on the same loaf. The loaf is the model. The slicing is the distribution plan. You can, and in practice must, combine them.

And there's a fourth axis that sits orthogonal to all three: data parallelism (DP) for serving means spinning up independent replicas of the whole engine. DP scales horizontally. TP, PP, and EP scale the same model across devices so it can run at all. I'll flag DP a couple of times below, but the focus here is inside-one-replica.

Tensor parallelism: split every matrix

Tensor parallelism is the most intuitive of the three once you remember that a transformer layer is mostly matmuls. Take a weight matrix $W \in \mathbb{R}^{d_{\text{in}} \times d_{\text{out}}}$. Split $W$ along the output dimension into $N$ column slices, one per GPU. Call them $W_1, W_2, \ldots, W_N$. Then:

Each GPU holds its $W_i$, computes $y_i = x W_i$ locally, and then a concatenate-or-sum collective assembles $y$. In practice, the pattern Megatron-LM popularized is: column-split the first matmul of a block, row-split the second, and the two splits cancel out so you only need one all-reduce per block instead of two. Still, "one all-reduce per block" means hundreds of all-reduces per token for a big model. The thing that makes TP work at all is an interconnect fast enough that those all-reduces are a small fraction of the compute time.

Two things from the viz:

• The weights are split, but the input and output are replicated. Every GPU in the TP group sees the same $x$ and ends up with the same $y$ after the all-reduce. That's what lets the next layer's TP matmul just work: the input is already everywhere it needs to be.
• The communication volume is proportional to the output dimension of each all-reduced tensor. So TP cost scales with the size of the activations, not the size of the weights. In a big model, the activations are tiny compared to the weights, which is why TP is viable.

Because TP is on the critical path of every layer, it's usually done inside a single node where GPUs are connected by NVLink (900 GB/s on H100, much faster than PCIe). Crossing out of the node for TP means your all-reduces are now gated by the NIC, and the math gets painful. So the standard move is TP within a node. Eight GPUs per node, TP=8 is the common setting.

Pipeline parallelism: stages in a factory

Pipeline parallelism is the "assembly line" split. GPU 0 gets the first block of layers, GPU 1 gets the next block, and so on. A token goes through stage 0, hands off its activations to stage 1 over the network, and disappears from GPU 0's memory.

If you do this naively, one input going through all stages before you send the next one, you get this:

The naive schedule is a disaster. GPU 0 works while GPUs 1-3 wait. GPU 1 works while the others wait. At any moment, only one GPU is doing anything.

The fix is micro-batching, which comes from GPipe. You split the batch into $K$ micro-batches and feed them into the pipeline one at a time. Once the pipeline is full, every stage is busy on a different micro-batch. The "bubble" at the start (stages 1-3 waiting for the first micro-batch to arrive) and at the end (stage 0 done, waiting for stage 3 to finish the last one) still exists, but it shrinks as $K$ grows. The classic bubble fraction is roughly $(N - 1) / (K + N - 1)$ for $N$ stages and $K$ micro-batches. With $N = 4$ and $K = 4$, that's about 43%. With $K = 16$, it's about 16%. You trade latency (bigger $K$ means more micro-batches flowing before any given request finishes) for throughput (less bubble waste).

For serving specifically, there's a nicer property: a typical serving workload already has many concurrent requests in flight. So you can treat each request, or a chunk of each request, as a micro-batch. The pipeline stays full because the traffic keeps coming. This is why PP plays well with continuous batching (see continuous batching).

PP's comm cost is small. You ship activations between stages once per stage boundary. An activation is shape $[B, T, d]$, which is orders of magnitude smaller than a weight matrix. So PP can stretch across nodes over standard datacenter networking, no NVLink required. Save NVLink for TP.

Expert parallelism: one per expert, shipped by all-to-all

Expert parallelism only applies to Mixture-of-Experts models. The quick version of MoE: instead of one giant feedforward network per layer, you have, say, 64 experts and a small router that picks a few experts per token. The total parameter count is huge, but only a handful of experts fire per token. (If you want the full picture, the MoE post covers the architecture.)

For serving, this has a natural parallelism story. Each expert lives on one GPU. Tokens get routed to the GPU that owns their chosen expert. After expert compute, tokens go back home. The shipping is done with an all-to-all collective, where every GPU sends different data to every other GPU simultaneously.

What makes EP tricky is load balance. If 80% of tokens want Expert 3, the GPU holding Expert 3 becomes a bottleneck while the others idle. MoE training has auxiliary load-balance losses exactly to prevent this, but at inference time you're stuck with whatever the model learned. Some engines add capacity limits (drop tokens past a cap) or use expert-choice routing variants. DeepSeek-V3 added shared experts on every GPU to absorb common patterns and keep the routed experts evenly utilized.

Two things about EP worth calling out. First, EP's comm cost is per-MoE-layer, not per-every-layer. Dense attention and MLP layers don't participate, so if 1 in 2 blocks is MoE, EP only costs you half as often as TP would. Second, EP combines well with TP. You can TP-split inside each expert and distribute experts across GPUs. The production MoE stacks do exactly this.

Why the choice depends on the network

Each axis has a different communication profile:

They look similar in volume. The difference is frequency. TP pays its comm bill on every single layer, which for a 70B model is something like 80 times per token. PP pays only at stage boundaries, maybe once or twice per token. EP pays twice per MoE layer, so maybe 20-40 times per token depending on density.

That's why TP demands a fast interconnect and PP doesn't. The standard recipe is: TP within a node, PP across nodes. Inside a node, NVLink gives you the bandwidth to absorb frequent all-reduces. Across nodes, the Ethernet or InfiniBand link is slower but adequate for PP's occasional activation hand-off.

For EP, the answer depends on the all-to-all fabric. On a DGX-class node with NVSwitch, all-to-all inside the node is fine. Across nodes, you want an interconnect that handles dense all-to-all well, which is exactly what "AI datacenter fabrics" (like NVIDIA's Quantum InfiniBand rail-optimized topology) are built for.

Worked example

Now the payoff. Let's plan a serving layout for Llama 3 70B in FP16 on 8 H100s (80 GB each).

The model has 140 GB of weights. 8 GPUs have $8 \times 80 = 640$ GB of aggregate memory. Plenty. But the distribution matters because we want to fit the KV cache, leave room for activations, and keep the interconnect happy.

Option A: TP=8, PP=1. Every GPU holds $140 / 8 = 17.5$ GB of weights. Every GPU has $80 - 17.5 - 4 \approx 58$ GB free for KV cache. Every layer does an 8-way all-reduce over NVLink, which on H100 is measured in tens of microseconds per layer, so total per-token comm cost is around 1-2 ms. Latency-friendly, single-node.

Option B: TP=4, PP=2. Each GPU holds $140 / 8 = 17.5$ GB still, because we've split by 4 in one axis and 2 in the other (total split = 8). Same per-GPU weight footprint as Option A. But now every layer only does a 4-way all-reduce (smaller group), and we need one cross-stage activation send per token. Slightly less TP overhead per layer. Bubble overhead shows up in PP, but micro-batching hides it.

One quick note before the viz: the default shown below assumes FP8 weights (so 8.75 GB/GPU), not the FP16 numbers from Option A/B above. At FP16 the same layout holds 17.5 GB/GPU. Same shape either way, just different byte width.

Either way the shape of the answer is the same: the total weight footprint divided evenly across 8 slots, each slot defined by (pipeline stage, tensor-parallel rank).

So which is better? It depends on what you're optimizing for. If you have a single 8-GPU node with NVLink and you want the lowest per-request latency, go TP=8 because everything stays on fast links, no cross-stage hops. If you have two 4-GPU nodes with slow interconnect between them, go TP=4 x PP=2 so TP stays inside each node (fast) and PP spans the node boundary (which is OK because it's rare). If you're latency-tolerant and want max throughput, add DP replicas on top of whatever you chose.

Real serving deployments of Llama 3 70B and Llama 3.1 405B on H100 clusters use TP=8 within a node, and stack PP across multiple nodes for 405B. Mixtral 8x22B and DeepSeek-V3 add EP to the mix because they're MoE.

A tiny implementation sketch

Here's how this looks in a modern inference engine's config. Nothing fancy; the key is to see that the parallelism plan is just a few numbers.

# --- Serving Llama 3 70B on 8x H100 ---
# TP=8 within a single NVLink-connected node.
from vllm import LLM, SamplingParams
 
llm = LLM(
    model="meta-llama/Llama-3-70b-hf",
    tensor_parallel_size=8,      # column-split every W across 8 GPUs
    pipeline_parallel_size=1,    # no PP on a single node
    dtype="float16",
    max_model_len=8192,
)
 
# --- Serving Llama 3.1 405B on 2 nodes x 8 H100 ---
# Combine TP (inside each node) with PP (across nodes).
llm = LLM(
    model="meta-llama/Llama-3.1-405B-FP8",
    tensor_parallel_size=8,      # TP=8 inside each node, NVLink-fast
    pipeline_parallel_size=2,    # PP=2 across nodes, only per-stage hops
    dtype="auto",                # FP8 weights on H100
    max_model_len=16384,
)
 
params = SamplingParams(max_tokens=256, temperature=0.7)
for out in llm.generate(["Write a haiku about all-reduce."], params):
    print(out.outputs[0].text)

The tensor_parallel_size and pipeline_parallel_size knobs are all the plan is. The engine handles the NCCL collectives, the activation hand-offs, and the micro-batching for you. Under the hood it's the same Megatron-style splits, just packaged.

Misconceptions

"More GPUs always mean lower latency." I had to unlearn this one. There's a sweet spot, and past it, communication dominates. Going from TP=4 to TP=8 on a single NVLink node usually helps. Going from TP=8 to TP=16 (now crossing the node boundary) often hurts latency even though it helps aggregate throughput via DP replicas. Each extra GPU in the TP group means a larger all-reduce ring, and the ring's latency is bounded by the slowest link in it. Add one PCIe-attached GPU and the whole ring slows down.

"Pipeline parallelism is just sequential execution of layers across GPUs." Without micro-batching, that's exactly what it is, and it would be useless. The entire point is that micro-batches overlap: while stage 3 is working on micro-batch A, stage 2 is on B, stage 1 is on C, and stage 0 is on D. The bubble only exists at the fill and drain of the pipeline, and continuous batching in a serving system keeps the fill/drain permanently primed by new requests.

"DP and TP are alternatives." They're orthogonal. DP replicates the whole model $R$ times for throughput. TP shrinks a single model across $N$ GPUs to make it fit. Real systems use both: a TP=8 engine as one "replica," then run $R$ of those replicas behind a load balancer. The DP axis is what you scale for QPS, and the TP/PP/EP axes are what you tune to fit the model.

"Expert parallelism is just a training-time concern." It used to be mostly true. In 2026, it's not. DeepSeek-V3, Mixtral 8x22B, and Llama 4 all ship MoE architectures at serving time, and production engines (vLLM, SGLang, TensorRT-LLM) have expert-parallel code paths. If you're serving an MoE model at scale, EP is as real as TP.

What's next

Parallelism is how big models fit onto hardware. But "fits" and "serves well" are different things. Every parallelism choice changes the throughput and latency numbers. The next post takes the output of all these architectural choices and asks how serving systems navigate the inevitable tradeoff between throughput and latency. Start with Throughput vs. Latency.

Additional reading (and watching)

• Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019). Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053. Introduced the column-then-row TP pattern that cuts per-block all-reduces to one.

• Huang, Y., et al. (2018). GPipe: Easy Scaling with Micro-Batch Pipeline Parallelism. arXiv:1811.06965. The foundational paper for micro-batched pipeline parallelism.

• Narayanan, D., et al. (2021). Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM. SC'21 / arXiv:2104.04473. The "PTD-P" paper that combines TP + PP + DP at scale, with the standard bubble-fraction analysis.

• DeepSeek-AI. (2024). DeepSeek-V3 Technical Report. arXiv:2412.19437. Explicit parallelism plan and load-balance strategy (shared experts + auxiliary-loss-free routing) for a frontier open MoE model.

• Pope, R., et al. (2022). Efficiently Scaling Transformer Inference. arXiv:2211.05102. Google's paper on serving-time parallelism choices, including how communication topology shapes TP decisions.

• Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP'23 / arXiv:2309.06180. The vLLM paper. The engine has since added PP and EP, but this is the foundation.

• Korthikanti, V. A., et al. (2022). Reducing Activation Recomputation in Large Transformer Models. arXiv:2205.05198. Introduces sequence parallelism, a refinement of TP that further shards LayerNorm/dropout activations along the sequence dimension.

• Lepikhin, D., et al. (2020). GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. arXiv:2006.16668. One of the original papers on sharding MoE models across devices; the all-to-all routing pattern shown in the viz descends directly from this.

• Fedus, W., Zoph, B., & Shazeer, N. (2021). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv:2101.03961. The Switch Transformer paper that popularized single-expert-per-token MoE and made expert parallelism a practical concern.

• Rajbhandari, S., Rasley, J., Ruwase, O., & He, Y. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. arXiv:1910.02054. Training-time memory optimization; the ideas (sharded optimizer state, parameters, gradients) inform how serving engines shard model state too.