Throughput vs. Latency: Picking Your Tradeoff

A batch can produce more tokens per second while making each person wait longer between tokens. The serving question is: how much useful work can the deployment finish while its responses remain timely?

The tradeoff is not inevitable at every operating point. Removing wasted work can improve throughput and latency together. But once a fixed setup is well tuned, additional load often raises latency faster than it raises completed work.

Measure the right interval

Throughput counts tokens or requests completed per second. Name the scope: an entire deployment, a node, or a rate normalized per GPU. Keep input tokens and output tokens separate unless you explicitly report their sum.

Time to first token (TTFT) includes queuing and prefill, plus any other work inside the chosen measurement boundaries. Inter-token latency (ITL) measures gaps in the stream. Time per output token (TPOT) usually averages the decode interval within a request; it does not preserve each individual gap. End-to-end latency measures the whole response.

For a response with $G$ generated tokens, the time from its first to last token contains $G-1$ gaps. A single average cannot tell you whether those gaps were steady or included a long pause.

A chat interface may need separate budgets for TTFT and streaming stalls. An overnight document job may care mainly about finishing before a deadline.

What batching changes

At a small decode batch, a dense model repeatedly reads many weights to process very few new tokens. For a BF16 matrix-vector multiply, roughly two arithmetic operations per weight and two bytes per weight give an idealized intensity near one FLOP/byte, before other traffic.

Batching lets a matrix multiply reuse those weights across more token positions. It increases arithmetic intensity and often improves aggregate throughput. It also increases compute and KV work per step. Once other costs dominate, gains diminish.

If every active request produces one token in a step lasting $t$ milliseconds, aggregate decode throughput is $1000B/t$ tokens/s for batch $B$. A batch of four at 25 ms per step produces 160 tokens/s in total. On an eight-GPU replica, that is 20 output tokens/s/GPU when normalized across the replica.

Long-context requests can increase shared step time, but a short request does not necessarily perform the long request's attention arithmetic. Packed and length-aware kernels avoid some padding; all requests still share finite execution resources. Static batches also retain empty lanes after short requests finish. Continuous batching admits replacements between iterations when budgets allow.

One fleet, different latency budgets

A small batch cap is one possible choice for chat, not a guarantee of low latency: the requests kept outside the active batch may spend longer queuing. Larger batches can help an offline queue finish sooner, but their benefit ends when execution saturates or memory pressure causes preemption.

Separate deployments let each workload use its own settings. Priorities inside one deployment provide another control, with their own fairness and starvation tradeoffs. They do not make the requests stop competing for the same GPU time.

Prefill can interrupt decode

Prefill processes many prompt positions together through each layer. Decode adds a new position for each active request while attending to cached keys and values. A long unchunked prefill can occupy the device long enough to create a noticeable gap in ongoing streams.

Chunked prefill limits the prompt work admitted in one iteration, allowing decode work to be scheduled between or alongside chunks. Smaller chunks may tighten streaming gaps at the cost of prefill efficiency; larger chunks make the opposite tradeoff. Each chunk still adds work to its scheduled iteration.

Keep scheduling dimensions separate. FCFS decides which waiting request has priority; continuous batching decides whether membership changes between steps; chunking decides how much prompt work runs at once. A production engine can combine all three. Sarathi-Serve studies the throughput/latency effects of chunking and mixed batches. Sarathi-Serve paper.

Separating prefill and decode

Disaggregated serving moves the two phases onto separate resources. This can reduce interference and let operators size each pool independently. It also adds a handoff: decode needs the KV state produced by prefill.

For a hypothetical full-attention model with 80 layers, eight KV heads, head dimension 128, and two-byte cache elements, a 4,096-token cache contains about 1.34 decimal GB before metadata. Dividing those bytes by effective link bandwidth gives a transfer-time lower bound. Sharding, topology, overlap, and protocol overhead determine actual time.

To test whether splitting pays off, compare colocated and split configurations at equal resource cost, with the same prompt and output lengths. Report goodput under the same latency budgets. DistServe and Splitwise study this design with different placements and workloads.

Two configurations to measure

The commands below are alternative starting points, each occupying its own eight-GPU replica. Run one at a time on a single node. The parameter values are experimental choices, not predicted latency or throughput results.

# Smaller active-request and per-iteration token budgets.
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 8 \
  --max-num-seqs 8 \
  --max-num-batched-tokens 2048 \
  --enable-chunked-prefill
 
# Alternative configuration for a throughput-oriented experiment.
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 8 \
  --max-num-seqs 64 \
  --max-num-batched-tokens 16384 \
  --enable-chunked-prefill

max-num-seqs limits scheduled sequences; it is not an end-to-end queue limit. The token budget covers work scheduled in an iteration, rather than simply setting the total supported context. Confirm behavior for the installed engine version in the vLLM serving reference.

Measure the complete response distribution under the intended arrival process. More replicas may reduce queuing; more tensor-parallel ranks may reduce per-device compute while adding communication. Compare them against the latency and capacity requirements of the deployment.

Next, compare inference engines using these workload and measurement boundaries.

Additional reading

• Yu et al., Orca.
• Agrawal et al., Sarathi-Serve.
• Zhong et al., DistServe.
• Patel et al., Splitwise.