Throughput vs. Latency: Picking Your Tradeoff

When you're running an inference service, you can't optimize for "speed." Speed is two numbers, and they fight each other. You pick which one matters for your workload and accept that you just made the other one worse.

The two numbers are throughput (how many tokens per second the GPU generates, summed across all in-flight requests) and latency (how long any one user waits for their next token). Operators want throughput, because that's how GPU hours turn into dollars. Users want latency, because watching a cursor blink is painful. An inference engine is the thing in the middle trying to convince both sides they're getting a good deal.

This post is about that negotiation. We'll look at why the tradeoff is a genuine curve (a Pareto frontier) and not a knob you can cheat, the two places in the stack where the tradeoff actually gets expressed (batch sizing and scheduling policy), and how 2026-era systems push the frontier outward by running prefill and decode on different hardware pools entirely.

If you've read Arc 5 you already know most of the vocabulary you need. If you haven't, the short version: an LLM request goes through two phases. Prefill processes the prompt all at once, producing the first token. Decode generates one new token per forward pass from there. The prefill is compute-heavy and bursty; decode is memory-bandwidth-bound and steady. Those two phases have completely different performance profiles, and they interact in ways that make scheduling interesting.

The two numbers

Let me be precise about what we're measuring, because the terminology is a mess across papers and vendor marketing.

Throughput is almost always reported as tokens per second per GPU, aggregated across all requests running on that GPU. If you have 32 users each generating 40 tokens/second, the GPU is running at 1280 tok/s. That's the number your finance team cares about because it divides directly into GPU-hour cost to tell you $/1M tokens.

Latency is actually three numbers that people sometimes squish into one:

• TTFT (time to first token): how long from the request hitting the server until the first token streams back. This is dominated by prefill plus any queueing delay.
• ITL (inter-token latency): the gap between successive tokens once generation is underway. Sometimes called TPOT (time per output token). This is the cadence a chat UI renders at.
• TTLT (total time): the whole thing, from request to final token. For short responses, TTLT is mostly TTFT. For long responses, it's mostly ITL times the response length.

These three numbers measure different parts of the experience. A 2-second TTFT with 20ms ITL feels snappy: a slight pause, then smooth typing. A 200ms TTFT with 200ms ITL feels sluggish: instant start, then chugging. Systems tune them differently, so when a system claims to be "fast," ask which one.

Why the tradeoff exists

So why can't we have all three? Low TTFT, low ITL, high throughput, everyone happy?

Because the thing that makes throughput go up (large batches) is the same thing that makes latency go up. Here's the mechanism.

Decode is memory-bandwidth-bound. As we worked out in the KV cache post and the roofline post, a single decode step for one request reads all of the model weights out of HBM, applies them to a single new token's worth of activations, and writes one new row to the KV cache. The arithmetic intensity is terrible: around 1 FLOP per byte of weights read. The GPU's compute units are starving for data the whole time.

If we batch 16 decode requests together and run them through the same forward pass, we read the weights once from HBM and use them for 16 different queries. Arithmetic intensity goes from ~1 to ~16. We're no longer memory-bound; we're much closer to compute-bound. Throughput, in tokens-per-second-per-GPU, goes up by roughly 16×.

Batching raises throughput because decode is memory-bound and batching shares the memory reads. It's basically free performance, if all you measure is throughput.

But here's where latency enters. Batching 16 requests means every request waits for the slowest request in the batch on every step. If one request has a KV cache that's 8k tokens long and another is only 200, the shorter one is now doing a forward pass with 40× more attention math than it needs. Its ITL is determined by the long one. And if we're doing static batching (the whole batch runs to completion before the next batch starts), short requests also wait for long ones to finish before anyone else gets in. Under continuous batching (which I'll summarize in a moment, and which you can read about in detail in that post) we can cycle new requests in as soon as any slot frees up, but the per-step time still grows with batch size.

So throughput climbs with batch. Latency climbs with batch. Both are real. The curve relating them is the Pareto frontier.

A few things to notice. Throughput doesn't climb forever. Past the knee (around batch 16 on this curve), each extra request buys you very little extra tok/s/GPU. KV cache memory and attention compute start to dominate, and the memory-bandwidth win saturates. Meanwhile latency keeps climbing, so you're paying full price for no real gain. The knee is the cheapest operating point for a given GPU; everything to the right of it is throwing user experience away for nothing.

The curve itself is a property of the system. Better kernels, paged attention, quantization, speculative decoding: each one of those shifts the curve outward. But for a fixed setup, you can slide along the curve with the batch dial. You cannot jump above it.

The same hardware, two products

The concrete version of this tradeoff is that the same physical GPU fleet can serve very different products by moving its operating point on the curve.

Picture a single cluster: Llama 3 70B running on 8× H100 SXM GPUs with TP=8, BF16 weights. Two product lines need to run on this cluster.

Product A: interactive chat. Users type, a cursor blinks, tokens stream back. The SLO is: p95 TTFT under 500ms, p95 ITL under 40ms. If ITL slips above 50ms the product feels laggy and users bail.

Product B: offline document summarization. The pipeline ingests a queue of 10k PDFs overnight, feeds each into the model with a prompt, collects the output. No user is watching. The SLO is: finish the queue by 6am, minimize dollars.

Same cluster. Same model. Two operating points.

For Product A we tune the engine to a small in-flight batch. Maybe max_num_seqs=4 in vLLM-speak. At batch 4 on this hardware we get roughly 1,500 tok/s/GPU and p95 ITL around 25ms. Throughput is mediocre but the UX is great. Lots of the GPU's compute is idle on any given decode step; we're paying for expensive hardware to deliver a snappy feel.

For Product B we raise max_num_seqs to 64 and let the scheduler pack. Throughput climbs to roughly 8,000 tok/s/GPU and p95 ITL rises to around 80ms. The GPU is much closer to saturated. No human notices the 80ms because no human is watching; the queue just drains about 5× faster in wall-clock.

Same model. Same hardware. The only thing that changed is the batch size knob and the resulting place on the curve. Product A operates near the bottom-left of the frontier, Product B near the top-right. If you tried to serve both from the same process at the same time with one setting, you'd be making at least one of them sad. The standard move is to run two deployments: a low-latency replica set sized to chat traffic, and a high-throughput replica set sized to the overnight queue. Some modern setups (vLLM, NVIDIA NIM, SGLang) let you apply per-request priority tiers so the same process serves both, but it's the same idea: you're operating at two points on the curve simultaneously, you just got the engine to do the routing.

Prefill and decode are different beasts

I said earlier that prefill and decode behave differently. This matters a lot for scheduling, so let me draw it out before we look at schedulers.

Prefill takes the whole prompt and runs one giant forward pass. If the prompt is 4k tokens, prefill does attention over a 4k×4k score matrix and feeds the whole thing through every layer. That's a big parallel workload with lots of FLOPs. On an H100 at FP16, a 4k-token prefill on a 70B model is on the order of 400ms of pure GPU time.

Decode takes one new query row at a time. The KV cache provides the keys and values from earlier positions; we just do one matmul, one softmax over a growing vector, one weighted sum. Each decode step on the same model is maybe 15-25ms depending on batch. The per-step work is tiny, but we do it hundreds of times per response.

The fact that prefill is a short high-compute burst and decode is a long memory-bound drip turns out to be the single most important shape in the serving story. A long prefill running on the GPU steals time from every other request's decode. You had a bunch of users happily streaming tokens at 40 tok/s each, then someone submits a 50k-token prompt, and for two seconds nobody's decode stream moves at all. Everyone's ITL spikes. That's classic head-of-line blocking.

FCFS (first come, first served) is what you get if you just queue requests and run them through. It's simple and fair in a naive sense, but p95 TTFT gets wrecked whenever a long prompt shows up. The 99th percentile is even worse. This is the default behavior most reference implementations start with, and it's rarely what you want in production.

The fix that actually works, and is now the default in vLLM, TGI, and most other engines, is chunked prefill (from Sarathi-Serve). Instead of running a 4k-token prefill as one big block, slice it into 512-token chunks. Between every chunk, let whoever's currently in decode take one step. The long prefiller finishes slightly slower in wall-clock (maybe 10-15% overhead from the chunking) but nobody else's decode stream ever freezes. The p95 TTFT for other requests stops being dominated by worst-case prompt length. Sarathi-Serve calls this "stall-free scheduling," which is exactly what it is: you never stall ongoing decodes to service a new prefill.

Chunked prefill sounds obvious once you see it. The hard part is picking the chunk size. Bigger chunks = higher arithmetic intensity = better prefill throughput, but also longer decode stalls between chunks. Smaller chunks = tighter ITL but worse prefill efficiency. The engine picks a size that's big enough to capture most of prefill's compute efficiency while small enough to keep ITL tight. Production systems tune this per model.

Other scheduling policies exist and get used in specific contexts. Shortest-job-first helps average TTFT at the cost of starving long requests; rarely used alone because of the starvation issue. SLO-aware scheduling looks at each request's latency budget and preempts lower-priority ones when a high-priority request is at risk of missing its SLO; this is what NVIDIA NIM does out of the box. Priority tiers are the user-facing version: Anthropic and OpenAI both let you mark requests as batch-priority (cheaper, higher latency) or standard-priority (more expensive, tighter latency), and the scheduler routes accordingly. Behind the scenes these are all just different ways of picking which requests get GPU time next.

Disaggregated serving

Chunked prefill is the clever software fix. The more radical fix is hardware.

Here's the observation. Prefill is compute-bound: it wants a GPU with high FLOPS. Decode is memory-bandwidth-bound: it wants a GPU with high HBM bandwidth. H100s are great at both, which is convenient but also means you're paying for half the hardware to sit idle at any given moment. The prefill GPU isn't using its memory bandwidth; the decode GPU isn't using its compute. Everything's wasted half the time.

What if we just didn't run them on the same GPU?

That's disaggregated serving, pioneered concurrently by DistServe (academic) and Splitwise (Microsoft). Run prefill on one pool of GPUs and decode on another. Each pool gets to run all the time at the shape of its workload. The prefill pool is never memory-starved; the decode pool is never compute-starved. You size each one independently based on your actual traffic mix (big prompts lean you toward more prefill GPUs, chatty users lean you toward more decode GPUs).

The catch is the handoff. When prefill finishes, the KV cache for that request now lives on the prefill pool's GPUs. Decode needs it on the decode pool's GPUs. So you transfer the cache over NVLink (if the pools share a node) or Infiniband (if they're across nodes). For a 70B model with 4k context, that's on the order of a few GB of transfer, which on modern interconnects takes ~20-50ms.

And this is the rub. If your QPS is low, that 30ms of transfer per request is pure overhead and you'd be better off just running co-located. If your QPS is high, the win from each pool running at its own roofline swamps the transfer cost, and you come out ahead by a lot. DistServe reports 7.4× more requests served or 12.6× tighter SLOs, at the same p95 latency, versus colocated vLLM. Splitwise reports 1.4× throughput at 20% lower cost, or 2.35× throughput at the same cost and power. These aren't marginal numbers; at scale, disaggregation starts to look less like an optimization and more like the obvious way to run things.

So when is it worth the operational complexity? A single-pool deployment is much simpler to reason about and debug. Disaggregation needs a cache transfer protocol, careful placement of which requests go to which prefill GPU so decode-side caches can land efficiently, and failure handling when one of the pools is overloaded. The rough rule of thumb in 2026 is: if you're serving more than a few hundred requests per second, disaggregation starts paying off. Under that, chunked prefill on a single pool is almost always the right move. This is why vLLM with --enable-chunked-prefill is the modern default: you get ~80% of the TTFT win without splitting your fleet.

A minimum implementation sketch

Here's roughly what configuring an engine for these two products looks like. This isn't real production code but it captures the shape.

# --- Interactive chat deployment (Product A) ---
# Small batch, tight ITL. Trade throughput for UX.
from vllm import AsyncLLMEngine, AsyncEngineArgs
 
chat_engine = AsyncLLMEngine.from_engine_args(
    AsyncEngineArgs(
        model="meta-llama/Llama-3-70B-Instruct",
        tensor_parallel_size=8,           # 8x H100
        max_num_seqs=4,                   # cap concurrent decodes low
        max_num_batched_tokens=2048,      # small prefill slices
        enable_chunked_prefill=True,      # never stall an active decode
        gpu_memory_utilization=0.85,
        # expected: ~1500 tok/s/GPU, p95 ITL ~25 ms
    )
)
 
# --- Batch summarization deployment (Product B) ---
# Large batch, throughput-optimized. ITL is irrelevant, no one's watching.
batch_engine = AsyncLLMEngine.from_engine_args(
    AsyncEngineArgs(
        model="meta-llama/Llama-3-70B-Instruct",
        tensor_parallel_size=8,
        max_num_seqs=64,                  # pack the batch
        max_num_batched_tokens=16384,     # bigger prefill chunks = higher TP
        enable_chunked_prefill=True,
        gpu_memory_utilization=0.95,      # squeeze KV cache capacity
        # expected: ~8000 tok/s/GPU, p95 ITL ~80 ms
    )
)

One model file, two processes, two points on the curve. The only knobs that meaningfully moved are max_num_seqs and max_num_batched_tokens. In production you'd likely front both with a router that picks the pool based on request metadata (priority tier, user class, workload tag). The router itself is about 200 lines of code. The real engineering is in running the two pools and getting the capacity sizing right.

Misconceptions

"More GPUs always means lower latency." Not unless you spend them the right way. You have two options. Add more GPUs to the same request via tensor or pipeline parallelism: one request now uses 16 GPUs instead of 8. That can reduce per-step decode time because the matmuls get split across more compute. But you hit scaling limits fast (communication overhead between GPUs), and past TP=8 you usually don't see much ITL improvement on decode. The other option is to use the extra GPUs to serve more concurrent requests. This buys throughput, not latency. In fact, it'll make your individual user's latency slightly worse as contention grows. More GPUs are only a latency win when you spend them on parallelism for one request, and parallelism has its own scaling ceiling.

"Continuous batching means there's no batching tradeoff anymore." Continuous batching (from Orca) is a real and important improvement, but it solves a specific problem: under static batching, a completed request leaves its slot empty until the whole batch finishes, wasting GPU time. Continuous batching lets new requests fill vacated slots immediately, which removes that particular kind of waste. What it doesn't do is eliminate the tradeoff curve itself. You still pick a max batch size. Smaller batch, lower latency, lower throughput. Larger batch, higher throughput, higher latency. The curve got more efficient. The curve is still a curve.

"TTFT is just prefill time." In a benchmark with one request, yes. In a real system, no. TTFT includes queueing delay, scheduling overhead, prefill compute, and any preemption. Under load, queueing dominates. If your p50 TTFT is 200ms but your p99 TTFT is 8 seconds, almost all of that tail is queue wait, not prefill math. This is why SLO-aware scheduling and priority tiers matter: they're managing the queue, not the math.

"Disaggregated serving is always faster." Only at the right concurrency. The KV cache transfer from prefill pool to decode pool costs something, and at low QPS that overhead dominates. At high QPS the per-pool efficiency wins swamp the transfer cost, sometimes by a large factor. The break-even is workload-dependent. When a paper reports a big speedup from disaggregation, the QPS matters as much as the ratio.

What's next

This post covered the Pareto curve that serving systems live on and the three things they can do to make the curve more friendly: pick the right batch size for your workload, use chunked prefill to keep long prompts from wrecking everyone else's decode, and disaggregate prefill from decode when QPS is high enough to earn it. Every modern inference engine is some combination of these three choices plus the memory tricks from earlier in Arc 6.

The next post catalogs the actual engines (vLLM, TGI, TensorRT-LLM, llama.cpp, SGLang) and shows how each one is the synthesis of the choices we've covered across Arc 6. Start with Comparing Engines.

Additional reading (and watching)

• Pope, R., et al. (2022). Efficiently Scaling Transformer Inference. arXiv:2211.05102. The framework paper for thinking about transformer inference as a multi-axis partitioning problem; introduces the latency/throughput analytical model this post builds on.

• Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS 2022. Makes the memory-bandwidth bottleneck of attention quantitative; the arithmetic-intensity argument for batching comes from the same mental model.

• Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. The vLLM paper. Chunked prefill, priority scheduling, and the continuous-batching implementation described here all live inside its successor codebase.

• Agrawal, A., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B. S., & Ramjee, R. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. OSDI 2024. The stall-free-scheduling paper; reports 2.6× serving-capacity gains on Mistral-7B and up to 5.6× on Falcon-180B from chunked-prefill plus uniform batch composition.

• NVIDIA. (2024). NVIDIA NIM for LLMs: Inference Microservices. Vendor documentation describing SLO-aware scheduling and priority classes as first-class deployment primitives.

• Zhong, Y., et al. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024. Disaggregated serving, with reported 7.4× request-rate or 12.6× SLO-tightening at matched latency versus colocated baselines.

• Patel, P., Choukse, E., Zhang, C., Goiri, Í., Shah, A., Maleki, S., & Bianchini, R. (2024). Splitwise: Efficient Generative LLM Inference Using Phase Splitting. ISCA 2024. Microsoft's production-grade phase splitting; 1.4× throughput at 20% lower cost, or 2.35× throughput at equal cost.

• Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., & Chun, B.-G. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022. The continuous-batching paper. Without Orca's iteration-level scheduling, none of the subsequent throughput wins described here work.

• Agrawal, A., Kedia, N., Panwar, A., Mohan, J., Kwatra, N., Gulavani, B., & Ramjee, R. (2023). Sarathi: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills. arXiv:2308.16369. The precursor to Sarathi-Serve; introduces the chunked-prefill-plus-decode-piggybacking primitive that became the industry default.