Prefill vs. Decode: The Two Phases of Inference

You send a 3,000-word prompt to an API. A few hundred milliseconds later, the first token appears. Then tokens start streaming in, one after another, at a steady rhythm. Maybe 30 or 40 per second.

Have you ever noticed that the wait before the first token is much longer than the gap between subsequent tokens? And that the initial wait gets worse when your prompt is longer, but the streaming speed stays about the same?

That's not a quirk of networking. It's the fundamental architecture of how inference works. Two separate phases happen under the hood, with different computational profiles, different bottlenecks, and different hardware requirements.

I put off learning this for too long. I kept treating inference as one undifferentiated "run the model" step. Once I understood the two-phase structure, a bunch of other things made sense: why input tokens are cheaper than output tokens, why prompt caching is so impactful, why speculative decoding helps generation speed but not prompt processing, why the H100's memory bandwidth upgrade mattered more than its FLOPS improvement for serving. It all comes back to this one distinction.

Prerequisites

You should be comfortable with the basics of transformer inference from Post 5.1, specifically the idea that training processes full sequences in parallel while inference generates one token at a time. That post also builds the hardware intuition this one leans on: arithmetic intensity, the roofline model, and the ridge point that separates memory-bound work from compute-bound work. You'll also want a working sense of matrix multiplication shapes (what happens when you multiply an $(n, d)$ matrix by a $(d, d)$ matrix) and the self-attention mechanism from Arc 4.

The setup

In Post 5.1 we talked about the gap between training and inference. During training, you shove the full sequence through the model in one parallel forward pass. During inference, you generate one token at a time, feeding each output back as input to produce the next.

But I glossed over something important. When you send a prompt to an LLM, the model doesn't start generating from scratch. It first has to process your entire prompt. And the way it processes the prompt looks completely different from the way it generates new tokens.

These are the two phases: prefill and decode.

Prefill: processing the prompt

When your prompt arrives at the model, all the tokens already exist. There's nothing to predict yet. The model just needs to build up its internal representation of the entire prompt, computing attention patterns and filling in the key-value cache for every token.

This looks a lot like a training forward pass. You stack all the prompt tokens into a matrix, multiply through the weight matrices in parallel, compute self-attention across the full sequence, and produce a hidden state for every position simultaneously. If your prompt has 1,000 tokens, the model processes all 1,000 in one big batch of matrix multiplications.

The key computation at each attention layer is:

where $X$ is the full prompt matrix with shape $(n_\text{prompt}, d_\text{model})$. That $QK^T$ multiplication produces an $(n_\text{prompt} \times n_\text{prompt})$ attention matrix. For a 2,000-token prompt, that's a 4-million-element matrix at each head, at each layer. Big matmuls, lots of arithmetic.

Prefill is compute-bound. The GPU's arithmetic units are the bottleneck. The matrices are large enough that the GPU's compute cores are fully saturated doing multiply-accumulate operations. Memory bandwidth isn't the limiting factor because the ratio of computation to data movement (arithmetic intensity) is high. You're doing $O(n^2 \cdot d)$ math on $O(n \cdot d)$ data.

The time this takes is your time-to-first-token (TTFT). The user is staring at a blank screen until prefill finishes and the model produces its first output token. Longer prompt, longer TTFT. It's roughly linear in prompt length (with a quadratic attention component that kicks in for very long sequences).

After prefill completes, the model has two things: a prediction for the first output token, and a KV cache populated with keys and values for every prompt token. That cache is critical. We'll cover it in detail in Post 5.4, but the short version is: it stores the K and V projections so the model doesn't have to recompute them for the prompt tokens during generation.

Decode: generating one token at a time

Once the first token is produced, we enter decode phase. Now the model is doing something fundamentally different.

At each decode step, you feed in exactly one new token. The model computes Q, K, V for that single token, appends the new K and V to the cache, then computes attention against the entire cache (all the prompt tokens plus all previously generated tokens). Out comes a probability distribution over the vocabulary, you sample a token, and repeat.

The matrix shapes change dramatically. Instead of multiplying an $(n, d)$ matrix by $(d, d)$ weight matrices, you're multiplying a $(1, d)$ vector by $(d, d)$ weight matrices. One row for one token. The actual arithmetic is tiny compared to prefill.

So what's the bottleneck? Memory bandwidth. At each decode step, the GPU has to read the entire model's weight matrices from GPU memory (HBM) to its compute cores, just to process that single token. For a 7-billion-parameter model in float16, that's about 14 GB of weights that need to be read from memory every single step. The GPU does a tiny amount of math with those weights (a vector-matrix multiply), then throws them out of the register file and reads them again for the next layer.

The arithmetic intensity is abysmal. You're doing $O(d)$ operations per weight parameter you load, compared to $O(n \cdot d)$ during prefill. The compute cores sit mostly idle, waiting for data to arrive from memory. This is what people mean when they say decode is memory-bound.

This is the roofline picture from Post 5.1, with prefill and decode landing on opposite sides of the ridge: prefill up around 1,000 FLOPs per byte, decode down at 1. The exact same hardware, running the exact same model weights, hits a completely different bottleneck depending on whether it's processing a batch of prompt tokens or a single generated token.

The time between consecutive generated tokens is your inter-token latency (ITL). The key observation: ITL barely changes with prompt length. Whether your prompt was 100 tokens or 10,000 tokens, each decode step does roughly the same amount of work. The attention over the growing cache gets slightly more expensive as the sequence grows, but the dominant cost is reading the weights, and that doesn't change. The streaming speed the user perceives is governed almost entirely by ITL.

Two metrics the user actually feels

When you interact with a chatbot, your subjective experience of speed maps directly onto these two metrics.

TTFT is the awkward pause after you hit send. You're waiting. Is it thinking? Did it crash? For interactive applications, TTFT above a second starts to feel sluggish. Above three seconds and users start re-submitting. This is pure prefill time (plus some network overhead).

ITL is the streaming rhythm. Once tokens start flowing, do they feel smooth and fast, or choppy and halting? An ITL of 25ms means 40 tokens per second, which is well above typical reading speeds and feels fully fluent. An ITL of 100ms means 10 tokens per second, which feels noticeably slower but is still readable. Above 200ms and the experience starts to degrade.

I think about it like this: TTFT is the response time of the system, and ITL is the throughput. They're governed by different bottlenecks, which means you optimize them with different levers. To bring TTFT down, you need faster compute (better GPUs, more tensor parallelism, or a shorter prompt). To bring ITL down, you need faster memory bandwidth (higher HBM bandwidth, quantized weights, or a smaller model).

This is also why API providers charge differently for input and output tokens. Processing a prompt token during prefill is cheap on a per-token basis. The GPU is doing efficient, parallelized matrix math with high hardware utilization. Generating an output token during decode is expensive. The GPU is underutilized, burning memory bandwidth to read all the weights for one token's worth of work. When Anthropic or OpenAI charge 3-5x more for output tokens than input tokens, they're reflecting the actual compute asymmetry.

A worked example

Let me trace through a concrete scenario to make the numbers tangible.

Say you send a 1,000-token prompt to a 70B parameter model running on 8 H100 GPUs with tensor parallelism. You want a 200-token response.

Look at the split. The prefill, all 1,000 tokens of it, took about 60 ms. The decode, for a fifth as many tokens, took 5 seconds. Over 98% of the wall clock went to the phase that produced the fewer tokens.

Prefill is fast because it's parallel and compute-bound. Decode is slow because it's sequential and memory-bound. When people say "inference is expensive," they almost always mean decode.

Implementation sketch

If you strip away the optimizations, the prefill/decode split is surprisingly simple in code. Here's the core loop, ignoring KV cache management and sampling details.

During prefill, model.forward gets an $(n, d)$ matrix to chew on. During decode, it gets a $(1, d)$ vector. Same weight matrices, same memory reads, dramatically different utilization.

Why this gets worse with long prompts

The example above used a modest 1,000-token prompt. But modern applications routinely stuff 10,000 to 100,000 tokens of context into the prompt. RAG pipelines retrieve chunks of documents. Agents inject tool results. System prompts carry extensive instructions.

For a 50,000-token prompt on the same setup, prefill dominates. At ~17,500 tokens per second, TTFT climbs to roughly 2.8 seconds. The user is waiting nearly three seconds before seeing anything. And if you're running a long-context model with a 128K window and the user actually fills it, TTFT can balloon to 7+ seconds.

The quadratic attention cost makes this worse than linear. That $(n \times n)$ attention matrix for 50,000 tokens has 2.5 billion entries per head per layer. FlashAttention helps enormously by avoiding materializing this matrix, but the compute still scales quadratically with sequence length.

Meanwhile, the decode phase is almost unchanged. ITL stays at ~25 ms whether the prompt was 1,000 or 50,000 tokens. The KV cache is larger (so attention computation during each decode step involves more key-value pairs), but the dominant cost of reading weight matrices doesn't change.

This creates a real tension for system designers. Users who send long prompts want them processed fast (low TTFT). Users who are mid-generation want their tokens to keep flowing without interruption (low ITL). But a long prefill hogs the GPU and blocks decode steps. You can't fully optimize for both simultaneously on the same hardware. This tension is exactly what chunked prefill tries to resolve.

Chunked prefill

There's a practical problem that arises when you're serving multiple users simultaneously. Say one user sends a 50,000-token prompt while another user is mid-decode. The naive approach is to process the entire 50,000-token prefill in one shot, which monopolizes the GPU for 2-3 seconds. During that time, the second user's decode steps are blocked, so their token stream freezes for several seconds. From the second user's perspective, the model just stopped working.

Agrawal et al. proposed a solution called chunked prefill in their Sarathi paper. The idea is simple: break the long prompt into smaller chunks (say, 512 or 2,048 tokens at a time) and interleave prefill chunks with decode steps from other requests.

Each prefill chunk is small enough that it finishes quickly, and between chunks, the scheduler can slip in decode steps for other users. The first user's TTFT gets slightly worse (the prefill is spread out over more wall-clock time), but the second user's streaming doesn't freeze. This is the kind of scheduling decision that inference engines like vLLM, Orca, and SGLang make automatically.

There's a subtlety here about chunk size. Smaller chunks give more opportunities to interleave decode steps, reducing tail latency for other users. But they also reduce the arithmetic intensity of the prefill computation, pushing it closer to the memory-bound regime. A chunk of 512 tokens still has reasonably high arithmetic intensity (you're doing 512 rows of matrix math per weight load), but a chunk of 32 tokens starts to look more like decode. Production systems typically pick chunk sizes in the 256-2,048 range as a sweet spot.

Chunked prefill also helps with memory. A 50,000-token prefill produces massive intermediate activations (that quadratic attention matrix, or at least the tiles of it that FlashAttention processes). Chunking limits the peak activation memory to whatever the chunk size requires. You trade a tiny amount of prefill efficiency for dramatically better memory predictability.

Batching: the escape hatch for decode

If decode is memory-bound because you're reading all the weights for a single token, the obvious question is: what if you read the weights once and process multiple tokens from different requests? That's batching, and on the roofline plot it's the lever that walks decode rightward up the slope, 32 users meaning 32 tokens of work per weight read instead of 1.

This is why serving systems obsess over batch size. A larger decode batch means better GPU utilization, which means lower cost per token. But there's a catch: each user in the batch also needs their own KV cache. With 32 concurrent users, you need 32 separate KV caches in GPU memory, and that memory adds up fast. A single KV cache for a 70B model with 128K context can easily consume 10+ GB. Managing this memory efficiently is what PagedAttention (Post 6.3) is all about.

The tension between wanting large batches for efficiency and having limited GPU memory for KV caches is one of the central engineering challenges in LLM serving. We'll come back to this repeatedly in Arc 6.

Common misconceptions

"Prefill and decode use the same amount of compute per token." They really don't. Prefill loads each weight matrix once and multiplies it against all 1,000 prompt tokens, so the memory traffic per token is roughly a thousandth of what decode pays for its single token. Same weights, same reads, wildly different amortization.

"Longer prompts mean slower generation." Sort of, but the mechanism is specific. Longer prompts mean slower TTFT (longer prefill). But ITL during decode is mostly independent of prompt length. The initial wait is longer, but once tokens start streaming, the speed is about the same.

What's next

This split between prefill and decode restructures how you think about every inference-related topic. Prompt caching saves money because it skips prefill. Quantization speeds up generation because it reduces the bytes the GPU reads per decode step. Providers charge differently for input and output tokens because the hardware cost structure is fundamentally different on each side.

The next post, Why One New Token Means One New Row, zooms in on a single decode step at the matrix level and really stares at why one new token means one new row, and why the GPU ends up so underutilized doing it. After that, Post 5.4 derives the KV cache by asking what's redundant across those steps.

Additional reading (and watching)

• Dao, T., et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS 2022.

• Agrawal, A., et al. (2023). Sarathi: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills.

• Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023.

• Yu, G., et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022.

• Patel, P., et al. (2024). Splitwise: Efficient Generative LLM Inference Using Phase Splitting. ISCA 2024. Splits prefill and decode onto separate machine pools and quantifies the savings.

• Zheng, L., et al. (2023). Efficiently Programming Large Language Models using SGLang. Introduces RadixAttention and prefix caching as first-class scheduling primitives, building directly on the prefill/decode asymmetry.