Prefix Caching and Prompt Reuse

Look at the token accounting on a typical LLM-backed service for a few minutes and you start to notice something. The vast majority of tokens are input tokens, not output tokens. Every request begins with the same system prompt. Sometimes it's a few hundred tokens. Sometimes (if you've been doing your context engineering) it's thousands. And every single request pays the full prefill cost for that prompt, over and over again.

Here's the thing. The transformer is deterministic. If request A and request B both start with the same 1,000 tokens, the key-value cache computed for those tokens is bit-for-bit identical. The same input always produces the same intermediate states. So why are we recomputing it every time?

That's the core idea behind prefix caching.

Prerequisites

You'll want to be comfortable with two earlier posts before this one. The KV cache from first principles post covers what keys and values actually are and why we store them. The prefill vs. decode post explains the two-phase structure of inference and why prefill is the expensive part. I'll recap the essentials below, but those posts have the full picture.

A quick refresher on prefill

When a request hits the model, inference happens in two phases. Prefill processes the entire input prompt in parallel, computing key and value vectors for every token across every layer. These get stored in the KV cache. Decode then generates output tokens one at a time, each new token attending to all the cached keys and values.

Prefill is the expensive part. It's compute-bound, meaning the GPU is doing a lot of matrix multiplications per token. For a 1,000-token system prompt, that's 1,000 tokens' worth of attention and FFN computation across every layer of the model. And it's the primary driver of time-to-first-token (TTFT), which is usually what the user is waiting on.

So if 100 requests all share the same 1,000-token system prompt, that's 100,000 tokens of prefill work. The same work, repeated 100 times.

Just cache the prefix

The fix is pretty simple. After computing the KV cache for a token sequence, hash the tokens and store the KV entries. When the next request comes in, check if its token prefix matches a stored hash. If it does, skip the prefill for that prefix entirely and load the cached KV vectors directly into GPU memory.

I want to be precise about what "identical" means here. The transformer computes key and value vectors as linear projections of the hidden state:

The hidden state $h_i$ at position $i$ depends on the token at that position and (through causal attention) all tokens before it. So the KV pair at position 500 depends on tokens 0 through 500. If those are the same across two requests, the KV vectors are the same. There's no randomness in the forward pass (dropout is off at inference time), and no stochastic rounding that would break this. Bit-for-bit identical.

This is why prefix caching works at the level of contiguous prefixes specifically. You can't skip to some chunk in the middle of the prompt and reuse it, because the hidden states at that point depend on everything that came before. The match has to be a walk from the start: compare token 1, then token 2, then token 3, and stop at the first one that differs.

What the savings actually look like

Let me make this concrete. Say you have a 1,000-token system prompt and three requests with different user messages.

Without caching, each request pays for the full prefill. With caching, the first request computes and stores the KV cache for the system prompt. Requests 2 and 3 load that cached KV and only prefill their unique suffixes.

Now let me scale this up. Say you're running a chatbot that handles 100 requests per minute. The system prompt is 1,000 tokens. Each user message averages about 150 tokens.

Without prefix caching: 100 requests $\times$ 1,150 tokens = 115,000 tokens of prefill per minute.

With prefix caching: 1,000 tokens (first request computes the full prompt) + 99 $\times$ 150 tokens (subsequent requests only prefill user messages) = 15,850 tokens of prefill per minute.

That's an 86% reduction in prefill compute. And because prefill is the bottleneck for TTFT, your users feel it too. The second request onward gets its first token much faster because the model jumps straight to processing the short user message.

RadixAttention: caching gets a data structure

The simple approach (hash the full prefix, look it up in a flat table) works fine when every request has exactly the same system prompt. But in practice, prefixes branch. Think about multi-turn conversations: turn 1 and turn 2 share the system prompt plus the first exchange, but diverge at the second user message. Or think about a service that has slightly different system prompts for different features, but they share a common header.

Zheng et al. addressed this in 2023 with RadixAttention, the caching system behind SGLang. Instead of a flat hash map, they organize cached KV blocks in a radix tree (also called a prefix tree or trie). Each node in the tree corresponds to a chunk of tokens, and edges connect chunks that appear consecutively.

When a new request arrives, the system walks the tree from the root, matching tokens greedily until the input diverges from any stored path. The longest match becomes the reusable cache. The rest gets computed fresh and inserted into the tree as a new branch.

The radix tree framing is elegant because it handles the branching case naturally. If you have 50 requests that share tokens 0 through 1,000, then branch into 5 different conversation histories, the tree stores the common prefix once and branches into 5 subtrees. Each subtree itself can share sub-prefixes with other requests. The reuse is maximally granular without any manual configuration.

vLLM implements a similar idea called automatic prefix caching (APC), where the system hashes token blocks and deduplicates them transparently. The user doesn't configure anything. The engine just notices prefix overlap and reuses what it can.

API-level prompt caching

If you're calling models through an API rather than self-hosting, the providers handle prefix caching on their end. And they pass the savings on to you (partially, at least).

Anthropic introduced prompt caching in mid-2024. When consecutive API calls share the same prefix, Anthropic caches the KV states server-side. Cached input tokens are billed at a 90% discount: you pay $0.30 per million tokens instead of$3.00 per million for Claude Sonnet. The cache has a 5-minute TTL that resets on each hit. You don't opt in or configure anything. It just works, as long as your requests share a prefix and arrive within the TTL window.

OpenAI launched a similar feature in late 2024. Cached input tokens get a 50% discount. Like Anthropic, it's automatic and server-side. The minimum cacheable prefix is 1,024 tokens, and the cache deduplicates at 128-token block boundaries.

Google followed with implicit caching on Gemini models, offering a similar discount structure for repeated prefixes.

Let me do the math on one concrete scenario. Say you're running a coding assistant with a 4,000-token system prompt. You handle 10,000 requests per day.

That 90% discount on cached tokens means Anthropic's prompt caching saves you about 90% on the system prompt portion of your bill. OpenAI's 50% discount is still meaningful but less dramatic. And this is just the system prompt. If you're doing multi-turn conversations (more on that below), the savings compound.

The pricing structure also tells you something about what these companies want you to do. They want you to write longer, richer system prompts. The cache makes it cheap, and longer system prompts mean better model behavior. The incentives are aligned.

Cache eviction: memory is finite

KV caches are big. For a 70B parameter model with 80 layers and 8 KV heads (GQA, like Llama 3 70B), the KV cache for 1,000 tokens in FP16 is roughly:

(The 2 at the front is for K and V, 128 is the head dimension, and the 2 at the end is bytes per FP16 value.) At older MHA ratios with 64 KV heads, the same cache would be 8x larger, roughly 2.6 GB. Either way, multiply by the number of concurrent prefixes you want to keep warm and the number gets scary fast.

You can't keep every prefix cached forever. GPU memory fills up. So engines use eviction policies.

LRU (Least Recently Used) is the default in most systems. The prefix that hasn't been hit in the longest time gets evicted first. This works well when traffic patterns are bursty, because popular prefixes stay warm.

Frequency-based eviction considers how often a prefix has been reused. A system prompt that gets 1,000 hits per minute should survive eviction over a prefix that was hit twice and then forgotten.

TTL (Time-To-Live) is what the API providers use. Anthropic's 5-minute TTL is simple and predictable: if no request hits the cached prefix within 5 minutes, it gets evicted. This means your application needs to maintain a steady request rate to keep the cache warm. For high-traffic services this is trivial. For low-traffic services, you might occasionally pay a "cold start" penalty when the cache expires.

In self-hosted engines like vLLM and SGLang, the eviction happens at the block level. The radix tree nodes that haven't been accessed get pruned, freeing up GPU memory blocks for new requests. It's the same memory management that PagedAttention uses for per-request KV caches, just applied to the shared prefix store.

Multi-turn conversations

Multi-turn chat is where prefix caching really shines. Think about what happens in a conversation:

• Turn 1: System prompt + user message 1 generates response 1
• Turn 2: System prompt + user message 1 + response 1 + user message 2
• Turn 3: Everything from turn 2, plus response 2 and user message 3

Each turn extends the prefix. Without caching, turn 3 re-prefills the entire conversation history from scratch, including the system prompt and the first exchange that haven't changed since turn 1.

With caching, each turn only needs to prefill the new content. The system prompt is cached from turn 1. The first exchange is cached from turn 2. By turn 5 of a conversation, you might have 3,000 tokens of cached context and only need to prefill the latest 100-token user message.

This is also why the radix tree structure matters for multi-turn. A conversation branches every time the user says something new. If user A and user B both had the same first two turns but diverge at turn 3, the tree stores the shared history once and branches at the divergence point. Both conversations benefit from caching up to their shared prefix.

A simple implementation sketch

The core logic is surprisingly compact. Hash the token prefix, check if you've seen it before, and either load the cached KV or compute and store it.

In practice, production systems like SGLang don't hash the full prefix as a single key. They break the sequence into fixed-size blocks (typically 16 or 64 tokens) and hash each block. This allows partial prefix matches at block boundaries and makes eviction more granular. But the intuition is the same.

Practical design implications

Once you know prefix caching exists, it changes how you structure your prompts.

Put stable content first. Your system prompt should be the prefix. Instructions, persona definitions, tool schemas, few-shot examples. All the stuff that doesn't change between requests. The variable content (the user's message, the retrieved context) goes at the end. This maximizes the cacheable prefix length.

Keep system prompts consistent. Even tiny changes break the cache. If you A/B test by inserting a sentence into the middle of your system prompt, that creates two separate cache entries and halves your hit rate. If you need to A/B test, vary the suffix, not the prefix.

Longer system prompts are cheaper than you think. With Anthropic's 90% cached discount, adding 2,000 tokens to your system prompt costs $0.60 per million requests at the cached rate instead of$6.00. If those extra tokens improve model quality (and they usually do), the ROI is enormous.

Batch similar requests. If your service handles different types of requests with different system prompts, route similar request types to the same inference workers. This keeps the cache warm for each prompt variant rather than thrashing between them.

For multi-turn, append rather than reformat. Some applications rebuild the full prompt from scratch each turn, sometimes reordering or reformatting prior messages. This destroys cacheability. Instead, append new content to the existing prefix so the cache can match the conversation history incrementally.

Misconceptions

"Prefix caching only helps if prompts are identical." Partial matches work. RadixAttention and vLLM's automatic prefix caching find the longest shared prefix, even if the requests diverge after that point. Two prompts that share 800 out of 1,000 tokens still get 800 tokens of cache reuse.

"The cache stores the text." The cache stores KV tensors, the computed intermediate values (key and value projections at every layer). The text is just used to compute the hash for lookup. The heavy objects being reused are the GPU tensors.

"This only matters for cost, not latency." It directly reduces TTFT. Prefill is the bottleneck for time-to-first-token. If you skip 1,000 tokens of prefill computation, the user sees the first token sooner. At high load, the latency savings can be dramatic because you're also freeing up GPU compute cycles for other requests.

What's next

This post covered prefix caching: the trick of noticing that the KV cache for a shared prompt is deterministic and bit-for-bit identical across requests, then reusing it. We walked through RadixAttention's tree structure, LRU eviction, the multi-turn case where each turn extends the prefix, and the design rules that come out of this (stable content first, append rather than reformat).

The next post covers structured generation, how inference engines constrain token sampling to guarantee valid JSON, regex patterns, or grammar-compliant output.

Additional reading (and watching)

• Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. vLLM's automatic prefix caching builds on PagedAttention.

• Anthropic. (2024). Prompt Caching documentation. 90% discount on cached input tokens, 5-minute TTL.

• OpenAI. (2024). Prompt Caching documentation. 50% discount, automatic server-side caching at 128-token boundaries.

• Google. (2024). Context Caching for Gemini. Implicit caching with reduced pricing for repeated prefixes.

• Anthropic. (2025). Claude Code documentation. Demonstrates prompt caching benefits in agentic coding workflows.

• vLLM Project. (2024). Automatic Prefix Caching. Benchmarks showing throughput gains with shared prefixes.