Prefix Caching and Prompt Reuse

Requests to an LLM-backed service often share a system prompt. It might contain a few hundred tokens or, with more extensive context engineering, thousands. Without prefix caching, each request repeats the prefill work for those tokens.

For a fixed model and inference configuration, causal attention lets us reuse the state of an unchanged prefix. If request A and request B both start with the same 1,000 tokens, the key-value cache computed for those tokens is mathematically equivalent. Cache keys must also distinguish model versions, adapters, position settings, and relevant non-text inputs. Prefix caching reuses those entries across requests.

Prerequisites

You'll want to be comfortable with two earlier posts before this one. The KV cache from first principles post covers what keys and values actually are and why we store them. The prefill vs. decode post explains the two-phase structure of inference and why prefill contributes to initial latency.

A quick refresher on prefill

When a request hits the model, inference happens in two phases. Prefill processes the entire input prompt in parallel, computing key and value vectors for every token across every layer. These get stored in the KV cache. Decode then generates output tokens one at a time, each new token attending to all the cached keys and values.

Prefill can be a substantial part of TTFT, especially for long prompts. It processes many positions in parallel. For a 1,000-token system prompt, that's 1,000 tokens' worth of attention and FFN computation across every layer of the model.

So if 100 requests all share the same 1,000-token system prompt, that's 100,000 tokens of prefill work. The same work, repeated 100 times.

Just cache the prefix

After computing the KV cache for a token sequence, hash the tokens and store the KV entries. When the next request comes in, check if its token prefix matches a stored hash. If it does, skip the prefill for that prefix entirely and load the cached KV vectors directly into GPU memory.

The transformer computes keys and values as linear projections of hidden states:

With the same model, adapter, positions, and inference settings, the hidden state $h_i$ depends on the token at position $i$ and, through causal attention, the tokens before it. Matching tokens 0 through 500 therefore gives reusable KV state through position 500.

This is why prefix caching works at the level of contiguous prefixes specifically. You can't skip to some chunk in the middle of the prompt and reuse it, because the hidden states at that point depend on everything that came before. The match has to be a walk from the start: compare token 1, then token 2, then token 3, and stop at the first one that differs.

What the savings actually look like

Consider a 1,000-token system prompt and three requests with different user messages.

For a larger example, consider a chatbot handling 100 requests per minute. The system prompt is 1,000 tokens. Each user message averages about 150 tokens.

Without prefix caching: 100 requests $\times$ 1,150 tokens = 115,000 tokens of prefill per minute.

With prefix caching: 1,000 shared tokens + 100 $\times$ 150 unique tokens = 16,000 tokens of prefill per minute.

That removes 86.1% of freshly processed input positions. FLOP savings are smaller because fresh suffix queries still attend over the cached prefix. It can also reduce TTFT when prefill accounts for much of the wait.

RadixAttention: caching gets a data structure

The simple approach (hash the full prefix, look it up in a flat table) works fine when every request has exactly the same system prompt. But in practice, prefixes branch. Think about multi-turn conversations: turn 1 and turn 2 share the system prompt plus the first exchange, but diverge at the second user message. Or think about a service that has slightly different system prompts for different features, but they share a common header.

Zheng et al. addressed this in 2023 with RadixAttention, the caching system behind SGLang. Instead of a flat hash map, they organize cached KV blocks in a radix tree (also called a prefix tree or trie). Each node in the tree corresponds to a chunk of tokens, and edges connect chunks that appear consecutively.

When a new request arrives, the system walks the tree from the root, matching tokens greedily until the input diverges from any stored path. The longest match becomes the reusable cache. The rest gets computed fresh and inserted into the tree as a new branch.

The radix tree stores shared prefixes at branch points. If you have 50 requests that share tokens 0 through 1,000, then branch into 5 different conversation histories, the tree stores the common prefix once and branches into 5 subtrees. Each subtree itself can share sub-prefixes with other requests.

vLLM implements a similar idea called automatic prefix caching (APC), where the system hashes token blocks and deduplicates them transparently. When enabled and supported by the model/backend, APC finds reusable token blocks. Its configuration and defaults depend on the engine version.

API-level prompt caching

Hosted APIs can reuse prompt prefixes too, but their billing and retention contracts differ. A cache hit requires matching eligible content and the provider's cache policy, not just sending similar prose.

• Anthropic supports automatic caching and explicit breakpoints through cache_control, with model-dependent minimum lengths and selectable retention options. Cache writes and reads have different prices.
• OpenAI supports model-dependent implicit and explicit caching behavior. Minimum lengths, boundaries, retention controls, and read/write pricing vary by model generation.
• Gemini offers implicit and explicit caching, with eligibility, storage charges, and expiry rules described in its caching documentation.

Use each provider's usage fields to check what actually hit. A repeated prefix does not guarantee a hit: it may have expired, been evicted, routed elsewhere, or changed in the serialized request.

For a prefix of $S$ tokens used in $N$ requests with exactly one cache write and $N-1$ hits:

Here the rates are per token; add uncached suffix tokens, output tokens, storage charges where applicable, and additional writes after misses. For example, 4,000 prefix tokens used 10,000 times yield 4,000 written tokens and 39,996,000 read tokens under those assumptions. Multiply by the selected model's actual rates rather than treating a provider-wide discount as permanent.

Cache eviction: memory is finite

KV caches are big. For a 70B parameter model with 80 layers and 8 KV heads (GQA, like Llama 3 70B), the KV cache for 1,000 tokens in FP16 is roughly:

(The 2 at the front is for K and V, 128 is the head dimension, and the 2 at the end is bytes per FP16 value.) At older MHA ratios with 64 KV heads, the same cache would be 8x larger, roughly 2.6 GB. Keeping multiple distinct prefixes resident multiplies this memory requirement.

You can't keep every prefix cached forever. GPU memory fills up. So engines use eviction policies.

LRU (Least Recently Used) is one common policy. The prefix that hasn't been hit in the longest time gets evicted first. This works well when traffic patterns are bursty, because popular prefixes stay warm.

Frequency-based eviction considers how often a prefix has been reused. A system prompt that gets 1,000 hits per minute should survive eviction over a prefix that was hit twice and then forgotten.

TTL (Time-To-Live) limits how long an entry remains eligible for reuse. Providers differ in retention options and whether hits refresh the lifetime. Expiry or memory pressure can make a later request cold again.

In self-hosted engines like vLLM and SGLang, the eviction happens at the block level. The radix tree nodes that haven't been accessed get pruned, freeing up GPU memory blocks for new requests. It's the same memory management that PagedAttention uses for per-request KV caches, just applied to the shared prefix store.

Multi-turn conversations

A conversation extends its cacheable prefix at each turn:

• Turn 1: System prompt + user message 1 generates response 1
• Turn 2: System prompt + user message 1 + response 1 + user message 2
• Turn 3: Everything from turn 2, plus response 2 and user message 3

Each turn extends the prefix. Without caching, turn 3 re-prefills the entire conversation history from scratch, including the system prompt and the first exchange that haven't changed since turn 1.

A simple implementation sketch

The sketch below looks up the longest cached prefix, reuses its K/V, and prefills the remaining suffix.

This toy uses exact tuple keys and copies tensors, so it is deliberately inefficient. Production engines use structures such as radix trees or chained block hashes, reference counts, and copy-on-write. Keys must include the preceding prefix, model/configuration identity, and relevant multimodal or adapter inputs; a block's local tokens alone are insufficient.

Practical design implications

Put stable content first. Within the required message roles and order, place shared instructions, tool schemas, and examples before variable content. That gives requests a longer matching prefix without changing which instructions govern them.

Keep system prompts consistent. An edit stops reuse at the first changed token. Inserting an A/B-test variant late in a prompt preserves more of the shared prefix than inserting it early, provided that placement still expresses the intended instructions.

Evaluate added context. Caching can reduce repeated prefill and input billing, but extra tokens still occupy cache memory and participate in decode attention. Keep them when they improve the task, and measure hit rates and quality.

Batch similar requests. If your service handles different types of requests with different system prompts, route similar request types to the same inference workers. This keeps the cache warm for each prompt variant rather than thrashing between them.

For multi-turn, append rather than reformat. Rebuilding a request from scratch can still hit the cache if its serialized token IDs match. Reordering or reformatting earlier messages can change those IDs and shorten the reusable prefix.

Misconceptions

"Prefix caching only helps if prompts are identical." Partial matches work. RadixAttention and vLLM's automatic prefix caching find the longest shared prefix, even if the requests diverge after that point. Two prompts sharing 800 of 1,000 tokens can reuse that prefix if compatible cache entries remain resident. Block-based engines may round the reusable portion down to complete blocks.

"The cache stores the text." The cache stores KV tensors, the computed intermediate values (key and value projections at every layer). The text is just used to compute the hash for lookup. The heavy objects being reused are the GPU tensors.

"This only matters for cost, not latency." Skipping 1,000 tokens of prefill removes that computation from the wait for the first token. It also frees compute for other requests. The latency saving depends on how much of the wait was spent on prefill rather than queuing or other work.

What's next

The next post covers structured generation, how inference engines constrain token sampling to guarantee valid JSON, regex patterns, or grammar-compliant output.

Additional reading (and watching)

• Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. vLLM's automatic prefix caching builds on PagedAttention.

• Anthropic. (2024). Prompt Caching documentation. Cache controls, eligibility, write/read pricing and retention options.

• OpenAI. (2024). Prompt Caching documentation. Model-dependent cache controls, retention and billing.

• Google. (2024). Context Caching for Gemini. Implicit caching with reduced pricing for repeated prefixes.

• Zheng, L., et al. (2023). SGLang: Efficient Execution of Structured Language Model Programs. RadixAttention and reuse of shared prefixes.

• vLLM Project. (2024). Automatic Prefix Caching. Benchmarks showing throughput gains with shared prefixes.