Memory Management: Fitting a Model and Deciding Concurrency

A serving engineer's job, stripped to the bone, is figuring out what fits on the GPU and how many people can use it at once. You have a chunk of HBM, a model, some number of concurrent users who want responses, and a wish to not run out of memory. Everything interesting about modern inference engines, vLLM's paged attention, TensorRT-LLM's engine builder, llama.cpp's mmap magic, is some version of trying to be clever about this one question.

I want to give you a mental model for it that holds up under pressure. Not "eh, roughly," but an actual thing you can pull out on a napkin when a teammate asks "can we fit Llama 3 70B on a pair of H100s and serve fifty people?" By the end of this post you should be able to answer that in about ninety seconds, and when the numbers don't work, know which lever to pull first.

We'll call the named handle for this the memory budget. Every byte on the GPU has a home, and there are only a handful of places it can live. Once you internalize those places, sizing becomes almost mechanical.

Where the bytes actually go

Open up nvidia-smi on a serving node and you'll see something like 78234 MiB / 81559 MiB. That's the card telling you that basically all of the HBM is in use. That's not waste. Engines are designed to fill the card, because unused HBM is wasted KV cache budget, which means fewer concurrent requests, which means lower throughput. A well-tuned server is always right up against the ceiling.

If you crack open what's actually in those bytes on a typical serving GPU, it's four things:

1. Model weights. The parameters. Fixed size once you pick a model and a precision. For Llama 3 70B in FP16: 140 GB. For Qwen 3 235B in INT4: about 117 GB. The weights are the rock the whole setup is built on.
2. CUDA context + library overhead. CUDA itself eats a few GB to hold its runtime context, cuBLAS/cuDNN handles, NCCL communicators if you're doing multi-GPU, workspace buffers. This is mostly a one-time tax, but it's real. Typically 1–4 GB per GPU.
3. Activation memory. Intermediate tensors during a forward pass. These exist briefly, get consumed by the next operator, and go away. In serving you usually see this as a peak — the biggest activation tensor that coexists with the others mid-layer. Bounded by batch size, hidden dim, and context.
4. KV cache. The per-request store of attention keys and values, so the model doesn't recompute them every decode step. (If you're coming in fresh: the KV cache post covers this from scratch.) KV is where all the dynamics live. Weights are fixed. Activations are bounded. KV grows with every token every request.

So the math of a GPU looks like:

The first three are effectively fixed once you pick a model, a batch shape, and a GPU. The KV cache is what's left. And the KV cache is where you trade off between "more concurrent users" and "longer context per user." Everything else is a decision you made before you started serving. KV is the dial you turn at runtime.

The memory budget, as a named handle

Let me give this a name so we can hold it. The memory budget for KV cache is:

Whatever's left after the first three is what you get to spend on KV. And KV directly determines how many token slots you have:

Those token slots are the raw material of concurrency. You can divide them up however you like:

Four users at 10k tokens. Sixteen users at 2.7k tokens. A hundred users at 400 tokens. Those are all the same engine, the same model, the same GPUs. Just different ways of slicing the same pie.

The calculator below is that formula, made interactive. Pick a model, a precision, some GPUs, a context length, and see the pie get sliced for you.

A few things to try in the calculator:

• Drop a 70B FP16 model onto a single H100 80 GB and watch it fail to fit. The weights alone are 140 GB. You need at least two GPUs, or you need to quantize.
• Flip that same 70B to INT4 and suddenly it fits on one H100 with room to spare. That's the quantization pitch in one click (and the next post walks through how).
• Crank context from 8k to 128k and watch max-concurrent collapse. KV scales linearly with context length. Long context is expensive.
• Swap KV precision to INT8 and concurrent capacity roughly doubles. This is why FP8 and INT8 KV are the two big cache-level levers of 2024–2026.

A quick minimum of math

I want to write down the one formula that does almost all the work, because once you see it you'll recognize it everywhere. The per-token KV cache cost, in bytes, for a single request, is:

Where $L$ is the number of transformer layers, $n_{\text{kv}}$ is the number of KV heads per layer, $d_{\text{head}}$ is the per-head dimension, and $b$ is the byte width of your storage format (2 for FP16, 1 for INT8). The factor of 2 out front is because we store both K and V.

Note what's not in that formula: hidden size, number of query heads, MLP width, vocab size, anything about the MLP stack. The KV cache only cares about the attention K/V projections and how many layers you pay the bill across. That's why GQA (grouped-query attention) and MQA (multi-query attention) matter so much for serving; they shrink $n_{\text{kv}}$ independently of $H_q$ and you get a proportional KV cache savings.

Here's the viz from the KV cache post as a refresher, because the mechanics of what's actually in those bytes helps:

The 70B model, GQA with 8 KV heads, 80 layers, head dim 128, FP16:

Three hundred and twenty kilobytes per token, per request. Tuck that number away; we're about to use it.

The worked example: Llama 3 70B on 2× H100 80GB

Okay, let's actually do the napkin calc. The classic 2026 question: can we serve Llama 3 70B with full FP16 weights across a pair of H100 80 GBs? No quantization yet, just the vanilla deployment.

Step 1. Total HBM. Two H100s, 80 GB each. That's 160 GB total.

Step 2. Weights. 70 billion parameters at 2 bytes each is 140 GB. Sharded across both GPUs via tensor parallelism means each card holds 70 GB of weights, but the total weight bill is 140 GB.

Step 3. CUDA + activation overhead. Call it 4 GB of CUDA context across both cards and 2 GB of peak activations. That's 6 GB off the top.

Step 4. What's left for KV. $160 - 140 - 6 = 14$ GB. Fourteen gigabytes of KV budget, distributed across the two cards.

Step 5. Token slots. At 327,680 bytes per token:

About forty-three thousand token-slots of KV cache.

Step 6. Slice the pie. Forty-two thousand sounds like a lot until you start dividing. Some real scenarios:

• Four concurrent requests at ~10,600 tokens each. (Chat with fairly long transcripts.)
• Sixteen concurrent at ~2,600 tokens each. (Short chat, Q&A.)
• A hundred concurrent at ~427 tokens each. (Only good for short turns.)

And these are upper bounds assuming every request fully uses its slot. In practice you won't pack the bar tight. You leave a few GB of KV slack for new requests arriving and for completion variance. Call it a 10–15% haircut. So for this deployment you'd plan around something like "20 concurrent users at 2k context" or "six users at 6k context." That's the kind of sentence you want to be able to say without looking anything up.

If you need more, your options are the usual suspects: quantize the weights (INT4 gets you weights down to 35 GB, which frees up another 105 GB for KV), add GPUs (cost/communication trade), or cap max context per request (the cheapest lever). The quantization lever is the big one, and it's the next post.

Go back to the calculator and dial this in. Llama 3 70B, FP16, H100 80 GB, 2× GPUs, 32k context, FP16 KV. You should see the math land close to what we just did by hand.

Implementation sketch

The entire thing is about thirty lines of Python. There's nothing clever in it; once you've named the variables the math is mostly arithmetic:

# --- Napkin memory budget for serving ---
 
def kv_cache_bytes_per_token(layers, kv_heads, head_dim, bytes_per_elem=2):
    # factor of 2 for K and V
    return 2 * layers * kv_heads * head_dim * bytes_per_elem
 
def serving_budget(
    gpu_total_gb, num_gpus, model_params_b, weight_bytes,
    cuda_overhead_gb, activation_peak_gb,
    layers, kv_heads, head_dim, kv_bytes_per_elem,
):
    total_gb = gpu_total_gb * num_gpus
    weights_gb = (model_params_b * 1e9 * weight_bytes) / 1e9
    kv_budget_gb = total_gb - weights_gb - cuda_overhead_gb - activation_peak_gb
    per_token_b = kv_cache_bytes_per_token(layers, kv_heads, head_dim, kv_bytes_per_elem)
    slots = max(0, int((kv_budget_gb * 1e9) / per_token_b))
    return weights_gb, kv_budget_gb, per_token_b / 1024, slots
 
# Llama 3 70B, FP16, 2x H100 80GB
w, free, per_kb, slots = serving_budget(
    gpu_total_gb=80, num_gpus=2, model_params_b=70, weight_bytes=2,
    cuda_overhead_gb=4, activation_peak_gb=2,
    layers=80, kv_heads=8, head_dim=128, kv_bytes_per_elem=2,
)
print(f"Weights: {w:.1f} GB | Free for KV: {free:.1f} GB | "
      f"KV/token: {per_kb:.1f} KB | Total token slots: {slots:,}")
# Weights: 140.0 GB | Free for KV: 14.0 GB | KV/token: 320.0 KB | Total token slots: 42,724

The whole calculator in the widget above, the whole mental model in this post, everything an engine like vLLM does at startup to decide its KV pool size — it's all this snippet with better ergonomics and a few edge cases around reserved workspace.

Why KV is the main scaling dial

Once you've picked a model and a precision, the only thing that changes between "serve 2 users" and "serve 200 users" is the KV cache. Weights are fixed. Activation peak is basically fixed (it depends on max per-request tokens-in-flight, but it doesn't balloon like KV does). Everything that grows with concurrent users lives in KV.

So when you're tuning an engine, you're almost always really tuning the shape of the KV cache. Continuous batching stuffs more requests into the same KV pool. PagedAttention reclaims KV fragmentation. Prefix caching shares KV across requests with a common prompt. FP8 and INT8 KV shrink bytes-per-token so more tokens fit. Chunked prefill smooths activation+KV peaks so you can pack the pool tighter. Every one of these is a KV-centric trick.

And KV scales brutally with context length. Here's what that looks like visually:

The 80 GB reference line is where the whole weight of a single H100 would be. When the KV line crosses it, you are in "one request consumes an entire GPU" territory. That's the regime where you stop thinking about concurrency entirely and start thinking about sharding KV across GPUs, offloading, or telling the user to use less context.

This is also why context extensions like RoPE scaling and YaRN are attractive from a training-cost perspective but get expensive at inference: you stretched the model's usable context but the KV bill didn't go away. It grew linearly.

When you have to leave the GPU

Sometimes your model is just bigger than your HBM, and no amount of quantization rescues you on the hardware you have. The usual example: running a 70B locally on a consumer card with 24 GB of VRAM. You have two choices. Give up, or offload.

Offloading means stashing some of the weights (or KV cache, or activations) on a slower tier of memory. CPU DRAM is closer, at around 32 GB/s over PCIe 4.0. NVMe SSDs are farther, at 3–7 GB/s. GPU HBM, for reference, is 3,000+ GB/s on an H100. Every time the forward pass needs a weight block that's not in HBM, it pays the transit time to go grab it. And it has to grab it on every token.

llama.cpp's mmap-based loading is the most elegant version of this trick. It lets the OS page weights in and out transparently, and with enough DRAM buffer, a 70B runs on a laptop. FlexGen made the academic case: even NVMe-resident weights can serve throughput-oriented workloads if you're clever about scheduling. But for production latency-sensitive serving, offload is a last resort. The throughput collapse is too steep.

Misconceptions

"More GPU memory always means more concurrent requests." Only up to a point. Once you've saturated the model's compute (attention becomes the bottleneck, or the matmul units are full), adding memory buys you longer context per request, not more requests in parallel. Memory is necessary for concurrency; it's not sufficient. The continuous batching post goes deeper on where the compute ceiling kicks in.

"Quantizing weights is the only way to fit a bigger model." It's the cheapest, yes, and usually the first move. But there are other levers. KV cache quantization (FP8, INT8) cuts the KV bill in half. Grouped-query attention cuts KV by 4–8×. Shorter max context cuts KV by however much you cap it. If you treat weight quantization as the only dial, you're missing a lot of headroom.

"Activation memory is negligible." It's bounded, which is different from negligible. In long-context serving without activation recomputation, peak activations can balloon into double-digit GB for a single layer-worth of attention scores (before you have Flash-Attention-style streaming). Chunked prefill was invented in part to smooth this peak so more KV fits underneath.

"You can just keep adding GPUs." GPUs talk over NVLink or InfiniBand, both of which are orders of magnitude slower than HBM. Every extra GPU adds communication cost to every forward pass, and past a point you spend more on collective ops than you save in memory pressure. The parallelism post covers where the trade-off tips.

What's next

The cheapest single lever for fitting more model onto the same GPU is shrinking each weight to fewer bits. Every time the calculator above ran into "doesn't fit," your first instinct should have been to click INT8 or INT4 and watch it light up green. The next post walks through exactly how that works without the model getting dumber. Start with Quantization (INT8, INT4, GPTQ, AWQ, GGUF).

Additional reading (and watching)

• Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. The paper that made the KV cache sizing problem concrete for the serving community, and the origin of vLLM's approach to treating KV like paged virtual memory.

• Ainslie, J., et al. (2023). GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. EMNLP 2023. The grouped-query attention paper. Shrinking $n_{\text{kv}}$ independently of $H_q$ is where most of the modern KV savings come from; every modern open-weights model uses it.

• Dubey, A., et al. (2024). The Llama 3 Herd of Models. arXiv:2407.21783. Source for the actual Llama 3 70B architecture numbers used in the worked example (80 layers, 8 KV heads, 128 head dim).

• vLLM project. Memory Management docs. The gpu_memory_utilization flag is the user-facing handle for the formula in this post. Reading the engine's startup logs is a good sanity-check on whether you've picked reasonable numbers.

• Pope, R., et al. (2022). Efficiently Scaling Transformer Inference. arXiv:2211.05102. The PaLM serving paper; still the cleanest analytical treatment of how batch, context, and hardware interact, and where KV-centric optimizations come from.

• Gerganov, G. (2023). llama.cpp. The reference implementation for local LLM inference. Its mmap-based loader is the cleanest example of offload-as-default-mode I know.

• Sheng, Y., et al. (2023). FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU. ICML 2023. The paper that pushed NVMe-resident inference as a real option for offline, throughput-oriented workloads.

• vLLM project. vLLM GitHub. The engine's startup routine is the practical companion to Kwon et al.; the memory budget discovery happens in the LLMEngine init, worth reading alongside the paper.

• NVIDIA. TensorRT-LLM. Engine builder docs describe the max_batch_size / max_input_len / max_output_len contract; that's your compile-time memory budget, frozen into the engine.

• Shoeybi, M., et al. (2019). Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. arXiv:1909.08053. The early paper on sharding a model across GPUs when it doesn't fit on one. The framing here of "weights > one GPU → tensor parallelism" traces back to this work.