What an Inference Engine Actually Does

Where does the model actually live?

Not the architecture, and not the weights sitting on Hugging Face. I mean, when a user hits POST /v1/chat/completions on your API endpoint, what is the thing on the other side of that socket, receiving the request, turning it into tokens, running the forward passes, and streaming the response back? That thing is not the model. The model is a big pile of numbers. The thing doing the work is what people call an inference engine, and it's the layer that this entire arc is about.

If Arcs 4 and 5 were about how a transformer processes a sequence and how the decode loop produces tokens, Arc 6 is about everything that has to exist to turn those capabilities into a system you can actually serve traffic from. This post is the map. Before we dive into the individual tradeoffs (kernel fusion, paged attention, quantization, parallelism, scheduling), I want to lay out the whole block diagram. When you see "vLLM uses PagedAttention" or "TensorRT-LLM compiles the graph", I want you to know exactly which slot that statement is about.

The problem the engine solves

Imagine you have a trained model. It's a file on disk, maybe 140 GB of fp16 weights for a 70B parameter Llama. You can load those weights into memory in PyTorch and call model.generate(prompt) and get back a response. Does that count as "serving the model"?

Sort of. It's a research loop: a single-user, single-request, single-GPU, no-concurrency, no-metrics, no-admission-control, no-fault-tolerance script that generates one answer and exits. If you wrap it in a Flask endpoint, you have made a demo, not a production system. Once you point real traffic at it, you discover the problems:

• Requests arrive at different times, but PyTorch wants to process a batch.
• Some prompts are 20 tokens, some are 4,000. Some responses are 10 tokens, some are 2,000. Which ones run together?
• The GPU has to sit idle while one slow request finishes a long generation, because you batched it with shorter ones.
• Each request needs its own KV cache, and the cache can easily be bigger than the model weights.
• When the GPU runs out of memory mid-decode, what happens? Does the whole batch crash?
• How do you stream the first token back to the user 200 ms after they hit enter, not 8 seconds later when the whole thing finishes?
• How do you tokenize a chat with a system prompt, history, and tool calls? Each model family has its own template.
• How do you detokenize when a token is two bytes of a UTF-8 character and you haven't gotten the other two yet?

Every one of these problems has a name and a home. The place they all live is the inference engine. You can think of it as the operating system for a model: it owns everything between the weights and the wire. Model loading, tokenization, admission, scheduling, batching, KV cache management, kernel dispatch, detokenization, streaming. The model computes, and the engine orchestrates everything around it.

A quick refresher on the vocabulary

Arc 5 established a few terms that I'll lean on throughout Arc 6, and since not everyone reads in order, here are the one-sentence versions:

• Prefill is the forward pass over the entire prompt. It's embarrassingly parallel because every prompt token is known up front.
• Decode is the part where the model emits one new token per forward pass, in a loop, until it hits EOS or a length cap. Each step is expensive relative to how much useful work it does.
• KV cache is the per-request store of attention keys and values, kept so we don't recompute them every decode step. It grows linearly with sequence length.
• TTFT (time-to-first-token) is how long the user waits before the first token comes back, dominated by prefill. ITL (inter-token latency) is the time between subsequent tokens, dominated by decode step cost.
• Continuous batching is the scheduling trick that lets new requests hop into an already-running batch the moment an old one finishes, instead of waiting for the whole batch to drain.

If any of those feel shaky, the prefill vs decode post and the KV cache post go deep on the first three. But you shouldn't need them to follow this one.

The engine is the OS for a model

Here's the mental model I want you to leave this post with. The engine is a labeled box with subsystems inside it. Requests go in one side, tokens come out the other, and the GPU sits underneath feeding on whatever the engine gives it. Everything you're going to learn in the rest of Arc 6 is an optimization of one of these subsystems.

I want to walk through each box quickly, because naming the parts is half the battle.

Model loader + weights. When the engine starts up, it reads the model file from disk, deserializes the weights, decides on a precision (fp16, bf16, int8, int4), and shards the tensors across however many GPUs it was told to use. The choice of file format (safetensors, GGUF, TensorRT plan) matters here. So does the sharding strategy (tensor parallel, pipeline parallel, expert parallel, or some combination). This is the slot for post 6.5 on parallelism and post 6.3 on quantization.

Tokenizer. The incoming HTTP body contains text. The model wants integer IDs. The tokenizer is the thing that turns one into the other, and it is usually a BPE, SentencePiece, or WordPiece implementation with the model-specific merge rules. This is boring until you discover that your tokenizer is pure Python and your throughput is tokenizer-bound at 20,000 QPS, at which point you switch to the Rust-based tokenizer and forget about it.

Request queue and admission control. When a request arrives, something has to decide: do we accept it? If yes, do we run it now, or queue it? If the queue is full or the KV cache is nearly exhausted, do we reject with a 429 or block the caller? Different engines make very different choices here. vLLM will preempt in-flight requests to admit higher-priority ones. Others won't.

Scheduler. Every decode step, the engine has to pick which requests go in the next forward pass. With only one request this is boring. With a hundred, it's everything. The scheduler is where continuous batching, chunked prefill, priority classes, and disaggregated prefill/decode all live. This is the subject of post 6.7 on scheduling.

Batcher. Once the scheduler has decided who runs, the batcher assembles the actual input tensors: concatenating prompts of different lengths, building attention masks, zero-padding where needed, stitching together the KV cache handles for each request. It's the plumbing between "who's in this step" and "what shape tensor does the model see".

KV cache manager. Each active request has its own KV cache, and the total KV memory can easily exceed the weight memory. Someone has to allocate it, grow it token by token, evict it when the request finishes, defragment when it gets swiss-cheesed, and reuse shared prefixes across requests. Modern engines (vLLM, TGI, SGLang) do this with a paged allocator modeled after OS virtual memory. This is post 6.2 on paged attention and post 6.7 on prefix caching.

Kernel dispatcher. The actual math (matmuls, softmaxes, layer norms, attention) has to run on the GPU, and exactly how matters a lot. Do you call cuBLAS for each matmul individually? Fuse adjacent operations into one kernel? Compile the whole graph ahead of time into a TensorRT plan? Use FlashAttention v3 for attention and something custom for the output projection? This is post 6.1 on kernel fusion.

Detokenizer and streaming sink. Once the model emits a token ID, you need the text. Usually you stream it to the client as it's produced, not wait for the full response. Streaming means SSE (Server-Sent Events) in most OpenAI-compatible APIs, or websockets, or buffered JSON if streaming is off. And the detokenizer has to handle the case where a single character spans multiple tokens (UTF-8 continuation bytes), buffering partial bytes until a complete glyph emerges.

That's the engine. Weights + nine subsystems + a GPU. When someone says "vLLM does X better than TGI", they are almost always talking about one of those nine subsystems.

From HTTP to tokens: the concrete path

Let me walk a single request through those subsystems in order, because the sequence is what makes the block diagram feel real.

1. A POST request arrives at the engine's HTTP server. It contains a JSON body with a messages array, a max_tokens value, a temperature, maybe a tools array.
2. The engine applies the model's chat template to the messages, producing a flat string. The template is usually a Jinja2 snippet baked into the model's tokenizer config.
3. The tokenizer turns that string into a list of integer token IDs. Maybe 1,400 of them for a typical chat with some history.
4. The admission layer checks: is there room in the KV cache for 1,400 prompt tokens plus max_tokens of generation? Is the queue length under its SLO cap? If yes, accept. If no, queue or reject.
5. The request enters the scheduler. On the next forward-pass opportunity, it joins a batch along with any other admitted requests. If the engine supports chunked prefill, its 1,400-token prefill will be split into manageable chunks and interleaved with ongoing decodes from other requests. Otherwise it blocks decodes until it's done.
6. The prefill happens. The model runs a forward pass over all 1,400 prompt tokens in parallel, producing keys, values, and one set of final-layer logits. The KV cache for this request now holds 1,400 positions across every layer and every head. TTFT ticks.
7. The sampler picks the first generated token from the logits. A common path is softmax-with-temperature then sample, but post 5.7 covers the full menu.
8. The engine enters the decode loop. Every step, it runs one forward pass over one new token per request in the batch (its query), attends over that request's cached keys and values, samples a next token, and appends the new K/V row to the cache. The freshly-sampled token ID goes through the detokenizer.
9. The detokenizer yields a string fragment (or buffers if we're mid-UTF-8). The streaming sink ships it out over SSE as a chunk of the HTTP response body.
10. When the request emits an EOS token or hits max_tokens, the engine finalizes the response, frees the KV cache slots, and emits a [DONE] marker over SSE. Its batch slot is now available for the next queued request.

That's the shape of every request, and it's the shape every engine implements. The interesting question is how. What choices does each engine make at each stage? That's what Arc 6 is about, and it's a big menu:

A little math about what the engine manages

I don't want to pile on math, but there's one quick calculation that shows why the engine exists at all.

Suppose you want to serve Llama 3 70B (80 layers, 8 KV heads with GQA, head dimension 128) at 32k context in fp16. From the KV cache formula (Post 5.4):

Plug in: $2 \times 80 \times 8 \times 128 \times 32{,}768 \times 2 \approx 10.7\ \text{GB}$.

Read that as: two bytes per fp16 number, times 80 layers of K and V, times the heads and dimensions per layer, times 32k positions you might fill. The "2×" out front is for K and V.

Per request. The model itself is about 140 GB in fp16. On an 8xH100 node (640 GB total), you've got maybe 500 GB left after weights, which sounds like a lot, until you realize it's ~46 requests at max context. And those requests aren't arriving politely at the same length. Some finish after 200 tokens, some keep generating to 30,000. The engine is the thing that continually packs, unpacks, evicts, reuses, and shuffles those 46 slots so that the GPU is actually working on something useful every single cycle, instead of sitting on half-empty batches with stale cache.

Without the engine, you've rented $300k of hardware and are running it at 15% utilization. With a good engine, you're running it at 70–90%. That's the economic reason all this plumbing exists.

What a Python loop looks like next to a real engine

Concreteness helps. Here is the "naive" version of serving: a Python loop that loads a model, tokenizes, generates, and returns. This is the shape most tutorials show, and it's instructive to see what's missing from it.

# --- The naive version: what a research script does ---
from transformers import AutoModelForCausalLM, AutoTokenizer
 
tok = AutoTokenizer.from_pretrained("meta-llama/Llama-3-70b")
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3-70b", torch_dtype="float16", device_map="auto"
)
 
def answer(prompt: str, max_new_tokens: int = 512) -> str:
    input_ids = tok(prompt, return_tensors="pt").input_ids.cuda()
    out = model.generate(input_ids, max_new_tokens=max_new_tokens)
    return tok.decode(out[0], skip_special_tokens=True)
 
# Wrap in Flask, point traffic at it, watch it melt under 10 concurrent users.
 
# --- What a real engine (e.g. vLLM) does instead ---
from vllm import LLM, SamplingParams
 
# One-time setup: engine loads weights, shards across GPUs, preallocates
# a paged KV cache, spins up a scheduler thread.
engine = LLM(model="meta-llama/Llama-3-70b",
             tensor_parallel_size=8,
             enable_prefix_caching=True,
             max_num_seqs=128)  # up to 128 concurrent
 
params = SamplingParams(temperature=0.7, max_tokens=512)
 
# Many requests can be submitted concurrently. The engine batches them,
# preempts if memory pressure spikes, streams tokens back out of order,
# and keeps the GPUs pinned at high utilization the whole time.
outputs = engine.generate(prompts=[p1, p2, p3, p4, ...], sampling_params=params)
 
# The 10x–100x throughput gap between these two snippets is the engine.

The top half is a research loop. Every call is synchronous, every request runs alone, the KV cache is implicit and gets thrown away after each call, there's no admission control, no continuous batching, no prefix sharing, no metrics, no graceful degradation. The bottom half looks almost identical from the caller's perspective, but behind the LLM constructor sits the full nine-subsystem engine we just walked through. That's the gap.

Misconceptions

"The model is the inference engine." This is the one I most want to clear up, because the rest of Arc 6 depends on it. The model is a set of weights and an architecture. It has no concept of "requests", "batches", "a queue", "a KV cache eviction policy", or "SSE streaming". The engine is everything that surrounds the weights: the HTTP server, the tokenizer, the scheduler, the batcher, the KV cache manager, the kernel dispatcher, the detokenizer, the streaming sink. Two different engines can serve the exact same model weights and have wildly different throughput, TTFT, and cost profiles. The model is the instrument, and the engine is the orchestra around it.

"All engines are basically the same because they run the same model." Every engine implements the same handful of subsystems, but makes different choices at each one. vLLM leans on PagedAttention for the KV cache and wide continuous batching. TensorRT-LLM compiles a model-specific graph ahead of time and fuses aggressively. llama.cpp runs the whole thing on CPU+GGUF with minimal dependencies. SGLang adds RadixAttention to maximize prefix sharing across requests. The choices compound, and the resulting throughput / TTFT / ITL / cost profiles can differ by 3–10×.

What's next

We've built the block diagram. Every Arc 6 topic from here on zooms into one of its slots. The next post zooms into the hottest one: how the engine actually executes the math on the GPU, and why "just calling cuBLAS for every matmul" leaves massive performance on the table. Start with Kernel Fusion and FlashAttention.

Additional reading (and watching)

• Pope, R., et al. (2022). Efficiently Scaling Transformer Inference. arXiv:2211.05102. The Google paper that establishes the prefill vs decode decomposition, memory bandwidth analysis, and the sharding strategies that modern engines inherited.

• Pope, R., et al. (2022). See above. Section 3 derives the KV cache size formula and its cost implications at scale. The KV cache is the central scarce resource every modern engine manages.

• Yu, G.-I., et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022. The paper that introduced continuous (iteration-level) batching, which is now the default in vLLM, TGI, and TensorRT-LLM.

• HuggingFace. Tokenizers library documentation. Rust-based tokenizer implementations (BPE, WordPiece, Unigram) used by most production engines. The reference for the three dominant tokenizer families.

• Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023 (arXiv:2309.06180). Introduces the PagedAttention algorithm and the vLLM system. The canonical reference for modern KV cache management.

• Agrawal, A., et al. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. arXiv:2403.02310. Shows how chunked prefill and careful scheduling can push sustained GPU utilization to 70–90% while keeping TTFT bounded.

• vLLM project. vLLM documentation. The reference for deploying vLLM in production, including OpenAI-compatible API, distributed inference, and prefix caching.

• HuggingFace. Text Generation Inference (TGI). GitHub. HuggingFace's production serving engine, with continuous batching, streaming, and an OpenAI-compatible API. Good alternative to vLLM with a different set of tradeoffs.

• NVIDIA. TensorRT-LLM. GitHub. NVIDIA's graph-compiled inference engine, optimized specifically for Hopper and Blackwell GPUs.

• NVIDIA. TensorRT-LLM performance overview. Documentation of the compilation pipeline and its throughput wins over eager-execution engines for fixed model shapes.

• Gerganov, G. llama.cpp. GitHub. The reference CPU/edge inference engine and the GGUF quantization format, both of which have become de-facto standards for on-device LLMs.

• Zheng, L., et al. (2023). SGLang: Efficient Execution of Structured Language Model Programs. arXiv:2312.07104. Introduces RadixAttention for prefix-sharing-heavy workloads and the SGLang runtime.

• Leviathan, Y., et al. (2023). Fast Inference from Transformers via Speculative Decoding. ICML 2023 (arXiv:2211.17192). Context for why speculative decoding shows up as an engine-level feature in modern serving systems.