What an Inference Engine Actually Does

A model's forward pass maps tokens to logits. Serving it adds a different problem: many requests arrive at different times, use different amounts of memory, and expect a response while other requests are still running.

An inference engine coordinates that work. It loads weights, schedules model execution, manages cached attention state, and returns generated tokens. A server adds the surrounding API and operational controls. Some projects package both together.

Follow one request

A request first passes through validation and tokenization. Chat messages need the checkpoint's chat template; raw text and a list of message objects are not interchangeable model inputs. The tokenizer must match the weights.

The request then waits for admission. The scheduler decides whether it can run within the current sequence, token, and memory budgets. A full cache does not imply one universal response: an engine may queue work, preempt a request for later recomputation, offload state, or reject a request according to its configuration.

Prefill processes the prompt through the model and creates its KV entries. Prompt positions can be computed in parallel within a layer under the attention mask; the layers still execute in dependency order. The final prompt position supplies logits for the first generated token.

The sampler selects that token. For a streaming request, the server can detokenize and emit it immediately. On the next model pass, the selected token becomes input, its K and V are appended, and its logits produce the next selection. A token's KV entry is created when the model processes it, not merely when the sampler chooses it.

This repeats until a stop condition, length limit, cancellation, or error. Request-owned blocks can be released; reusable prefix blocks may remain cached under the engine's eviction policy. Streaming and detokenization happen throughout generation, rather than once after the complete decode loop.

What the scheduler coordinates

With a static batch, membership stays fixed until its work finishes. A short request can leave an empty lane while another continues. Continuous batching allows completed requests to leave and new work to enter between iterations, subject to the scheduler's budgets.

A high slot-occupancy percentage is useful, but it is not the same as tensor-core utilization or memory-bandwidth utilization. A busy GPU can still spend much of its time moving data. Actual throughput depends on the kernels, batch sizes, sequence lengths, and communication as well as admission.

A long prefill can also delay ongoing decode work. Chunked prefill and continuous batching let an engine distribute prompt work across iterations. Priority policies choose which work gets those iterations. These are separate decisions, and their effects need workload-specific measurement.

The cache is part of the capacity plan

For a uniform full-attention decoder, cache bytes per stored token across the whole model are

The factor two counts keys and values. Use the number of KV heads, not query heads. The formula assumes equal dimensions in every layer; hybrid attention and compressed-cache architectures need their own accounting.

For 80 layers, eight KV heads, dimension 128, and BF16 cache elements, that is 327,680 bytes per token, or 320 KiB. A 32,768-token request needs about 10.74 decimal GB of KV data before page rounding and metadata.

Weights, runtime workspace, graph allocations, and safety headroom occupy memory too. Adding GPU capacities gives an optimistic aggregate bound only when sharding and placement make that memory usable. Replicated KV heads or weights can reduce it. The memory-management post works through this budget.

Where engines make different choices

Kernels determine how matrix operations and attention use the device. Paged KV allocation determines how growing sequences occupy memory. Quantization changes storage formats and arithmetic paths. Parallelism distributes work and adds communication.

Offline generation and online serving

The vLLM offline API takes a finite collection of prompts and returns generated results. It can schedule that batch internally; this example does not stream results or listen for new network requests.

from vllm import LLM, SamplingParams
 
# Requires a compatible installation, suitable hardware, and model access.
llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")
conversations = [
    [{"role": "user", "content": "Explain why the sky looks blue."}],
    [{"role": "user", "content": "Name three common sorting algorithms."}],
]
results = llm.chat(conversations, SamplingParams(temperature=0, max_tokens=80))
for result in results:
    print(result.outputs[0].text)

For an online service, start the server instead, in a separate process with enough resources for its model:

The offline API and online-serving guide document these entry points.

Next: kernel fusion, where avoiding an intermediate memory round trip can matter as much as reducing arithmetic.

Additional reading

• Yu et al., Orca: iteration-level scheduling for generative models.
• Kwon et al., PagedAttention: block allocation and sharing for KV state.
• vLLM scheduler configuration: current admission, token-budget, and scheduling options.