Comparing Engines: vLLM, TGI, TensorRT-LLM, llama.cpp, SGLang

If you've been reading Arc 6 in order, you've spent the last several posts staring at a single inference problem from different angles: kernel fusion, paged KV, continuous batching, tensor and pipeline and expert parallelism, quantization formats, API surfaces. Each post picked one slice and went deep.

This post is the zoom-out. It's the one where all those slices stop being separate posts and start being columns in a table.

An "inference engine" is not a monolithic piece of software. It's a synthesis of choices along every axis we've covered. vLLM is one synthesis. TGI is another. TensorRT-LLM is a third. llama.cpp and SGLang are the fourth and fifth. They all run the same matmuls on the same weights and produce the same tokens. What makes them feel completely different is the particular combination of batching policy, KV layout, kernel set, quantization support, parallelism strategy, and API surface each one committed to.

That's what we're going to compare. Not which engine is "best" (there is no best), but which synthesis matches which problem.

A very quick refresher

Before we dive in, let me re-establish a few bits of Arc 6 vocabulary so the post stands on its own if you're dropping in.

The KV cache is the per-request store of attention keys and values. Caching it is what lets the decode phase avoid recomputing the whole prefix every step. Paged KV (paging the cache into fixed-size blocks) makes that cache behave like virtual memory: no fragmentation, cheap sharing across requests. Continuous batching is a scheduler policy where new requests can join the batch on every step instead of waiting for the current batch to finish. Prefix caching reuses the KV blocks of a shared system prompt or template across requests so you don't re-prefill the same tokens twenty thousand times an hour.

Hold those four ideas. They show up in every column of the matrix below.

Five engines, stated as bets

Let's name each engine by the bet it's making. A "bet" here is the thing the engine prioritized so aggressively that it shaped everything else about the codebase.

vLLM bets on PagedAttention plus continuous batching plus an OpenAI-compatible API, on every accelerator that will run CUDA or ROCm. The paper that introduced it (Kwon et al., SOSP 2023) was the first widely-read demonstration that paging the KV cache was a real win, and vLLM rode that into being the OSS default for GPU inference. Today it supports 200+ model architectures, FP8/FP4/INT4/GGUF, tensor and pipeline and expert parallelism, and speaks OpenAI's, Anthropic's, and gRPC APIs. Flexibility first, extreme peak performance second.

TGI (Text Generation Inference, HuggingFace) bets on being the production engine for HuggingFace's ecosystem. If your team lives in transformers, pulls models from the Hub, and wants multi-accelerator support (NVIDIA, AMD, Gaudi, Inferentia, TPU) with minimal lift, TGI is the path of least resistance. It absorbed continuous batching and paged attention early and wraps them in a turnkey server. As of December 2025 its repo is officially in maintenance mode, which matters for the roadmap question but doesn't change what it does today.

TensorRT-LLM (NVIDIA) bets on maximum throughput on NVIDIA silicon, at the cost of build complexity. It compiles the model graph ahead of time, fuses kernels aggressively, and ships FP8/FP4/INT4 AWQ/GPTQ paths tuned per-architecture (Hopper, Blackwell, Grace-Hopper). You get in-flight batching, paged KV, prefix caching, speculative decoding. You also get an AOT build step per model per precision per shape regime, which is a different posture from "point it at a checkpoint and go." If you live in NVIDIA's data center and can absorb the build complexity, this is usually the fastest option.

llama.cpp (Georgi Gerganov and a huge community) bets on running a quantized transformer on whatever hardware the user has in front of them. CUDA, Metal, ROCm, Vulkan, SYCL, plain CPU, any combination of the above in one process. It invented GGUF (the ubiquitous on-disk quantized format) and supports 1.5-bit through 8-bit integer quantizations. It's the only engine in this post that is not fundamentally about datacenter serving. It is, however, the dominant engine at the edge, and the thing that makes "ChatGPT-quality on my MacBook" a real sentence in 2026.

SGLang bets on prefix sharing and programmable frontends. Its core contribution is RadixAttention, which stores KV blocks in a radix tree keyed on token prefixes so that shared system prompts, few-shot examples, and agent scaffolds hit an in-memory cache instead of re-prefilling. On top of that it puts an LM-program frontend (you write Python; it orchestrates the requests) and a compressed-FSM grammar decoder that is roughly 2-2.5× faster at structured output than the naive approach. When your workload is "agents issuing thousands of prompts that share 80% of their tokens," SGLang's bet pays hard.

The first axis that prunes hard in real procurement decisions is hardware. Before you ever think about PagedAttention or RadixAttention, you have to answer "what accelerator are we actually running on." So let me show you that shape first.

Notice the shape. llama.cpp and the four datacenter engines barely overlap in their strongest habitat. The datacenter four all want H100s. llama.cpp wants whatever's in front of the user. That split is the first thing a good decision tree will branch on.

The comparison, as a table

Here's the one view I wish I'd had when I started picking between these. Each row is an engine. Each column is one axis we've been covering in Arc 6. Hover a row or column to isolate it; click a header to sort.

First, if you squint at the "paged KV," "continuous batching," and "prefix caching" columns, the four datacenter engines (vLLM, TGI, TensorRT-LLM, SGLang) are converging. Every one of them has absorbed every one of those techniques within about twelve months of it being invented. This is the cross-pollination story: the field has a short enough feedback loop that "novel engine feature" and "table stakes" are separated by a few months.

Second, the quantization column is where you see the most divergence. llama.cpp's GGUF tree of formats (Q4_K, Q5_K_M, IQ4_XS, and the rest) has no real counterpart in the datacenter engines. Datacenter inference mostly cares about FP8 and a couple of INT4 flavors. llama.cpp cares about squeezing a 70B model into 32 GB of unified memory on a Mac Studio.

Third, the API surface column is where the business decisions live. "OpenAI-compatible" is the table stakes; beyond that, vLLM speaks Anthropic Messages and gRPC, TensorRT-LLM pipes through Triton or NIM or Dynamo, SGLang exposes its LM-program SDK, and llama.cpp keeps its HTTP server minimal on purpose.

How did we get here?

Every engine in this post is a response to a specific 2022–2023 paper or piece of infrastructure. The current state of the field is the absorbed sum of those moves.

The pattern I want you to notice is the absorption time. FlashAttention shipped in 2022 and was in every engine by late 2023. PagedAttention shipped in vLLM in 2023 and was in TGI, TensorRT-LLM, and SGLang within a year. RadixAttention came from SGLang in 2024 and versions of prefix caching are now everywhere. The engines aren't really competing on feature sets anymore; they're competing on how early they shipped the feature, how clean the implementation is, and which pieces of infrastructure they fit into.

A decision tree (heuristic, not gospel)

If you collapse the matrix into "which one should I actually pick," you can walk yourself through a small decision tree. This is not a production runbook. It's the kind of thing you'd draw on a whiteboard before a procurement conversation.

Read it left to right. The hardware question prunes the hardest: if you're on a Mac or an edge device, llama.cpp is basically the only choice. If you're on an NVIDIA datacenter GPU, the question becomes whether you want flexibility (vLLM), build-complexity-in-exchange-for-throughput (TensorRT-LLM), ecosystem glue (TGI), or prefix-sharing muscle (SGLang).

A worked example

Let me make this concrete. Consider a team serving a production chatbot.

Setup: 50M monthly active users. AWS p4d instances (8× A100 40GB per node). Model: Llama 3 70B, served at INT8 via AWQ. Chat workload: average input length ~800 tokens (system prompt + recent history), average output length ~150 tokens, p99 ITL target under 40ms, and maybe 40% of prompts share a long company-style system prompt.

Walk the tree.

1. Hardware: NVIDIA datacenter. Prune llama.cpp.
2. Model: Llama family. We're in the "Llama / Mistral / Qwen / DeepSeek family" branch.
3. Workload: chat traffic at scale, 40% shared prefix, ITL matters.
4. Team capacity: assume a small platform team, not a dedicated NVIDIA systems engineer.

vLLM is the default answer. PagedAttention handles the 70B KV cache budget across concurrent requests without fragmentation. Continuous batching keeps the ITL low when request arrivals are bursty. The OpenAI-compatible endpoint drops into every client library the team already uses. Prefix caching handles the 40% shared system prompt. vLLM is the safe, boring, correct choice.

Consider TensorRT-LLM if the team can budget for the AOT build complexity and wants ~1.3–1.6× more throughput per dollar. The win is real. The build pipeline (compile per model × per precision × per input-length regime, then validate with TRT-LLM benchmarks) is also real. If they're running this at enough scale that a 30% throughput improvement pays for a dedicated inference engineer, it's worth it. If not, it's overkill.

Consider SGLang if the shared-prefix fraction goes up further (say, 80% of traffic has a 2,000-token system prompt with retrieved context). RadixAttention was built for exactly this case, and the compressed-FSM grammar decoder is useful if they're also enforcing JSON schemas on the output.

Consider TGI if the team is already deep in HuggingFace tooling (trainer, PEFT, datasets, auto-scaling with Inference Endpoints). The ~10–20% throughput gap versus vLLM may be worth the time not spent on glue code.

Reject llama.cpp here. It's wrong for this workload. The right shape for it is an on-device app, not a datacenter chat service.

Now, the question to ask your team: which of these constraints might change in the next twelve months? If the answer is "we're going to open an on-device mode so users can run the model locally," you want an architecture where the serving layer and the inference engine are separable. Because you'll probably end up with vLLM in the datacenter and llama.cpp on the device, and that's fine.

What the numbers actually look like

Here's a rough, heavily-caveated view of throughput on that Llama 3 70B setup. I am showing it because not showing it feels like a cop-out, but I want to be extremely clear that these numbers are an illustrative snapshot, not a measurement you should take to your planning doc. Post 6.9 is entirely about why published benchmarks do not transfer to your workload, and everything it says applies here.

A few things to notice:

• The gap between the top datacenter engine and the bottom one is not that big. It is almost always less than 2× for a given reasonable workload. The gap between the top datacenter engine and llama.cpp is enormous, but that's an "apples to pomegranates" comparison (single-request-at-a-time on consumer hardware vs. concurrent requests on 8× H100).
• The ranking shifts between the chat and batch workloads. TensorRT-LLM's lead widens on long outputs because its fused kernels amortize the AOT compile cost over more tokens. SGLang's lead on prefix-heavy workloads (not shown in this simple chart) widens dramatically as the shared-prefix fraction grows.
• A 10–30% throughput difference is often the right price to pay for a much better development loop. Nobody's business case depends on picking the absolute fastest engine. Most business cases depend on shipping something and iterating.

A minimal config, in two engines

To make the "same workload, different engine" idea concrete, here's the same Llama 3 70B chat server launched two ways. Both expose an OpenAI-compatible endpoint on port 8000. The numbers match the worked example above.

# --- vLLM, the boring correct choice ---
# One command, tensor-parallel over 8 GPUs.
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-70B-Instruct \
  --tensor-parallel-size 8 \
  --quantization awq \
  --enable-prefix-caching \
  --max-num-seqs 256 \
  --port 8000
 
# Prefix caching is on, paged KV is on by default,
# continuous batching is on by default.
 
# --- TensorRT-LLM, the NVIDIA path ---
# Step 1: compile the graph ahead of time (this takes ~20 min).
trtllm-build \
  --checkpoint_dir ./llama3-70b-awq-ckpt \
  --output_dir ./engines/llama3-70b-awq-tp8 \
  --gemm_plugin auto \
  --paged_kv_cache enable \
  --use_paged_context_fmha enable \
  --max_batch_size 256 \
  --max_input_len 4096 \
  --max_seq_len 4352 \
  --tp_size 8
 
# Step 2: serve the compiled engine.
trtllm-serve \
  ./engines/llama3-70b-awq-tp8 \
  --port 8000 \
  --openai_api

Two different postures. vLLM is one command. TensorRT-LLM is a build step and then a serve step, and you get to make decisions at build time that vLLM picks for you at runtime. Both produce an OpenAI-compatible endpoint, and both will serve the same tokens from the same weights. The differences live entirely in the scheduling, the kernels, and the memory layout.

What I left out

This post covers five engines. The broader ecosystem is bigger, but almost everything else in the space is a thin layer on top of one of these five.

NVIDIA NIM is TensorRT-LLM in a container with an OpenAI API. Ollama and LM Studio are llama.cpp with a GUI. Aphrodite is a vLLM fork with community patches. Together and Modal run vLLM or TRT-LLM underneath. HF Inference Endpoints run TGI. The closed stacks at OpenAI, Anthropic, and Google are their own beasts and we don't have public details.

The framework is what matters. The specific snapshot of the market changes every quarter.

Misconceptions

"There is a best engine." There isn't. The matrix is the answer. Pick the synthesis that maps to your hardware, model, workload shape, and API needs, and run your own benchmarks before committing.

"vLLM is always fastest on NVIDIA." No. On a fair apples-to-apples setup, TensorRT-LLM is usually faster than vLLM on NVIDIA GPUs, often by 20–50% on batch-heavy workloads. The reason people reach for vLLM anyway is that the build-and-iterate loop is shorter, the model coverage is wider, and the throughput gap is often small enough that it doesn't affect the business case. vLLM trades 10–30% raw throughput for massive flexibility.

"llama.cpp is just for hobbyists." llama.cpp is the dominant on-device LLM engine on earth. Ollama, LM Studio, Jan, LocalAI, and a large fraction of local-first AI startups sit on top of it. If you count "inference sessions per day," llama.cpp is probably in the top two engines globally, alongside vLLM. The right axis to think on is datacenter versus on-device, not hobby versus production.

"Picking an engine is a one-time decision." This space moved more between 2023 and 2026 than any other part of the LLM stack. Every engine shipped a major feature in that time that some other engine matched within months. Architect your serving layer so the engine is swappable: a stable OpenAI-compatible interface above, different engines below. You will swap, probably more than once.

What's next

The post you just read is, by design, a snapshot that won't age well. Every one of these engines will have shipped something new by the time you're reading this. What does age well is the framework: engines are syntheses of choices, those choices are the axes we spent Arc 6 unpacking, and the right engine for your problem is the one whose synthesis matches your constraints.

The next post is the practical companion: how to actually measure whether your chosen engine is performing on your workload, instead of trusting vendor blog numbers. It's the most important post in this arc if you're the person who has to defend a serving choice in a review. Start with Benchmarking Your Inference Stack.

Additional reading (and watching)

• See Arc 5's KV cache from first principles for the full derivation of why we cache keys and values and what the memory math looks like.

• Yu, G.-I., et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022. The paper that coined continuous (iteration-level) batching, which is now table stakes in every datacenter engine in this post.

• Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023 (arXiv:2309.06180). The vLLM paper; introduced paged KV and demonstrated 2–4× throughput improvements over prior systems.

• vllm-project/vllm on GitHub. The canonical source for what vLLM currently supports; see also the vLLM blog at vllm.ai/blog for release-by-release feature notes.

• huggingface/text-generation-inference on GitHub. The TGI repository; note the maintenance-mode notice as of December 2025. Covers feature matrix, supported quantizations, and hardware targets.

• NVIDIA/TensorRT-LLM on GitHub. Official repository. Documents the AOT compile flow, supported precisions (FP8/FP4/INT8 SmoothQuant/INT4 AWQ/INT4 GPTQ), and parallelism combinations (TP/PP/CP/EP).

• ggerganov/llama.cpp on GitHub. The original C++ llama inference implementation, the GGUF format specification, and the long list of hardware backends it supports.

• Zheng, L., et al. (2023). SGLang: Efficient Execution of Structured Language Model Programs. arXiv:2312.07104. Introduces RadixAttention and the LM-program frontend; see also sgl-project/sglang for the current feature set.

• NVIDIA. (2024). NVIDIA NIM for LLMs. Overview page for the productized, container-packaged version of TensorRT-LLM with an OpenAI-compatible front end.

• Dao, T., et al. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS 2022. The fused-kernel attention paper that every engine in this post now ships a variant of; included for the timeline.