Comparing Engines: vLLM, TGI, TensorRT-LLM, llama.cpp, SGLang
Runtime choices
High-throughput serving with paged cache management and iteration-level scheduling.
Check the installed version, model format, and hardware support before comparing performance.
An engine choice starts with a concrete deployment: a checkpoint, some hardware, and requests that must finish within a useful time. The engine has to support that combination before its benchmark numbers matter.
The preceding posts explain the parts of that decision: memory budgeting, quantization, parallelism, and scheduling. Here we use them to build a shortlist and a fair comparison.
What each project gives you
vLLM is a serving runtime built around batched execution and paged KV storage. Its model, hardware, and quantization support is broad, but support is a combination: a model working in BF16 does not establish that its chosen low-bit format works on your accelerator.
SGLang combines a serving runtime with radix prefix caching and interfaces for language-model programs.
TensorRT-LLM targets NVIDIA hardware. Older instructions centered on converting checkpoints and building TensorRT engines. NVIDIA's current documentation removes that backend in favor of the PyTorch execution backend, with AutoDeploy also using it. Check your installed release before following a build guide; the old workflow and the current one are materially different. TensorRT-LLM migration documentation.
llama.cpp runs GGUF models through CPU and several GPU backends. It is useful for local inference and also includes an HTTP server. Its backends differ in kernels and supported serving features.
Text Generation Inference (TGI) remains relevant to existing deployments, but its repository is archived and its README directs new work toward other runtimes. Maintenance is part of the comparison, even when an existing binary still runs successfully. TGI repository.
Check GPU generation, CUDA version, memory capacity, and the required precision kernels. vLLM, SGLang, TensorRT-LLM, and llama.cpp have NVIDIA paths; their supported combinations differ.
On Apple Silicon, include Apple's MLX as another local-inference candidate.
Compare supported combinations
A starting comparison, not a compatibility guarantee. The linked version’s documentation is authoritative.
| Engine | Emphasis | Verify before deployment |
|---|---|---|
| vLLM | Paged KV and batched serving | Check the model × accelerator × quantization combination |
| SGLang | Serving runtime with radix prefix caching | Measure prefix hits, scheduling, and constrained decoding on your trace |
| TensorRT-LLM | NVIDIA inference runtime | Current PyTorch backend loads supported checkpoints directly; older releases used compiled engines |
| llama.cpp | GGUF inference across CPU and GPU backends | Check backend-specific kernels, memory placement, and server concurrency |
| TGI | Existing Hugging Face serving deployments | Repository archived March 2026; evaluate maintenance and migration needs |
Two engines may both support prefix caching while differing in eviction, request placement, and how much of your traffic actually hits the cache. Both may support structured output while accepting different schema features. “OpenAI-compatible” similarly describes an interface family, not a guarantee that every endpoint, streaming event, or request option is identical.
Keep model quality in the comparison. Changing the quantized checkpoint to make one engine run faster changes the experiment. Even with identical weights, floating-point kernels and sampling can produce different outputs; a shared architecture does not guarantee identical tokens.
Tiles attention to avoid materializing the full score matrix in HBM.
Demonstrated iteration-level scheduling for autoregressive model serving.
Developed local inference and the GGUF model container across CPU and GPU backends.
Mapped logical KV blocks onto a shared physical pool, reducing allocation waste.
Hugging Face serving project; archived in March 2026.
NVIDIA inference runtime; its original compiled-engine workflow differs from the current PyTorch backend.
Combined radix prefix caching with a language-model program frontend.
Chunked prefill and disaggregated serving became additional design choices; support remains version-dependent.
Trace how serving techniques spread between projects.
A shortlist for a chatbot
Suppose a team has an eight-GPU NVIDIA node, a supported 70B checkpoint, and a chat workload with repeated system prompts. Its product budgets are p95 TTFT below 500 ms and p95 inter-token latency below 60 ms. Those are hypothetical requirements, not predictions for the hardware.
First check that each candidate supports the checkpoint's precision and the node's GPU generation. Then run the same request trace, with the same prefix-cache policy, on the surviving candidates. Record memory use, successful requests, tail latency, and aggregate output throughput. Tune each engine within the same resource limit and keep the resulting configurations.
The decision can then be concrete: one configuration may meet the latency budgets at more arrivals per second; another may be easier for this team to operate.
- Filter by the exact checkpoint, hardware, precision, and required API features.
- Check project maintenance and the deployment team’s existing tooling.
- Replay the same workload on the remaining candidates, with latency budgets and quality checks.
- Choose using measured capacity, operating cost, and deployment effort; record the versions and revisit after material changes.
Launching a comparable baseline
These commands describe alternative deployments on the same eight-GPU node. Run one at a time. Both use the same unquantized checkpoint.
# Requires checkpoint access and a compatible vLLM installation.
vllm serve meta-llama/Llama-3.1-70B-Instruct \
--tensor-parallel-size 8 \
--port 8000
# Alternative: current TensorRT-LLM CLI, on compatible NVIDIA hardware.
trtllm-serve meta-llama/Llama-3.1-70B-Instruct \
--tp_size 8 \
--port 8000These are startup examples, not matched performance configurations. Pin the checkpoint revision and engine versions, then align context limits, request budgets, precision, and cache settings. The current command surfaces are documented by vLLM and TensorRT-LLM.
Build a comparison from measurements
- p95 time to first token
- p95 inter-token latency
- Requests meeting both latency budgets
- Output tokens/s across the complete deployment
Hold hardware, model revision, precision, request trace, and cache policy constant. Record engine versions and tuning. Compare measured successful output rates under the same latency budgets.
Keep the layers distinct
- 1. Client and API
Request format, authentication, streaming, retries
- 2. Serving platform
Routing, autoscaling, replicas, observability
- 3. Inference runtime
Scheduling, KV storage, model execution
- 4. Kernels and hardware
Matrix products, attention, collectives, memory transfers
Products can span several layers or support several runtimes. An API-compatible service does not reveal which engine executes its requests.
A platform can add authentication, routing, autoscaling, and deployment management around an engine. Some platforms support several engines or their own execution code. Avoid inferring a provider's internals from a familiar endpoint or a historical integration announcement.
The next post turns this comparison into a repeatable experiment: Benchmarking Your Inference Stack.
Additional reading
- vLLM documentation and SGLang documentation: supported models, serving configuration, and hardware-specific setup.
- llama.cpp repository: backends, GGUF support, and the server.
- MLX-LM repository: model generation and fine-tuning on Apple Silicon.
- Zheng et al., SGLang: radix caching and language-model programs.