Benchmarking Your Own Inference Stack

A serving benchmark is useful when it predicts what happens to your requests. A high token rate on short, repeated prompts does not tell you whether a long coding prompt will interrupt an active chat stream.

Start with a question you can test: how much traffic can this deployment serve while meeting its latency budgets? Then keep enough detail to reproduce the answer.

Name the measurements

Time to first token (TTFT) measures the interval from a specified start point to the first generated token. Client-observed TTFT includes network and front-end work as well as queuing and prefill. Server-side TTFT may use different boundaries; label them.

Inter-token latency (ITL) is the sequence of gaps between generated tokens. Time per output token (TPOT) is usually a per-request average over the decode interval. For $G>1$ output tokens, a common definition is

The average does not reveal an isolated long stall. Keep token-gap tails and per-request averages separate, and state whether end latency includes the final protocol event.

End-to-end latency is the time to the final output. When the boundaries match, it is TTFT plus the sum of the remaining token gaps. There are $G-1$ such gaps, not $G$.

Output throughput is completed output tokens divided by elapsed wall-clock time. Report whether it covers a device, a node, or the entire deployment. If an eight-GPU tensor-parallel replica generates 800 tokens/s in total, its normalized rate is 100 tokens/s/GPU, not 800. Input-plus-output throughput is a different metric and must be named separately.

Report the mean, median, and relevant tail percentiles alongside sample counts and failures. Means are useful for aggregate accounting; they cannot stand in for tail latency.

A streaming HTTP event is not necessarily one model token. It may contain only a role field, several tokens, usage data, or a finish marker. A client that increments a token counter for every event and timestamps the first event as TTFT can report convincing but incorrect numbers. Inspect the protocol and the benchmark's counting method.

Make the workload representative

Record prompt and output length distributions, request arrival times, generation settings, and prefix sharing. Also retain content where it affects execution: prefix hits depend on actual token identity; speculative acceptance, MoE routing, early stopping, and constrained decoding can depend on the text.

Long prompts deserve explicit coverage. Dense attention still has quadratic arithmetic even when FlashAttention avoids storing the complete score matrix in HBM.

Use both controlled synthetic inputs and a representative trace. Synthetic cases isolate a change; trace replay tests whether that change helps the workload you intend to serve. Keep prefix caching conditions explicit so a warm repeated prompt is not compared against a cold unique one.

Sweep load, and count good results

A closed-loop concurrency sweep keeps up to a fixed number of requests outstanding. Each completion makes room for another. This is useful for exploring saturation, but slower responses automatically reduce the offered arrival rate.

An open-loop test schedules arrivals independently of completions. It can expose a growing queue that a closed-loop test hides. Replay realistic bursts or use a documented arrival process, and ensure the client can sustain it. Distinguish time waiting in a client-side semaphore from time waiting at the server.

For interactive traffic, measure the request rate that meets both TTFT and streaming-latency requirements. Goodput counts successful work meeting the chosen budgets. A failed, timed-out, or truncated request must not silently disappear from the denominator. Leave operating headroom for variability rather than deploying exactly at one observed threshold.

Control the measurement conditions

Warm up the execution paths you intend to measure, and record the warmup policy. Compilation, graph capture, and allocation may affect initial requests. Separately measure startup or cache-miss behavior if it matters to the product; “warm” is not automatically more representative.

Use the model's tokenizer and chat template. Confirm whether reported counts include prompt tokens, special tokens, or speculative drafts. Pin the checkpoint revision, quantization configuration, engine version, drivers, hardware topology, power settings, and sharing/isolation policy.

A few hundred requests may characterize a median while providing very few observations near p99. Repeat runs, report variability, and use enough duration and samples for the tail you care about.

A reproducible starting command

vLLM provides an online benchmark CLI with workload and load-pattern controls. Use the CLI belonging to the installed release; flags and supported datasets can change. Its benchmark guide explains the metric boundaries and arrival controls.

# In one terminal, after installing a compatible vLLM build:
vllm serve meta-llama/Llama-3.1-8B-Instruct
 
# In another terminal, inspect this installed version's options:
vllm bench serve --help
 
# A small synthetic baseline against the running server:
vllm bench serve \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --dataset-name random \
  --num-prompts 200 \
  --max-concurrency 8 \
  --save-result

This example requires checkpoint access and suitable hardware. It leaves synthetic length defaults to the installed release, so record those defaults or set them explicitly before comparing runs. It is a smoke measurement, not a production capacity result. Extend it with a documented length distribution, warmup, arrival-rate sweep, and saved detailed records.

Convert achieved throughput into cost

Suppose a complete deployment costs an assumed $2/hour and sustains an assumed 1,800 output tokens/s over the billed interval. Its compute-only cost is

per million output tokens. These inputs are hypothetical, not a current GPU rental quote or a measured Llama result.

Use an achieved rate over the same interval as the cost. If you instead start from a peak saturated rate, account for idle time separately; do not apply an extra utilization factor to a rate that already includes it. Host memory, networking, storage, replicas, and other operating costs may belong in the final budget.

An increase in concurrency at the latency threshold lowers cost per token only if successful throughput rises relative to provisioned cost. Report both alongside the workload and latency measurements.

The next arc turns from serving weights to producing them, beginning with Pretraining Data.

Additional reading

• vLLM serving benchmark implementation: the actual definitions and aggregation logic.
• MLPerf Inference: standardized scenarios, rules, and measurement tooling.
• Zhong et al., DistServe: optimizing goodput under latency constraints.
• Agrawal et al., Sarathi-Serve: scheduling tradeoffs between throughput and latency.