Benchmarking Your Own Inference Stack

This is the closing post of Arc 6. We spent the arc pulling apart the guts of a modern inference engine: the KV cache, prefill vs decode, continuous batching, paged attention, quantization, parallelism, scheduling. You can now read a vLLM or SGLang config file and mostly know what every knob does. So.

Now you want to know whether your stack is actually good. And this is where things get tricky.

It's easy to open a vendor blog post, see "2× faster than framework X on Llama 3 70B," and go rearrange your serving layer based on that one number. Six weeks later p99 TTFT is 4× worse in production than it was in the benchmark, and nobody can reproduce the number internally.

The fix is to stop trusting numbers that came off someone else's rig and start measuring your own. This post is about how to do that without lying to yourself.

Published numbers don't transfer

Vendor benchmarks are nearly useless for your capacity planning.

A benchmark number is a function of at least six things: the hardware (A100? H100? H200? B200?), the model (and its exact quantization), the prompt length distribution, the generation length distribution, the concurrency pattern, and whether you're measuring with or without prefix caching, speculative decoding, chunked prefill, and a dozen other features. Every one of those is a free parameter. The vendor picked the configuration that made their product look good. You're picking the configuration your actual users create.

Your numbers will differ on every axis. Your prompts are longer or shorter. Your users are chatty or bursty. You're running INT8 weights, they were running FP16. You have a shared GPU, they had a dedicated node. The cross-product of all this is enormous, and the right number, the only one that matters, is the one you get from your stack under your workload.

Vendor numbers are fine as sanity checks. ("Is my setup within 2× of the advertised number?" is a reasonable question.) They are not your operating envelope.

The metrics that matter

Before we talk about how to measure, let's pin down what to measure. Inference is a streaming workload, so latency is always at least three numbers, and each one maps to a different user-visible thing.

Time to first token (TTFT). From the moment the request hits your server to the moment the first generated token comes back. This is the prefill-plus-scheduling latency. As a quick refresher from earlier in Arc 6: prefill is the big parallel pass that processes the prompt and fills the KV cache; decode is the serial token-by-token generation after that. TTFT is what the user perceives as "how responsive does this feel." A chat app with a 3-second TTFT feels broken. A 300ms TTFT feels snappy.

Inter-token latency (ITL). The gap between one streamed token and the next, averaged over the decode phase. This is perceived as the "smoothness" of the streaming output. 30ms ITL reads fluidly. 200ms ITL reads like someone is typing each word by hand. Sometimes you'll see this called TPOT (time per output token) or TBT (time between tokens). Same thing.

End-to-end latency (TTLT or e2e). Total wall-clock from request in to last token out. Roughly TTFT + ITL × output_tokens. This is what matters for non-streaming use cases (batch jobs, tool calls, structured outputs) and for SLO calculations on the whole request.

Throughput. Tokens per second per GPU, usually as "total tokens generated across all concurrent requests, divided by seconds, divided by GPUs." This is the number your finance team cares about, because it maps directly to cost per million tokens. It rises as you add concurrency, until it doesn't.

Tail percentiles (p50 / p95 / p99). Every one of the above is a distribution, not a number. Report the median, the 95th percentile, and the 99th. The mean is almost always useless and sometimes misleading. The tail is what your users complain about, and the tail is what wakes you up at 3am.

$/M tokens. The only number leadership actually reads. Throughput × utilization × hourly cost, pushed through the appropriate unit conversions. Every optimization you make in this post eventually has to justify itself in this currency.

That histogram is the single most important plot in this whole post. If you take one thing away: report distributions, not averages. A "mean TTFT of 330ms" reassures the room. A "p99 TTFT of 1.4s" starts an actual engineering conversation.

Designing a representative workload

OK, so you know what to measure. Now the harder question. What do you measure it on?

The job is picking an input distribution that predicts production. If you benchmark with 200-token prompts and your real users send 4000-token coding contexts, your TTFT numbers will be optimistic by a factor that scales with prefill cost (attention compute is quadratic in prompt length, and while FlashAttention keeps the memory traffic linear, the FLOPs themselves still grow as $N^2$). Your scheduling will also be wrong because prefill bottlenecks show up very differently at long context.

There are four axes to pin down:

1. Prompt length distribution. Not an average. The full distribution. Your median might be 120 tokens, but if 5% of your traffic is 20k-token coding prompts, those prompts dominate prefill time and KV cache usage. You want to reproduce the shape.
2. Generation length distribution. Same deal. "Average output is 200 tokens" hides the fact that 10% of your traffic might generate 2000+ tokens for reasoning-heavy prompts. Those requests sit in the decode loop much longer and affect concurrency.
3. Concurrency profile. Is your traffic steady (a chat product at midday) or bursty (a morning-spike news analysis tool)? Bench both the steady-state and the burst if they matter to your SLO.
4. Prefix-sharing rate. How much of each prompt is a shared system prompt or shared document context that prefix caching could hit? If your engine supports prefix caching (Arc 6 covered this: Prefix Caching) and your traffic has high prefix-sharing, your cache-miss benchmark will wildly understate real performance.

The easiest way to get these distributions right is to sample from production logs. If you have an access log with prompt lengths, output lengths, and timestamps, that's your benchmark spec. Synthesize prompts of matching lengths (content doesn't matter for latency, only token count does) and replay them with matching inter-arrival times.

If you skip this step, your benchmark numbers will not predict production behavior. Synthetic benchmarks on narrow distributions are where prefill bottlenecks go to hide until they ship.

Throughput is meaningless without a latency budget

So. Throughput is not an objective on its own. An engine that can "do 8,000 tokens/sec" at a p95 TTFT of 6 seconds is running a queue with a throughput meter bolted to the side, and users experience that as broken.

The right way to think about this is: you have a service-level objective (SLO). Something like "p95 TTFT under 500ms, p95 ITL under 60ms." This is a contract with your users (or with your product) about how fast the service feels. Every throughput number has to be read against that SLO.

What you actually want to find is the concurrency where your latency SLO just barely holds. That's your max safe concurrency, and the throughput at that concurrency is your operating throughput. Push past it and your latencies collapse; stay below it and you're leaving money on the table.

Drag the slider. What you're seeing is the underlying shape of every serving system in the world. Continuous batching extends the flat part of the throughput curve (more concurrency before saturation). Paged attention lets you pack more requests into memory. Speculative decoding pushes the ITL down. None of those optimizations change the basic pattern: throughput goes up, latency goes up, the SLO defines where "good" ends.

Once you have the curve, the optimization game becomes concrete: move the SLO-crossing concurrency to the right, and your $/M tokens goes down proportionally.

Common pitfalls

A short list of things that are easy to get wrong, roughly in the order they tend to come up.

Benchmarking on a cold cache. If you send a batch of requests right after starting your engine, your first-request TTFT includes CUDA graph compilation, kernel autotune, weight loading into L2, and probably a JIT pass or two. None of that repeats in production. Always warmup, usually with a few hundred throwaway requests, and report steady-state numbers. Mention warmup explicitly in your writeup so readers know.

Ignoring p99. A common pattern: run a benchmark, see "mean TTFT 200ms," declare victory. Six weeks later, on-call gets paged because p99 is 4 seconds. The mean hid a bimodal distribution where 1% of requests queue behind a long prefill. Always report tails. Mean and median are not interchangeable for skewed distributions, and latency distributions are always skewed.

Tokenizer mismatch. If your client counts tokens using tiktoken and your server uses the model's actual tokenizer, your "tokens per second" numbers are based on a different count of tokens than what the GPU did. The drift can easily be 10–15%. Make your benchmark client use the same tokenizer as the serving model.

Ignoring SLOs. Throughput without a latency budget is a meaningless number, as we just covered. If someone reports a throughput improvement, the first question is "at what p95?" If they don't know, the number is incomplete.

Running on an idle GPU. If you bench on a dedicated H100 but production runs two replicas on a shared node, you are measuring the wrong machine. Reproduce production's isolation setup (cgroups, MIG partitions, NUMA pinning) in your benchmark or accept that your numbers will be optimistic.

Tiny prompt distributions. A benchmark with 100-token prompts doesn't exercise your prefill path. The first time a real 10k-token prompt lands, you discover that your scheduler queues everything behind it. Benchmark the tail, especially if your engine supports chunked prefill (we covered this in Chunked Prefill and Budgets).

The two-phase loop: micro-bench, then load test

The last structural thing I want to pin down. A good benchmarking setup has two distinct phases, and they serve different purposes.

Micro-bench. One request at a time, no concurrency, measured carefully. Varies prompt length, output length, and generation settings one at a time. This tells you the fundamental curve: for a prompt of length $P$ and output length $G$, what is the single-request TTFT and ITL? This is the calibration layer. It is dead simple to run and catches gross bugs in your tokenizer, your server config, your network round-trip.

Load test. Full concurrency, realistic distributions, replay from production logs if you can. This is the validation layer. Its job is to check that the micro-bench curves compose correctly under load and that your SLOs hold at the concurrency you plan to deploy at.

Both matter. A micro-bench without a load test tells you what a quiet Tuesday at 3am looks like but nothing about peak traffic. A load test without a micro-bench gives you numbers you can't interpret because you don't know what single-request performance looks like. If concurrent p95 TTFT is 500ms, is that because single-request TTFT is 500ms and there's no queueing, or because single-request TTFT is 80ms and you're queued 6-deep? The micro-bench tells you.

Worked example

Let's ground all of this in a specific, reproducible setup. Say I have a single A100 40GB and I want to deploy Llama 3 8B behind a chat product. Here is the plan, start to finish.

Step 1. Pin the SLO. Chat product, streaming UI. I set p95 TTFT ≤ 500ms and p95 ITL ≤ 60ms. Anything slower than that and the UI feels laggy. These are product decisions, not engineering decisions. Your PM should be in the room when they're set.

Step 2. Sample the prompt distribution. From 24 hours of production logs: median prompt length 120 tokens, p95 2,100 tokens, p99 8,400 tokens. Median output 180 tokens, p95 700 tokens. I pull 2,000 representative (prompt length, output length) pairs and make those my replay spec.

Step 3. Pick a concurrency sweep. Start at 1, then 2, 4, 8, 16, 32, 64, 128. I expect the knee to be somewhere around 32-64 on this hardware for this model, but I won't know until I measure.

Step 4. Warmup. 200 throwaway requests at each concurrency level to let the caches warm, the CUDA graphs compile, and the scheduler settle. I throw out these numbers.

Step 5. Run. 500 requests per concurrency level, measure TTFT, ITL, and e2e for each. Use the server's actual tokenizer on the client side so token counts match.

Step 6. Plot. Two curves on one plot, concurrency on the x-axis: p95 TTFT (left y-axis) and throughput in tokens/sec (right y-axis). Draw the SLO line. Where TTFT crosses it is your max concurrency. For this setup I'd expect something like c=48 before p95 TTFT hits 500ms. Your number will be different.

Step 7. Compute cost. At c=48, throughput is, say, 1,800 tok/s. A100 40GB at roughly $2/hr means $2 / 3600s × (1s / 1800 tok) × 10⁶ tok/M ≈ $0.31 per million tokens. That's your honest operating cost at SLO.

Now you can reason about knobs. Does INT8 quantization push the knee to c=80 without breaking SLO? That's roughly a 65% cost drop. Does chunked prefill let you handle the 8k-token p99 prompts without stalling? That's protecting your SLO against tail traffic.

That's the shape of honest benchmarking: pin the SLO, replay real traffic, measure distributions, find the knee, cost it out, and iterate.

An implementation sketch

Here is a minimal async load-test harness that hits an OpenAI-compatible endpoint (vLLM, SGLang, TensorRT-LLM, TGI, and most managed providers speak this protocol). It's about as simple as a real load tester gets, and you can extend it with a log-replay prompt sampler, warmup phase, and concurrency sweep.

import asyncio, random, time
from openai import AsyncOpenAI
 
# Point at your engine's OpenAI-compatible endpoint.
client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
 
# Realistic-ish prompt distribution: most short, some long.
def sample_prompt():
    n = int(random.lognormvariate(mu=4.5, sigma=1.2))  # median ~90, long tail
    return "Write a paragraph about: " + ("topic " * n)
 
async def one_request(prompt, gen_tokens=128):
    t0 = time.perf_counter()
    ttft, n = None, 0
    stream = await client.chat.completions.create(
        model="my-model",
        messages=[{"role": "user", "content": prompt}],
        max_tokens=gen_tokens,
        stream=True,
    )
    async for chunk in stream:
        if ttft is None:
            ttft = time.perf_counter() - t0
        n += 1
    e2e = time.perf_counter() - t0
    itl = (e2e - ttft) / max(n - 1, 1)
    return ttft, itl, e2e
 
async def load_test(concurrency=32, n_requests=500, warmup=50):
    sem = asyncio.Semaphore(concurrency)
    async def worker(p):
        async with sem:
            return await one_request(p)
    # Warmup (discard)
    await asyncio.gather(*(worker(sample_prompt()) for _ in range(warmup)))
    # Measure
    return await asyncio.gather(*(worker(sample_prompt()) for _ in range(n_requests)))
 
def pct(xs, p): return sorted(xs)[int(len(xs) * p / 100)]
 
results = asyncio.run(load_test(concurrency=48, n_requests=500))
ttfts = [r[0] * 1000 for r in results]
itls  = [r[1] * 1000 for r in results]
print(f"TTFT p50={pct(ttfts,50):.0f} p95={pct(ttfts,95):.0f} p99={pct(ttfts,99):.0f} ms")
print(f"ITL  p50={pct(itls,50):.1f}  p95={pct(itls,95):.1f}  p99={pct(itls,99):.1f}  ms")

The concurrency is deliberately pinned at 48 to match the worked example's knee. Run at a few different concurrency levels and you have the beginnings of a real sweep. For production-grade tooling, vLLM ships benchmark_serving.py, which does this and more (Poisson arrivals, goodput-at-SLO, prefix-sharing knobs). Anyscale's llmperf is the other defacto tool.

Misconceptions

"Throughput is the number to optimize." Only with a latency budget. A system that does 10k tok/s at p99 TTFT of 10 seconds is not serving; it's queuing. A throughput claim without a latency budget doesn't tell you much. The right question is always "what is the throughput at your SLO?"

"More requests means more throughput." Up to the knee, yes. Past the knee, no. Past the knee, extra requests queue, p95 latency climbs linearly with queue depth, and throughput saturates at whatever the compute ceiling is. Throwing more concurrency at a saturated GPU does not produce more tokens per second; it just produces more unhappy users.

"You can use vendor benchmarks to plan capacity." No. Use them as ballpark sanity checks ("am I within 2× of the claim?"). Your actual capacity number has to come from your hardware, your model config, and your traffic. Everything else is someone else's workload dressed up as yours.

What's next

That closes Arc 6. We built up the serving stack from kernel fusion through benchmarking, and you should now be able to look at a vLLM or SGLang or TensorRT-LLM deployment and reason about where time goes, where memory goes, where money goes, and how to tell if it's working.

The next arc kicks off with Pretraining Data: Mixtures, Curation, and What the Model Sees, which opens up the question of how the weights you've been so carefully serving get made in the first place.

Additional reading (and watching)

• Patel, P., et al. (2023). Splitwise: Efficient generative LLM inference using phase splitting. arXiv:2311.18677. Characterizes the two phases (prefill and decode) with different compute and memory profiles, which is the basis for why TTFT and ITL behave so differently.

• Yu, G.-I., et al. (2022). Orca: A Distributed Serving System for Transformer-Based Generative Models. OSDI 2022. Introduced continuous batching, and, more importantly for this post, argued that "goodput" (throughput at a latency SLO) is the right serving metric.

• Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. The vLLM paper. Its benchmark methodology section is a good template; the paper also quantifies how KV cache memory shapes concurrency ceilings, which is what ultimately sets your $/M tokens.

• OpenAI. Chat Completions API reference. The streaming protocol that most 2026 inference engines speak. Using a common client library (openai) against your own vLLM/SGLang/TGI server is how realistic load testing usually happens in practice.

• vLLM project. benchmark_serving.py in the vLLM repository. The defacto open-source LLM load tester in 2026. Supports Poisson arrival, ShareGPT sampling, goodput-at-SLO, and prefix-sharing simulation.

• Anyscale / Ray Project. LLMPerf: A library for validating and benchmarking LLMs. The other widely used OSS load testing tool. Emphasizes cross-provider correctness checks alongside throughput/latency measurement.

• MLCommons. MLPerf Inference benchmark suite. The industry-standardized cross-vendor benchmark. Useful for comparing hardware platforms; not a substitute for benchmarking your specific workload.

• Agrawal, A., et al. (2024). Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve. OSDI 2024. Quantifies the throughput-vs-latency tradeoff that this post's concurrency-sweep visualization depicts, and introduces chunked prefill as a way to push the knee further right.

• LMSYS and vLLM teams. Inside vLLM: Anatomy of a High-Throughput LLM Inference System. A 2025 deep dive on why the scheduler, KV cache, and batching policies interact to produce the curves we've been measuring. Useful companion reading once you've got benchmark numbers and want to explain them.