Sampling Strategies: Temperature, Top-k, Top-p, and Min-p

A forward pass produces logits, one score per vocabulary token. Decoding code turns those scores into the next token.

The sampling strategy selects a token from those scores. You can change it without retraining the model, but output quality and repetition also depend on the learned distribution and the prompt. Here we will compute how temperature and truncation change one example distribution.

Prerequisites

Softmax and probability distributions (Post 1.3) explains how logits become probabilities. Next-token prediction describes how the model produces those logits; this post compares ways to select a token from them.

The Problem

So we have a problem: the model just produced a probability distribution over something like 128,000 tokens. How do you pick one?

Greedy decoding picks the highest-scoring token. Sampling instead draws from a distribution, optionally reshaped by temperature or truncated to a subset of tokens.

If you've ever opened the docs for an LLM API and seen parameters like temperature, top_p, and top_k alongside your prompt, those are all sampling strategy knobs. They don't change the model, they don't touch the neural network's weights or computation, they only control what happens to the logit vector after the model produces it.

And this applies at every single token. A model generating a 500-token response runs through the sampling strategy 500 times. Each time, it gets a fresh logit vector, applies the sampling pipeline, picks a token, appends it to the context, and runs the model again.

Greedy Decoding: Just Pick the Winner

Greedy decoding selects the largest logit. Softmax preserves the ranking, so computing probabilities is unnecessary when all you need is the argmax.

Let's say the model outputs these logits for the next token position:

"the": 5.2   "a": 3.1   "this": 2.8   "my": 2.1   "his": 1.5
"our": 0.9   "her": 0.3  "one": -0.2  "that": -0.8  "it": -1.4

After softmax, "the" dominates at around 76.2%. Greedy decoding picks "the" and moves on, and next step it does the same thing: argmax, append, repeat.

This works fine for factual tasks where there's really one correct continuation, like answering "What is the capital of France?" For some open-ended tasks, greedy decoding can produce repetitive or bland text. The model gets stuck in loops because the highest-probability token at each step tends to lead to contexts where that same token is again highest-probability. The output looks like "the the the the" or an endlessly repeating phrase.

The fundamental issue is that text generation is sequential. Each token becomes part of the context for the next. When you always take the argmax, you're committing to a single path through a massive tree of possible continuations. That path can miss useful alternatives, and locally choosing the most likely token does not maximize the probability of the complete sequence.

Temperature: Reshaping the Entire Distribution

Before applying softmax, temperature scaling lets you divide every logit by a scalar $T$.

When $T = 1.0$, nothing changes. This is the model's "native" distribution, exactly what it learned during training. But when you change $T$, you reshape the curve.

$T < 1.0$ sharpens. Dividing by a number less than 1 scales the logits up, spreading them further apart. The gaps between high and low logits get amplified, so the winner wins by even more. At $T = 0.1$, the distribution is so sharp it's basically greedy.

$T > 1.0$ flattens. Dividing by a number greater than 1 compresses the logits toward each other. The differences shrink, the probabilities become more uniform, and lower-ranked tokens get a real shot. At $T = 2.0$, even tokens the model considered unlikely have a non-trivial chance.

For our logit vector, at $T = 1.0$, the top token ("the") has about 76.2% probability and the second token ("a") has about 9%. At $T = 0.5$, "the" jumps to roughly 97.4% and "a" drops to around 1.5%. The distribution collapses onto the leader. At $T = 1.5$, "the" falls to about 55.2% and "a" rises to around 13.6%. The tail tokens become plausible choices.

Temperature changes the sampling distribution; it does not change what the model knows or make its confidence calibrated.

The name "temperature" comes from statistical mechanics, if you're curious. In a Boltzmann distribution, temperature controls how spread out particles are across energy states. High temperature means particles explore more states. Low temperature means they settle into the lowest-energy state. The analogy to token sampling is direct: high temperature explores more of the vocabulary, low temperature settles on the most likely tokens.

One thing to notice: temperature doesn't change the ranking of tokens. If "the" was most likely before temperature scaling, it's still most likely after. Temperature only changes how much more likely it is than the alternatives. It stretches or compresses the gaps.

What happens at the extremes? As $T \to 0$, mass concentrates on the maximum-logit tokens. With a unique maximum, the limit is a one-hot vector; exact ties share mass equally. An implementation that special-cases zero as argmax also needs a tie-breaking rule. As $T \to \infty$, all logits get divided down toward zero, the exponents all approach 1, and you get a uniform distribution. Every token is equally likely.

Top-k: A Hard Cutoff

Top-k sampling, introduced by Fan et al. in their 2018 story generation paper, takes a blunter approach. Instead of reshaping the distribution, you just chop off the tail. Keep the $k$ highest-probability tokens, zero out everything else, and renormalize so the remaining probabilities sum to 1.

With $k = 5$ applied to our example, we'd keep "the", "a", "this", "my", and "his". Everything else goes to zero. Then we renormalize those five so they sum to 1, and sample from that set.

But it has a real problem: $k$ is the same regardless of context. Sometimes the model is very confident and the top 3 tokens account for 95% of the probability mass. In that case, $k = 50$ retains many low-probability tokens. Other times the model is genuinely uncertain across dozens of plausible continuations, and $k = 50$ might cut off reasonable options.

Top-p (Nucleus Sampling): Adapting to the Distribution

Top-p sampling, proposed by Holtzman et al. in 2019, solves exactly this problem. Instead of keeping a fixed number of tokens, you keep tokens until their cumulative probability reaches a threshold $p$.

Here's how it works. Sort the tokens by probability (descending). Walk down the sorted list, accumulating probability as you go. Stop when the running total hits or exceeds $p$. Zero out everything you didn't include. Renormalize.

With $p = 0.9$ on our example at $T = 1.0$, we'd sort by probability and start accumulating: "the" (0.76), plus "a" (0.09, cumulative 0.85), plus "this" (0.07, cumulative 0.92), and we stop. Three tokens cover the nucleus because the model was already confident. When it's uncertain the same threshold might pull in 30 or more tokens. The cutoff adapts to the shape of the distribution.

This is why top-p is also called "nucleus" sampling. You're sampling from the nucleus of the distribution, the core set of tokens that together represent most of the model's probability mass.

Holtzman et al. analyzed repetitive, incoherent output from greedy and beam search decoding. When they measured the probability that a language model assigned to actual human-written text, the human-chosen tokens were often not in the very top of the model's distribution. Humans regularly pick tokens that the model considers somewhat unlikely. The most probable sequence according to the model isn't the most human-sounding sequence.

Min-p: Scaling with the Leader

Min-p keeps any token whose probability is at least $p \times p_\text{max}$, where $p_\text{max}$ is the probability of the most likely token.

So if the top token has probability 0.46 and you set min-p to 0.1, you keep everything with probability at least 0.046. If the top token only has probability 0.17 (the model is very uncertain), the threshold drops to 0.017, which lowers the cutoff. How many tokens survive still depends on the other probabilities.

Unlike top-p, you don't need to sort the entire vocabulary and walk through it accumulating probabilities. You just compare each token's probability against a single threshold. A direct threshold avoids a full sort, although optimized implementations and hardware determine the actual sampling cost.

With top-p, the threshold is about the cumulative mass of the distribution. With min-p, the threshold is about each token's relationship to the leading candidate. At min-p 0.1, a token must be at least 10% as likely as the leader; at top-p 0.9, the retained set must cover at least 90% of the probability mass. Hewitt et al. give a more formal framing, treating these truncation methods as correcting the long-tail smoothing that shows up in language model training.

There are other members of this family. Locally typical sampling keeps tokens whose information content is close to the entropy of the full distribution, rather than the most likely ones. It's less widely deployed than top-p or min-p, but it's the same kind of move: filter the distribution based on a shape-aware criterion, then renormalize and sample.

Playing With the Distribution

The bars show the probability distribution over 15 example tokens. Drag the sliders to apply different strategies and watch the bars update. Blue bars are tokens that survived the filter. Excluded tokens have zero probability. Hit "Sample!" to draw a random token weighted by the current distribution.

Set temperature to 0.3 and notice how "the" completely dominates. Then crank it to 1.8 and see the distribution flatten out. Now set temperature back to 1.0 and turn on top-k at 5, then switch to top-p at 0.9 instead and compare. You'll notice top-p keeps fewer tokens when the distribution is sharp.

Also try min-p at 0.1 with temperature 1.0. Compare what it keeps versus top-p at 0.9. They often agree on the "obvious" tokens but disagree on the marginal ones, and that disagreement is where the choice of strategy matters.

Combining Strategies: The Pipeline

Strategies can be composed. One possible ordering is:

logits → temperature → top-k or top-p → min-p → sample

Temperature comes first because it reshapes the raw logits before any truncation happens. If you apply top-p and then temperature, you're sharpening or flattening a distribution that's already been truncated, which gives different results than sharpening first and then truncating. The convention is temperature first, truncation second.

For an experiment, compare temperature 0.7 with top-p 0.9 against temperature 1.2 with top-p 0.95. The first concentrates probability on fewer tokens; the second retains a broader set. Check the retained tokens as well as the sampled output.

When you combine temperature with top-p, the effective number of tokens in the nucleus changes. A lower temperature concentrates probability mass on fewer tokens, so even with a generous top-p like 0.95, you might end up sampling from only 5 or 6 tokens. Conversely, high temperature spreads mass across many tokens, making the top-p cutoff include a larger set. The two parameters interact, and tuning them independently can give you unexpected results.

A Worked Example

Starting logits:

Step 1: Temperature = 0.7. We divide each logit by 0.7:

Step 2: Softmax. The sharpened logits produce probabilities where "the" dominates: about 90.7% for "the", 4.5% for "a", 2.9% for "this", and the rest quickly falling off.

Step 3: Top-p = 0.9. We accumulate probabilities starting from the top. "the" alone gives us 90.7%, which already crosses the 0.9 threshold. So the nucleus is just {"the"}. Everything else gets zeroed out. Sharpening hard with $T = 0.7$ made the top token already account for most of the mass by itself.

Step 4: Renormalize and sample. The surviving token gets its probability rescaled to 1.0. At this setting we'll always draw "the", which is a bit like falling back to greedy.

Now compare: with the same top-p of 0.9 but $T = 1.5$ instead of 0.7, "the" drops to about 55.2% after softmax. The distribution is flatter, so no single token dominates, and the nucleus pulls in "a", "this", and a few more tokens before it reaches 90%. Sampling from that set gives you real variation.

Repetition Penalty: Fixing the Loop Problem

There's one more knob that isn't strictly a sampling strategy but is supported by many sampling implementations: the repetition penalty. Even with temperature and top-p, models can fall into repetitive patterns.

A repetition penalty lowers logits for tokens that have already appeared, before sampling. Implementations differ in which prior tokens they consider and how they lower the scores; the next example distinguishes positive and negative logits.

A common multiplicative rule divides positive logits by a factor above one and multiplies negative logits by it; dividing a negative logit would incorrectly reward it. Additive penalties subtract a value instead. Frequency penalties scale with occurrence count; presence penalties apply once when a token has appeared. OpenAI Chat Completions exposes these parameters for supported models; they are not universal Responses API parameters.

Repetition penalty is a blunt instrument. It penalizes tokens regardless of whether repeating them would be natural. Words like "the" or "is" appear many times in normal text, and penalizing them too aggressively makes the output sound unnatural. Some implementations allow a recent-token window. Any penalty can suppress legitimate repetition, so test it on the task instead of assuming that a stronger penalty improves quality.

Misconceptions

"Temperature 0 is deterministic." The softmax formula is undefined at $T = 0$. Implementations commonly special-case zero as argmax, with a tie-breaking rule; epsilon-clamping is a different approximation. Argmax is deterministic for fixed logits and tie handling, but repeated API calls can still differ because execution and serving behavior may change the logits.

"Top-p is always better than top-k." Top-p adapts the retained count to distribution shape. Top-k gives a hard upper bound on how many tokens the model can choose from. That cap may be useful for a particular sampling policy. Neither method has a universal quality advantage, and truncation does not avoid computing the model’s logits.

"Higher temperature = more creative." Higher temperature means more random, which is not the same thing. Increasing temperature flattens the learned distribution; it does not erase or improve the model’s knowledge. You might get surprising token choices, but "surprising" and "creative" are not synonyms. Evaluate candidate settings on the writing task; there is no model-independent temperature range that guarantees creativity.

What's next

The next post covers speculative decoding, where a small draft model proposes several tokens at once and the big model verifies them in a single pass, potentially reducing latency while preserving the target sampling distribution.

Additional reading (and watching)

• Fan, A., Lewis, M., & Dauphin, Y. (2018). Hierarchical Neural Story Generation. ACL 2018. Introduces top-k sampling as a fix for the flatness and repetition of beam search in open-ended generation.

• Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019). The Curious Case of Neural Text Degeneration. ICLR 2020. The nucleus (top-p) sampling paper; the diagnosis of why maximum-likelihood decoding fails is as good as the fix.

• Nguyen, M. N., et al. (2025). Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs. Defines and evaluates the relative-to-maximum probability threshold.

• Hewitt, J., Manning, C. D., & Liang, P. (2022). Truncation Sampling as Language Model Desmoothing. EMNLP Findings 2022. Frames top-p and its cousins as fixing the long-tail "smoothing" that language model training introduces.

• Meister, C., Pimentel, T., Wiher, G., & Cotterell, R. (2023). Locally Typical Sampling. TACL 2023. A cousin of min-p that draws from tokens whose information content is close to the expected value of the full distribution.

• OpenAI. (2025). Responses API reference: sampling parameters. Provider documentation for supported Responses parameters; model support varies.

• Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. The vLLM paper; section 4 describes how sampling lives in the serving engine alongside the KV cache and continuous batcher.