Sampling Strategies: Temperature, Top-k, Top-p, and Min-p

There's something easy to miss about how language models generate text. We talk about a model "choosing" a word, as if there's some internal decision-making process. But the model doesn't choose anything. It outputs a vector of numbers, one per vocabulary token, and those numbers are called logits. The model's job is done.

Everything that happens after that point (which token actually gets picked, how creative or conservative the output feels, whether the model repeats itself) is determined by a separate piece of code called the sampling strategy. The sampling strategy is modular. You can swap it out without touching the model. The same model with the same weights will produce wildly different text depending on how you decode those logits.

Prerequisites

You should be comfortable with softmax and probability distributions (Post 1.3). The chain rule of probability matters here conceptually, though we won't need to derive anything. If you've read the next-token prediction post, you already know the model outputs a logit vector at each step. This post is about what happens to that vector after the model is done with it.

The Problem

So we have a problem: the model just produced a probability distribution over something like 128,000 tokens. How do you pick one?

The naive answer is "pick the most likely one," and we'll start there. But you'll quickly see why that's not enough, and why this seemingly small decision has so much leverage over the quality of generated text.

If you've ever opened the docs for an LLM API and seen parameters like temperature, top_p, and top_k alongside your prompt, those are all sampling strategy knobs. They don't change the model, they don't touch the neural network's weights or computation, they only control what happens to the logit vector after the model produces it.

And this applies at every single token. A model generating a 500-token response runs through the sampling strategy 500 times. Each time, it gets a fresh logit vector, applies the sampling pipeline, picks a token, appends it to the context, and runs the model again. The sampling strategy sits in the tightest loop of the generation process.

Let me walk through each one, building from the simplest to the most modern.

Greedy Decoding: Just Pick the Winner

The simplest possible approach is to just take the logit vector, run softmax to get probabilities, and pick the token with the highest probability every single time.

Let's say the model outputs these logits for the next token position:

"the": 5.2   "a": 3.1   "this": 2.8   "my": 2.1   "his": 1.5
"our": 0.9   "her": 0.3  "one": -0.2  "that": -0.8  ...

After softmax, "the" dominates at around 76%. Greedy decoding picks "the" and moves on, and next step it does the same thing: argmax, append, repeat.

This works fine for factual tasks where there's really one correct continuation, like answering "What is the capital of France?" But for anything creative or conversational, greedy decoding produces flat, repetitive text. The model gets stuck in loops because the highest-probability token at each step tends to lead to contexts where that same token is again highest-probability. The output looks like "the the the the" or an endlessly repeating phrase.

The fundamental issue is that text generation is sequential. Each token becomes part of the context for the next. When you always take the argmax, you're committing to a single path through a massive tree of possible continuations. And that single path tends to be boring, because the "safest" next token at every step produces the blandest possible sequence.

There's also a deeper issue. The model was trained on human text, and humans don't write by always choosing the highest-probability next word. Human text has variety, surprises, minor detours. A decoding strategy that never allows any randomness produces text that doesn't actually look like the training distribution. The model "learned" a rich probability landscape, and greedy decoding ignores most of it.

Temperature: Reshaping the Entire Distribution

Temperature is the first real knob. The idea is simple: before you apply softmax, divide every logit by a scalar $T$.

When $T = 1.0$, nothing changes. This is the model's "native" distribution, exactly what it learned during training. But when you change $T$, you reshape the curve.

$T < 1.0$ sharpens. Dividing by a number less than 1 scales the logits up, spreading them further apart. The gaps between high and low logits get amplified, so the winner wins by even more. At $T = 0.1$, the distribution is so sharp it's basically greedy.

$T > 1.0$ flattens. Dividing by a number greater than 1 compresses the logits toward each other. The differences shrink, the probabilities become more uniform, and lower-ranked tokens get a real shot. At $T = 2.0$, even tokens the model considered unlikely have a non-trivial chance.

Let me make this concrete with our logit vector. At $T = 1.0$, the top token ("the") has about 76% probability and the second token ("a") has about 9%. At $T = 0.5$, "the" jumps to roughly 97% and "a" drops to around 1.5%. The distribution collapses onto the leader. At $T = 1.5$, "the" falls to about 55% and "a" rises to around 13%. The tail tokens become plausible choices.

Think of temperature as a confidence dial. Low temperature means "the model is very sure, just go with its top pick." High temperature means "the model has opinions but let's hear from the minority voices too."

The name "temperature" comes from statistical mechanics, if you're curious. In a Boltzmann distribution, temperature controls how spread out particles are across energy states. High temperature means particles explore more states. Low temperature means they settle into the lowest-energy state. The analogy to token sampling is direct: high temperature explores more of the vocabulary, low temperature settles on the most likely tokens.

One thing to notice: temperature doesn't change the ranking of tokens. If "the" was most likely before temperature scaling, it's still most likely after. Temperature only changes how much more likely it is than the alternatives. It stretches or compresses the gaps.

What happens at the extremes? As $T \to 0$, the distribution approaches a one-hot vector: all the probability mass collapses onto the single highest-logit token. You get greedy decoding. As $T \to \infty$, all logits get divided down toward zero, the exponents all approach 1, and you get a uniform distribution. Every token is equally likely. Neither extreme is useful for generation, but they're helpful for building intuition about the knob.

Top-k: A Hard Cutoff

Top-k sampling, introduced by Fan et al. in their 2018 story generation paper, takes a blunter approach. Instead of reshaping the distribution, you just chop off the tail. Keep the $k$ highest-probability tokens, zero out everything else, and renormalize so the remaining probabilities sum to 1.

With $k = 5$ applied to our example, we'd keep "the", "a", "this", "my", and "his". Everything else goes to zero. Then we renormalize those five so they sum to 1, and sample from that set.

Top-k is easy to understand and implement. But it has a real problem: $k$ is the same regardless of context. Sometimes the model is very confident and the top 3 tokens account for 95% of the probability mass. In that case, $k = 50$ keeps a bunch of garbage tokens that the model actively doesn't want. Other times the model is genuinely uncertain across dozens of plausible continuations, and $k = 50$ might cut off reasonable options.

The right value of $k$ depends on the shape of the distribution at each step, and top-k can't adapt to that. It treats a peaked distribution (the model knows exactly what comes next) the same way it treats a flat one (the model is genuinely unsure). This mismatch is the core limitation.

Top-p (Nucleus Sampling): Adapting to the Distribution

Top-p sampling, proposed by Holtzman et al. in 2019, solves exactly this problem. Instead of keeping a fixed number of tokens, you keep tokens until their cumulative probability reaches a threshold $p$.

Here's how it works. Sort the tokens by probability (descending). Walk down the sorted list, accumulating probability as you go. Stop when the running total hits or exceeds $p$. Zero out everything you didn't include. Renormalize.

With $p = 0.9$ on our example at $T = 1.0$, we'd sort by probability and start accumulating: "the" (0.76), plus "a" (0.09, cumulative 0.85), plus "this" (0.07, cumulative 0.92), and we stop. Three tokens cover the nucleus because the model was already confident. When it's uncertain the same threshold might pull in 30 or more tokens. The cutoff adapts to the shape of the distribution.

This is why top-p is also called "nucleus" sampling. You're sampling from the nucleus of the distribution, the core set of tokens that together represent most of the model's probability mass.

The Holtzman paper is worth reading, by the way. It has a great analysis of what they call "degenerate" text: the repetitive, incoherent output you get from greedy and beam search decoding. Their key finding was this. When they measured the probability that a language model assigned to actual human-written text, the human-chosen tokens were often not in the very top of the model's distribution. Humans regularly pick tokens that the model considers somewhat unlikely. The most probable sequence according to the model isn't the most human-sounding sequence.

Top-p captures this by keeping a dynamically-sized set of "reasonable" tokens rather than always going with the safest choice. It preserves the model's uncertainty when it exists, and collapses to near-greedy when the model is confident.

Min-p: Scaling with the Leader

Min-p is a newer approach that deserves more attention. The idea is clean: keep any token whose probability is at least $p \times p_\text{max}$, where $p_\text{max}$ is the probability of the most likely token.

So if the top token has probability 0.46 and you set min-p to 0.1, you keep everything with probability at least 0.046. If the top token only has probability 0.17 (the model is very uncertain), the threshold drops to 0.017, which lets most of the distribution through.

Min-p scales naturally with the model's confidence. When the model is sure, the threshold is high and you get a tight distribution. When it's unsure, the threshold is low and you preserve the spread. Unlike top-p, you don't need to sort the entire vocabulary and walk through it accumulating probabilities. You just compare each token's probability against a single threshold. It's computationally cheaper and arguably more intuitive.

With top-p, the threshold is about the cumulative mass of the distribution. With min-p, the threshold is about each token's relationship to the leading candidate. "Keep anything that's at least 10% as likely as the best option" is an easier handle to reason about than "keep tokens until we've covered 90% of the probability." Hewitt et al. give a more formal framing, treating these truncation methods as correcting the long-tail smoothing that shows up in language model training.

There are other members of this family. Locally typical sampling keeps tokens whose information content is close to the entropy of the full distribution, rather than the most likely ones. It's less widely deployed than top-p or min-p, but it's the same kind of move: filter the distribution based on a shape-aware criterion, then renormalize and sample.

Playing With the Distribution

I built a widget so you can see all of this in action. The bars show the probability distribution over 15 example tokens. Drag the sliders to apply different strategies and watch the bars update. Blue bars are tokens that survived the filter. Gray bars got zeroed out. Hit "Sample!" to draw a random token weighted by the current distribution.

A few things worth trying in there. Set temperature to 0.3 and notice how "the" completely dominates. Then crank it to 1.8 and see the distribution flatten out. Now set temperature back to 1.0 and turn on top-k at 5, then switch to top-p at 0.9 instead and compare. You'll notice top-p keeps fewer tokens when the distribution is sharp.

Also try min-p at 0.1 with temperature 1.0. Compare what it keeps versus top-p at 0.9. They often agree on the "obvious" tokens but disagree on the marginal ones, and that disagreement is where the choice of strategy matters.

Combining Strategies: The Pipeline

In practice, you almost never use just one strategy. Most inference engines apply them in a pipeline:

logits → temperature → top-k or top-p → min-p → sample

Temperature comes first because it reshapes the raw logits before any truncation happens. This matters. If you apply top-p and then temperature, you're sharpening or flattening a distribution that's already been truncated, which gives different results than sharpening first and then truncating. The convention is temperature first, truncation second.

A common combination for conversational AI is temperature 0.7 with top-p 0.9. This sharpens the distribution a bit (you're slightly more conservative than the model's training distribution) and then trims the tail to prevent truly bizarre tokens from sneaking through. For creative writing, you might use temperature 1.0 or even 1.2 with top-p 0.95, preserving more of the model's uncertainty.

For code generation, lower temperatures usually work better (0.2 to 0.5). Code has stricter syntax constraints, and you generally want the model to commit to its best guess rather than exploring unlikely alternatives. Some people use greedy decoding for code, but a small amount of randomness can help the model avoid getting stuck in repetitive patterns.

One subtlety: when you combine temperature with top-p, the effective number of tokens in the nucleus changes. A lower temperature concentrates probability mass on fewer tokens, so even with a generous top-p like 0.95, you might end up sampling from only 5 or 6 tokens. Conversely, high temperature spreads mass across many tokens, making the top-p cutoff include a larger set. The two parameters interact, and tuning them independently can give you unexpected results.

This is one reason why practitioners often pick a "recipe" and stick with it rather than tweaking individual parameters. Temperature 0.7, top-p 0.9 is a reasonable default for most conversational tasks. If the output feels too conservative, bump the temperature up a notch. If it's too chaotic, bring it down. You rarely need to fiddle with more than one knob at a time.

A Worked Example

Let me trace through the full pipeline with concrete numbers. Starting logits:

Step 1: Temperature = 0.7. We divide each logit by 0.7:

Step 2: Softmax. The sharpened logits produce probabilities where "the" dominates: about 90.7% for "the", 4.5% for "a", 2.9% for "this", and the rest quickly falling off.

Step 3: Top-p = 0.9. We accumulate probabilities starting from the top. "the" alone gives us 90.7%, which already crosses the 0.9 threshold. So the nucleus is just {"the"}. Everything else gets zeroed out. Sharpening hard with $T = 0.7$ made the top token already account for most of the mass by itself.

Step 4: Renormalize and sample. The surviving token gets its probability rescaled to 1.0. At this setting we'll always draw "the", which is a bit like falling back to greedy.

Now compare: with the same top-p of 0.9 but $T = 1.5$ instead of 0.7, "the" drops to about 55% after softmax. The distribution is flatter, so no single token dominates, and the nucleus pulls in "a", "this", and a few more tokens before it reaches 90%. Sampling from that set gives you real variation.

So it's the same model and the same logits, with completely different behavior, just from changing one number in the pipeline.

Repetition Penalty: Fixing the Loop Problem

There's one more knob that isn't strictly a sampling strategy but shows up in every inference engine: the repetition penalty. Even with temperature and top-p, models can fall into repetitive patterns.

The fix is to look at which tokens have appeared recently (in the last $n$ tokens) and penalize them. The simplest version divides or subtracts from the logit of any recently-used token before applying softmax. This pushes the model away from tokens it has already generated, breaking the feedback loops that cause repetition.

Some implementations use a multiplicative penalty (divide the logit by a factor like 1.2), others use additive (subtract a fixed value). There are also frequency and presence penalties that scale with how many times a token has appeared. OpenAI's API exposes both frequency_penalty and presence_penalty as separate parameters, which is a common pattern.

The details vary by provider, but the core idea is the same: modify the logits based on what the model has already said, before running the sampling pipeline. Conceptually, repetition penalty is just another logit transformation that slots into the pipeline before temperature and truncation.

Repetition penalty is a blunt instrument. It penalizes tokens regardless of whether repeating them would be natural. Words like "the" or "is" appear many times in normal text, and penalizing them too aggressively makes the output sound unnatural. Better implementations use a sliding window (only penalize based on the last $n$ tokens) and apply lighter penalties to common function words. Getting the tuning right is tricky, which is why most API providers handle it for you and don't even expose it as a parameter.

Misconceptions

"Temperature 0 is deterministic." Sort of. In practice, setting temperature to 0 (or very close to 0) gives you greedy decoding, which is deterministic. But mathematically, $T = 0$ makes the softmax undefined since you'd be dividing by zero. What actually happens is that the implementation clamps $T$ to some small epsilon and the result is so sharply peaked that it's functionally argmax. The model itself isn't "confident" in any meaningful sense. You're just ignoring most of the distribution.

"Top-p is always better than top-k." Top-p adapts to the distribution shape, which is usually what you want. But top-k has a useful property: it gives you a hard upper bound on how many tokens the model can choose from. If you're running constrained generation or care about latency in structured output pipelines, top-k's predictability can matter. In practice most people use top-p, but top-k isn't obsolete.

"Higher temperature = more creative." Higher temperature means more random, which is not the same thing. At $T = 2.0$, the model is effectively ignoring much of what it learned. You might get surprising token choices, but "surprising" and "creative" are not synonyms. Good creative output usually comes from moderate temperatures (0.8 to 1.2) combined with appropriate truncation.

What's next

Sampling is one of those topics that seems simple on the surface but has real depth once you start thinking about distribution shapes and how strategies compose. The model gives you logits. What you do with them is up to you, and a handful of numbers in the decoding step can completely change the character of the output without touching the weights.

The next post covers speculative decoding, a clever trick where a small draft model proposes several tokens at once and the big model verifies them in a single pass, making generation faster without changing what gets generated.

Additional reading (and watching)

• Fan, A., Lewis, M., & Dauphin, Y. (2018). Hierarchical Neural Story Generation. ACL 2018. Introduces top-k sampling as a fix for the flatness and repetition of beam search in open-ended generation.

• Holtzman, A., Buys, J., Du, L., Forbes, M., & Choi, Y. (2019). The Curious Case of Neural Text Degeneration. ICLR 2020. The nucleus (top-p) sampling paper; the diagnosis of why maximum-likelihood decoding fails is as good as the fix.

• Hewitt, J., Manning, C. D., & Liang, P. (2022). Truncation Sampling as Language Model Desmoothing. EMNLP Findings 2022. Frames top-p and its cousins as fixing the long-tail "smoothing" that language model training introduces.

• Meister, C., Pimentel, T., Wiher, G., & Cotterell, R. (2023). Locally Typical Sampling. TACL 2023. A cousin of min-p that draws from tokens whose information content is close to the expected value of the full distribution.

• Touvron, H., et al. (2023). Llama 2: Open Foundation and Fine-Tuned Chat Models. arXiv:2307.09288. Section 4 documents real sampling-parameter choices (temperature 1.0, top-p 0.9) used for RLHF-tuned open models, which became a community default.

• OpenAI. (2025). Responses API reference: sampling parameters. Canonical provider documentation for temperature, top_p, frequency_penalty, and presence_penalty as of 2026.

• Kwon, W., et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023. The vLLM paper; section 4 describes how sampling lives in the serving engine alongside the KV cache and continuous batcher.