Calibration and Abstention

When a model assigns 80% confidence to an answer, does that number predict how often answers like it are correct? That is a different question from whether the model sounds certain, and a different question from its overall accuracy.

Calibration compares stated probabilities with observed outcomes. Abstention uses a confidence signal to decide which answers to withhold or route elsewhere. They are related, but improving one does not automatically improve the other.

Cross-entropy and loss supplies the probability arithmetic, and sampling strategies introduces temperature. Here temperature is a possible post-hoc calibration transform; we will keep that separate from changing a generation policy.

Calibration describes groups of predictions

Let $C$ be a confidence score for a defined correctness event $Y\in\{0,1\}$. Perfect calibration on the target distribution means

Among many predictions assigned 0.8 confidence, about 80% should be correct. A finite group will fluctuate. One wrong answer at 80% confidence does not by itself demonstrate miscalibration.

Calibration is also separate from accuracy. A predictor that assigns 0.7 to every answer can be calibrated if its accuracy is 70%. It still has no confidence ranking with which to identify its likely mistakes. A more accurate model can be poorly calibrated if it systematically assigns probabilities that are too high or too low.

First define what the confidence refers to. For multiple choice, it might be the correctness of the selected option. For a long response, it could refer to a particular factual claim or a specified rubric outcome. Open-ended tasks can have defined events, but “confidence in this whole answer” is ambiguous without such a definition.

Measuring calibration with bins

A reliability diagram groups predictions by confidence and compares each group’s mean confidence with its observed accuracy. Plot the actual mean, not automatically the midpoint of the interval containing it. Show counts because a bin with five examples provides different evidence from a bin with five thousand.

Expected Calibration Error, or ECE, summarizes the absolute discrepancies:

Empty bins contribute zero. The usual empirical estimate depends on the number and placement of bins, and finite samples introduce error. Equal-width and equal-count bins make different tradeoffs. A zero binned ECE on a finite test does not prove perfect calibration: averaging can conceal overconfidence and underconfidence within a bin.

In the constructed example, three groups each contain ten predictions. Their mean confidences are 0.3, 0.6, and 0.9, while their observed accuracies are 0.3, 0.5, and 0.7. ECE is $(0+0.1+0.2)/3=0.1$.

Use reliability groups to locate errors that an aggregate ECE hides. Negative log-likelihood or Brier score can supplement this binned summary.

Which confidence signal are we evaluating?

A language model predicts tokens. The probability of a chosen token is not automatically the probability that the answer containing it is true. The same answer can have several wordings; a common false claim can be highly predictable. Multiplying token probabilities gives the probability of that exact sequence under the model, not a factual verification score.

The length-normalized geometric mean, $\exp(\frac1T\sum_t\log p_t)$, is useful for comparing sequence likelihoods under a specified tokenization. It remains a likelihood-derived feature. If it is used as answer confidence, test its relationship to correctness rather than naming it “confidence” and assuming the relationship.

Likewise, probabilities renormalized over four offered answer options concern those candidates. They differ from the full vocabulary distribution and from the probability of an unconstrained generated answer.

Verbalized confidence is a generated answer to a request for a probability. Tian et al. found that this signal often gave better calibration than conditional token probabilities for several models tuned with human feedback on the QA datasets they studied. They elicited it through prompting, without training a separate confidence model. Test that relationship on the task where you intend to use it.

Training can change calibration

Cross-entropy encourages accurate conditional token probabilities under its training distribution, but a finite model trained imperfectly on a different distribution has no automatic calibration guarantee for answer correctness.

Instruction tuning and preference optimization change those probabilities and the generated responses. In the GPT-4 technical report’s MMLU comparison, post-training reduced calibration. Check the direction and size of the effect for the model and task you use.

Prompting changes the conditional distribution too. Few-shot examples, answer format, and chat instructions can affect both correctness and confidence. Measure calibration with the prompt and decoding procedure you intend to use. Reusing a calibration transform after changing that procedure is an assumption that needs checking.

Temperature scaling

For a classifier with logits $z$, temperature scaling uses one fitted positive number $T$:

Larger $T$ flattens a nonuniform distribution; smaller $T$ sharpens it. Every positive $T$ preserves the argmax within an example. It therefore leaves deterministic top-class predictions and their accuracy unchanged, while changing confidence.

A standard fitting procedure minimizes negative log-likelihood on a held-out calibration set. Evaluate the fitted transform on a separate test set. A single temperature may be inadequate under distribution shift or class-dependent errors.

The following implementation uses explicit logits and labels. Optimizing log-temperature keeps $T$ positive. The interval is a declared numerical search bound; a fit at its edge deserves inspection.

This requires logits or an equivalent complete probability distribution. For complete probabilities, $q_j\propto p_j^{1/T}$ gives the same transform. A truncated list of top-token probabilities generally cannot recover the exact normalization over omitted alternatives. API availability varies by model and endpoint; inspect what the interface actually returns.

Abstention depends on ranking as well as calibration

Choose a threshold $\tau$ and answer only when confidence is at least $\tau$. Coverage is the fraction answered. Selective accuracy is correctness among those answers:

If no answers remain, accuracy is undefined. It is not 100%.

Abstention works well when confidence ranks likely correct answers above likely incorrect ones. A strictly increasing remapping of a scalar confidence score preserves that ranking and all attainable coverage–accuracy pairs, even if it improves calibration substantially. Binary temperature scaling has this monotonic property for top-class confidence; multiclass scaling can change the ordering of confidence across examples, despite preserving each example’s winning class.

Empirical accuracy need not rise at every threshold. A confidently wrong answer can survive after less confident correct answers are removed. Select a threshold on development data using the costs of wrong answers and deferrals, then report coverage and error rate on a separate test set. Include what happens to deferred requests: a human review queue or fallback system has its own limits and costs.

An ECE calculation with explicit edge cases

Other confidence signals include disagreement across sampled answers and uncertainty over their meanings. Agreement is not proof: several samples can repeat the same misconception, and their errors can be correlated.

A useful report names the correctness event, confidence signal, calibration data, test distribution, and abstention policy. Then it shows both probability quality and the outcomes of the decisions those probabilities support.

Additional reading

• Guo et al., On Calibration of Modern Neural Networks. Reliability measurement and temperature scaling fitted by likelihood.
• Tian et al., Just Ask for Calibration. Prompted verbal confidence for models tuned with human feedback.
• OpenAI, GPT-4 Technical Report. The model-specific calibration comparison in Figure 8.
• Geifman and El-Yaniv, Selective Classification for Deep Neural Networks. Prediction with a reject option and coverage–risk tradeoffs.