Context Windows: Advertised vs. Effective

A context window tells you what the system can accept. It does not tell you how reliably the model will find a fact, combine two distant passages, or summarize everything you supplied. Those tasks need different evaluations.

This distinction matters when choosing between retrieval and a large prompt. A short retrieved context can remove irrelevant material and reduce cost. It can also omit the only passage that answers the question. A full document set avoids that retrieval miss if it fits, while asking more of the model’s ability to find and use evidence.

What the limit means

Check the model and API’s input, output and total-token limits. A headline context length may include output capacity rather than describing an input-only allowance. Tokenization, tool definitions and message framing also consume space.

Within that limit, define effective context for a particular task and acceptance threshold. For example: the greatest tested length where at least 90% of questions are answered correctly across uniformly sampled evidence positions. Change the task, threshold or position distribution and that number may change. Report those choices with the result.

The relevant prerequisites are attention, positional encoding and the retrieval methods earlier in this arc. They explain how the model processes positions and how retrieval selects its inputs.

What softmax does and does not imply

For a standard softmax attention head, weights over the allowed keys sum to one:

The mean weight is therefore $1/N$. But an average says nothing by itself about the weight on the answer-bearing key. Attention can stay sharply concentrated while the number of nearly ignored keys grows.

Give one relevant key logit $\Delta$ and all $N-1$ distractor keys logit zero. Its attention weight is

At a fixed gap, more distractors reduce this weight. To retain weight $p$, the required gap is $\log((N-1)p/(1-p))$. A real model can change its logits, uses many heads and layers, and transforms the resulting representations further. The example isolates competition between one key and equal-logit distractors.

Lost in the middle

Liu and colleagues varied the position of relevant material in multi-document question answering and key–value retrieval. In their evaluated models, performance was often better when evidence appeared near the beginning or end than in the middle. The result concerns task accuracy, not a direct measurement of attention mass.

For an application, move the same answer-bearing passage through the context while holding the question and distractors fixed. Measure the fraction of correct answers over repeated examples at each position.

Needle retrieval and harder tasks

A needle-in-a-haystack test inserts a fact into a long input and asks for it back. Sweep length and depth to expose failures hidden by one average score.

Lexically distinctive needles may be easier to locate than ordinary evidence surrounded by similar material. Multi-hop reasoning and synthesis require additional tests that ask the model to combine or summarize evidence.

RULER adds controlled task families including multiple needles, variable tracking, aggregation and question answering. Its 2024 study found substantial differences between claimed lengths and evaluated performance for several tested models.

How position sampling changes a score

Suppose an authored profile has edge recall 0.85 and a 0.30 middle-depth drop:

This is a function chosen for the example, not a fitted model result. Averaging it at different positions gives different headline numbers.

import numpy as np
 
def recall_at(p, edge=0.85, depth_drop=0.30):
    return edge - depth_drop * (1 - 4 * (p - 0.5) ** 2)
 
depths = np.linspace(0.025, 0.975, 20)
r = recall_at(depths)
print(f"boundary-only:   {r[[0, -1]].mean():.2f}")
print(f"middle-only:     {r[8:12].mean():.2f}")
print(f"uniform average: {r.mean():.2f}")
# boundary-only:   0.82
# middle-only:     0.55
# uniform average: 0.65

For measured results, document the positions, sample count, scoring rule and uncertainty so the average can be interpreted against its sampling protocol.

Retrieval or a longer prompt?

The question is whether the supplied evidence is sufficient and usable. Retrieval can focus a factual question on a few passages; a summary may need coverage across the whole document. Hybrid approaches can retrieve broadly before selecting or summarizing.

Li and colleagues’ 2024 comparison found that sufficiently resourced long-context models outperformed RAG on average in their evaluated setting, while RAG offered lower cost. Their comparison illustrates a quality–cost tradeoff to measure in the intended deployment.

Test both approaches with the same model, questions and source corpus. Account for retrieval misses, answer correctness, source support, latency and input cost. For large corpora that cannot fit, some selection or compression is unavoidable.

Adding material can change performance

Longer prompts can introduce distractors, contradictions or extra dependencies. They can also supply missing evidence and improve answers. “Context rot” is a useful name for observed degradation in particular evaluations; it is not a law that every added token reduces reasoning quality.

Prompt caching changes the work of processing repeated prefixes and, depending on the provider, their billed cost. It does not shorten the logical input or automatically improve the model’s ability to use it. Likewise, compaction can reduce input size while losing the detail needed for a later question.

The next post, context engineering, examines those choices over a growing conversation: what to preserve, summarize, clear and retrieve.

Additional reading (and watching)

• Liu, N. F., et al. Lost in the Middle. Position-sensitive question answering and key–value retrieval.
• Hsieh, C.-P., et al. RULER. Controlled long-context tasks and model-specific evaluation results.
• Kamradt, G. Needle In A Haystack. An implementation of length/depth sweeps.
• Li, Z., et al. Retrieval Augmented Generation or Long-Context LLMs?. An empirical comparison and hybrid routing approach.
• Peng, B., et al. YaRN. A method for extending context through positional interpolation and fine-tuning.
• OpenAI. Prompt caching. Prefix reuse and caching behavior.