Context Windows: Advertised vs. Effective

Every frontier lab in 2026 has a context-length number on its datasheet. Gemini 2.5 Pro quotes 1M tokens. Claude Opus went from 200k to a 1M beta. GPT-5 accepts up to 272k input tokens (400k total with output). Qwen and Llama variants advertise 1M windows via extended RoPE tricks. These numbers are real. You can genuinely feed that many tokens to the model without an error.

But using 1M tokens is not the same as recalling from them. The spec window and the model's actual working-memory capacity are two different things, and the gap between them has widened as vendors raced to stretch the advertised number. This post is about that gap, how we measure it, and why a 4k-token RAG setup still beats a 128k stuff-the-context approach on most factual question-answering tasks.

I spent a while confused about this. When Claude went from 100k to 200k, I assumed the useful-context number had roughly doubled. When Gemini 1.5 shipped 1M, I assumed the useful number was somewhere around 1M. Both times I was wrong in the same direction. The advertised number moved; the number of tokens the model could actually reason across moved a lot less. Working through the literature on long-context evaluation took me to a mental model I want to hand over here.

I'll try to keep this post about measurement and the fundamental limit. The next post in this arc is where we get into what to do about it.

Two numbers that don't match

When a provider says "1M context window," they mean the model accepts up to a million tokens in its input without truncation or architectural failure. The positional encoding covers that range, the attention kernels run, nothing crashes.

What they don't mean: "the model can reliably find, combine, and reason about facts anywhere inside that million tokens."

Those are different claims, and the second one is always weaker.

The first claim is a property of the serving stack. The second is a property of the model's learned attention patterns, its working memory, and the tasks you care about. The serving-stack number is easy to quote, because it's a single integer you can check by feeding in a long input and seeing what happens. The task-dependent number is hard to even define, let alone benchmark. It's a surface in (task, degradation tolerance, document structure) space, not a single number.

So the datasheet gets the easy number, and the reader has to do the work to figure out what the real number is for their use case.

This mismatch between what's easy to measure and what matters shows up everywhere in ML, but it's unusually stark here because the advertised number has become a headline feature. When a lab ships "2M context," the press release is about 2M, not about the task-weighted effective length at 85% of short-context performance. The reader learns the big number and nothing else. The question "can we just use the 1M model and skip RAG?" comes up a lot, and the answer always starts with this same preamble.

I think of it as two separate axes. One is spec, what the system admits. The other is effective recall, the length at which the model is still a useful reasoner.

The hero viz at the top is literally that: the spec axis explodes through orders of magnitude while the effective-recall axis creeps. By the time you hit 1M, they're off by more than an order of magnitude.

A quick caveat on what I mean by "useful reasoner." I'm not claiming a model does nothing useful with its 800-thousandth token. It very much does. But "do something with this token" and "integrate this token reliably into a multi-step answer about a hundred other tokens scattered across the prompt" are different bars. The spec guarantees the first. Nothing guarantees the second. The two are easy to conflate when reaching for context length as a design lever, and then the long-context version of an app turns out flakier than the short-context one.

The rest of this post is about making that vague statement precise.

Prerequisites

Before the measurement part, a brief list of what we're assuming from earlier posts:

• Attention computes a weighted average over every position in context. Every query vector dot-products with every key vector. That's arc 4.
• Positional encoding is how a model knows token $i$ is different from token $j$. RoPE, ALiBi, and friends. Arc 4.
• Long-context training is expensive and rare; many "1M" models are trained at 32k or 128k and extended afterwards. Arc 7.
• Retrieval gives you a way to pick a small number of relevant passages out of a huge corpus. The previous post.

If you have a rough feel for each, you're ready.

One small piece of math that will help below. The total attention mass from any query position sums to 1, because the scores pass through a softmax. That's not a minor detail. It means if you expand the number of keys from 4k to 128k, the average attention weight per key has to fall by a factor of 32 just to keep the total at 1. In formula form, the mean attention per key is $1/N$ where $N$ is the context length. The model's capacity to redistribute that mass onto the few relevant keys is finite; at some point, the signal that ought to be on the answer span is competing with a very long tail of irrelevant-but-not-zero attention weights. That's the clearest first-principles reason to expect degradation with length, and one reason I keep saying that "more tokens in the prompt" is not a free lunch even before you get to any specific benchmark.

That constraint is at the root of every long-context failure mode in this post.

Lost in the middle

The first serious measurement of the gap came from Liu et al. in 2023. They took a question-answering task, made the answer dependent on one specific passage, and varied where in the context that passage sat. Everything else held constant. The model should not care where the answer lives; attention is position-free at the computational level. Turns out it cared a lot, and the recall-vs-position curve had a clean U-shape. Passages at the very start or very end of the context got recalled near-perfectly. Passages in the middle got recalled dramatically worse. The dip widened as the total context got longer. At 8k it was a gentle sag. At 128k it became a canyon.

Why the U? A few things pile up. First, the relative positional signal is weakest for positions far from both ends; most RoPE-style encodings discriminate less well in the interior than at the edges, especially after context extension. Second, long-context training data, to the extent it exists, tends to put the important stuff near the boundaries (intros, conclusions, first-mention-then-elaborate), so the model learned to check there. Third, the attention distribution is bounded in total mass, and a model that hasn't seen dense coverage of the middle in training will spread its attention more thinly there. None of those are individually catastrophic. Stacked, they produce the U.

There's also a softer fourth factor I think is underrated. When you're training on mixed-length sequences, the loss signal for long-context reasoning is diluted by the enormous number of short-context examples where the same "attend to nearby context" shortcut works. The model learns short-range tricks because they're cheap, and it only reaches for long-range tricks when the short ones fail. The middle of a long document is exactly the regime where short-range tricks quietly succeed without quite being right.

The named handle I use for this: the ends are loud, the middle is quiet. If you're reading a paper that claims a long-context result, the first thing to ask is where the answer lived in the prompt. The second is whether the evaluation sampled depths uniformly or concentrated near the boundaries. It's the latter a lot of the time.

Needle-in-a-haystack: the standard stress test

Greg Kamradt turned the lost-in-the-middle pattern into a canonical benchmark, now just called "needle-in-a-haystack" (NIAH). The setup is simple enough to explain in one sentence: hide a single fact (the needle) somewhere in a long document (the haystack), ask the model to retrieve it, repeat at many depths and many document lengths, plot a heatmap.

NIAH is genuinely useful because it produces a picture. You can look at a 10×10 grid and tell immediately whether a model's long context is smoothly working, partially failing, or catastrophically broken. Vendors started publishing NIAH charts, and for a while green-heatmap screenshots were the marketing proof that a long window worked.

But NIAH is also limited. Finding one fact, verbatim, is the easiest possible long-context task. A model can score 100% on needle retrieval and still fall apart on any task that requires tracking multiple entities, cross-referencing, or aggregation across the document. Passing NIAH is necessary but nowhere near sufficient. When I see a vendor publishing only NIAH numbers, I assume the harder benchmarks didn't look as good.

There's another subtle issue with NIAH as commonly reported. The "needle" is usually a lexically distinctive sentence planted in a sea of unrelated text. That's almost an adversarial test for the model's attention, because the needle is trivially easy to locate by surface features. Real long-context tasks look nothing like this. The relevant spans are lexically similar to the surrounding material, ambiguous, and sometimes spread across multiple locations. A model that aces NIAH because its key-query match is strong on outlier tokens might do poorly the moment the needle is phrased like everything else in the haystack.

RULER: a real long-context benchmark

The corrective came in 2024 from Hsieh et al., with a benchmark called RULER. The key idea: NIAH is too easy, so build a family of related tasks at calibrated difficulty levels, and measure each separately.

RULER's tasks include multi-needle retrieval (several facts, several questions), variable tracking (follow an identifier through many rebindings), multi-hop reasoning, and aggregation over the document. Each task has a clean definition and scales with context length in a controlled way, so you can plot recall vs length and compare across models fairly.

When you plot recall against context length, task-type-by-task-type, the lines fan out. Single-needle retrieval stays high for longer than anything else. Multi-needle degrades faster. Variable tracking falls off earlier still. Aggregation collapses first, often well below the advertised window.

The takeaway that stuck with me: effective context length is task-dependent. A model can be "good at 128k" for retrieval and "good at 32k" for aggregation, simultaneously. The single headline number hides that.

This has a named analogue in benchmarking more broadly. A single metric for a multi-task capability is always an average over tasks, and averages hide shape. For short-context benchmarks we learned to report per-category breakdowns (reasoning vs knowledge vs code). For long-context benchmarks the equivalent move is per-task-type degradation curves, and RULER is the first benchmark I know of that normalised this reporting practice. RULER also introduced a rough heuristic of the effective length, the context length at which performance drops below some threshold (say 85% of short-context performance). Those numbers are typically 2-4× smaller than the advertised window, sometimes 10× smaller.

So when someone asks "does this model really have a 1M context?", the correct answer has the form "for what task, and with what degradation tolerance?" Any shorter answer is skipping past the part that matters.

The RULER framing also makes sense of a pattern that shows up across published results. Models that look similarly capable on short inputs diverge sharply as context grows, and the divergence isn't uniform across tasks. One model might hold up beautifully on retrieval out to 200k but crumble on variable tracking at 32k. Another might be the reverse. The intuitive shorthand "some models are better at long context than others" stops being useful here. The question becomes "better at long context for which task family?", and the answers start to line up with what the provider actually trained on.

A hand-worked example: the per-position curve

Let's make this concrete with a small worked example. Imagine one realistic document, say a 32k-token codebase plus documentation. We hide a specific fact (call it the "API rate limit is 60 requests per minute") at 20 different depths, one evaluation per depth, and score whether the model returns it.

For illustration, here's a plausible per-position recall curve for a frontier model that advertises 1M tokens:

where $p \in [0, 1]$ is the fractional depth.

That's a U-shape centered at $p = 0.5$, with edge recall around 0.85 and mid-depth recall around 0.55. The mean across all 20 positions is about 0.65. None of these numbers are calibrated to a specific model; the shape is the point.

Now here's the trap. If you only tested at $p = 0.02$ and $p = 0.98$ (which a lot of NIAH plots effectively do, because they over-sample the boundaries), you'd report recall around 0.82 and call the model "good at 32k." If you tested at $p = 0.5$, you'd report 0.55 and call the same model "bad at 32k." Both are true statements about the same model. The difference is which portion of the U-curve you drew your samples from.

This is why the RULER paper insists on uniform depth sampling across its task battery, and why the Liu paper plots the full curve rather than a summary. If you're doing your own long-context evaluation, it's worth the extra wiring to sample depths uniformly and report the whole curve back, not just the mean. The shape of the curve tells you more about how the model will behave on your real inputs than any single number will.

import numpy as np
 
# Per-position recall on a 32k doc, averaged across 20 depths.
# Shape: edge high (~0.85), middle low (~0.55).
def recall_at(p, edge=0.85, depth_drop=0.30):
    u = 4 * (p - 0.5) ** 2           # 0 at middle, 1 at edges
    return edge - depth_drop * (1 - u)
 
depths = np.linspace(0.025, 0.975, 20)
r = np.array([recall_at(p) for p in depths])
 
# What you'd report if you only tested at the boundaries:
edge_mean = r[[0, -1]].mean()
# What you'd report if you only tested in the middle:
mid_mean  = r[8:12].mean()
# What the true average over uniform depths is:
full_mean = r.mean()
 
print(f"boundary-only:   {edge_mean:.2f}")
print(f"middle-only:     {mid_mean:.2f}")
print(f"uniform average: {full_mean:.2f}")
# boundary-only:   0.82
# middle-only:     0.55
# uniform average: 0.65

The sampling strategy is doing most of the work of producing the headline number. A well-designed benchmark picks depths uniformly and reports the whole curve, not just the mean. A badly-designed one picks two positions and reports a percentage.

A practical test I run on any long-context claim I can't immediately trust: ask whether the number is a boundary-only, middle-only, or uniform average. If the paper or datasheet doesn't say, assume boundary-only, because that's the version that makes the model look best. A lot of headline numbers turn out to be produced by benchmarks that under-sample the middle. The U-curve hides inside an averaged score the way the mean hides a skewed distribution.

Why RAG with 4k beats stuffing 128k

Given all that, here's the practical claim: for most factual question-answering workloads, sending the model a small retrieved context (say 4k tokens of the most relevant passages) beats sending it a large stuffed context (say 128k tokens of the whole document set).

The reasoning is not mysterious.

A good retriever is already doing the hard work of picking relevant passages. The model's attention, given 4k tokens of mostly-relevant material, behaves the way we want attention to behave: it can read everything carefully, it can cross-reference, it can pay attention to the middle because the middle is also relevant.

Give the same model 128k tokens where only 2-3k are relevant, and attention has to do the retrieval work itself, across 30× more material, with the same fixed total attention mass. The relevant passages are now competing with 125k tokens of noise.

The lost-in-the-middle effect kicks in, the task-dependent effective-length ceiling kicks in, and you end up paying more in latency, cost, and cache footprint for worse answers. Every part of that sentence is doing work.

The one case where stuffing tends to match or beat RAG is when the task genuinely needs broad coverage: document summarisation, "what are the themes in this book," "rewrite in a different tone across all chapters." For those, a retriever that returns 4k tokens is structurally wrong; the 4k slice loses information the task needs.

That exception is the clue to what RAG is actually doing. RAG is good when the answer lives in a small, findable fraction of the document. It's bad when the answer depends on the whole document. The rest of the long-context vs RAG debate is just this distinction restated in different vocabulary.

So the practical question I ask is: does my task's answer live in 4k-ish findable tokens, or does it need the whole thing? If the former, RAG wins on accuracy, cost, and latency. If the latter, long-context is structurally required and you're paying the context-rot tax whether you like it or not.

One more wrinkle: the RAG-wins result is strongest when the retriever is good. If your retriever returns 4k tokens that don't include the answer, stuffing 128k will win by default because the stuffed version at least contains the right passage somewhere. The comparison I care about isn't "RAG with a broken retriever vs stuffing"; it's "RAG with a well-tuned retriever vs stuffing." That's why the previous post spent so much time on retrieval quality. The whole case for a short context rests on the retriever finding the right passage. When it doesn't, the argument flips. This is also why the right lever for a failing RAG system is almost always "improve retrieval," not "send more tokens to the model."

Context rot: the ceiling keeps moving

There's a related phenomenon, less precisely characterised than lost-in-the-middle but robustly observed: as context grows, the model's per-token reasoning quality degrades even for tokens near the end. Anthropic described a version of this in their long-context posts; the academic literature sometimes calls it "context rot." The intuition is that the model's attention has a finite budget in some soft sense; stretching that budget over more tokens means each token gets thinner slices of it.

You can see this most cleanly in generation tasks. Ask a model a precise question with a short context: you get a clean answer. Prepend 100k tokens of irrelevant material and ask the same precise question: the answer gets vaguer, hedges more, occasionally invents. The same weights are doing the inference. The only thing that changed is how much context they were asked to integrate. This is why "just stuff more into the prompt" isn't free even when the relevant material is at the end where lost-in-the-middle is kindest.

I don't want to overclaim the mechanism, because the field is still sorting out how much of this is attention-mass saturation, how much is softmax tail-bleeding, how much is positional-encoding drift, and how much is training-distribution mismatch (most training sequences are much shorter than 100k, so the model is rarely actually doing what you're asking it to do). Whatever the mix, the empirical fact is stable: the longer the prompt, the less sharp the reasoning, even when the advertised window isn't close to being full.

There's a tempting analogy to working memory in cognitive science, and I use it informally, though I don't want to lean on it too hard. A person reading a 50-page document can answer simple questions about it, but as the questions start requiring cross-references and aggregation, performance degrades even for a motivated reader with unlimited time. The bottleneck isn't reading speed; it's the capacity to hold and manipulate a structured representation of what's been read.

Long-context models feel like they're hitting a similar ceiling, and "working memory" is at least a workable placeholder name for it while the mechanism gets pinned down. It's the right shape of explanation, even if the details will look different once someone writes the definitive interpretability paper.

The practical consequence of context rot, whatever its ultimate mechanism, is that you can't treat the spec window as a budget to fill. Filling it makes the model worse at the task even when the new tokens are technically relevant. The right design stance is to put just enough context in to do the job. That's a disposition this series will keep coming back to.

Misconceptions

"A 1M context model has 1M tokens of memory." The advertised window is what the system accepts as input, full stop. It's not a guarantee about recall, reasoning, or integration across those tokens. For most tasks, the effective context (the length at which the model is still near-peak on the task you care about) is 2-10× shorter than the advertised one. Treat the big number as the admission criterion, not the performance criterion.

"Passing needle-in-a-haystack means the long context works." NIAH is a floor, not a ceiling. It measures one-shot verbatim retrieval, which is the easiest long-context task. A model can pass NIAH at 1M and collapse on multi-hop reasoning at 32k. When someone shows you a green heatmap, ask what the RULER-style multi-task numbers are. If they aren't published, assume they're worse.

"If RAG is good at 4k, it'll be even better at 128k." Tempting, and wrong in the direction people don't expect. Adding more retrieved passages past a point typically hurts, not helps. The relevant passages stop dominating attention. Lost-in-the-middle and context-rot kick in. Most retrievers and rerankers are tuned for top-$k$ where $k$ is small for a reason. The right dial when RAG underperforms is usually retrieval quality, not context length.

"Prompt caching makes context length free." Caching makes re-sending a prefix cheap. It doesn't make the model's reasoning over that prefix any better. You still pay the effective-context tax on accuracy, and you still pay generation latency at the tail. Caching moves the cost curve; it doesn't move the recall curve. A system built around "cache the whole corpus into every request" will hit the same lost-in-the-middle and RULER-degradation walls as the uncached version, just more cheaply.

"Adding context never hurts." It feels like it should be true: more information, better answer. It isn't, when the extra information is mostly irrelevant. Adding irrelevant context dilutes attention, pushes lost-in-the-middle into the previously-safe zone, and often measurably drops task accuracy. In the RAG vs stuffing numbers above, the stuff-1M configuration doesn't just fail to beat stuff-128k; on most tasks it does worse. Adding tokens past the retrieval-quality sweet spot is a choice that has a cost, not a conservative default.

What's next

This post was about measurement and limits, what a long context gives you and how to tell. We looked at how the advertised window diverges from the effective one, how lost-in-the-middle and RULER-style task degradation show up in real evaluations, and why a 4k-token RAG setup typically beats stuffing 128k or 1M tokens for the kinds of factual Q&A that most production apps are doing.

The named handles to take away: the ends are loud, the middle is quiet; the spec clock is fast, the reasoning clock is slow; effective length is task-dependent; adding irrelevant context usually hurts. If those stick, the post did its job.

The next post, Context Engineering: Compaction, Clearing, and Memory, is about what to do inside a context window that you now know is leakier than the datasheet suggests: compaction, structured prompt layouts, tool-result clearing, and the emerging craft of packing a bounded attention budget well.

Additional reading (and watching)

• Gemini Team, Google. (2024). Gemini 1.5: Unlocking Multimodal Understanding Across Millions of Tokens of Context. arXiv:2403.05530. The first serious 1M-token technical report, including NIAH and long-context reasoning results.

• Peng, B., Quesnelle, J., Fan, H., Shippole, E. (2023). YaRN: Efficient Context Window Extension of Large Language Models. arXiv:2309.00071. The most widely-adopted recipe for extending a pretrained model's context length with minimal fine-tuning.

• Liu, N. F., et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. TACL 2024. The canonical paper documenting the U-shaped recall-vs-position curve.

• Kamradt, G. (2023). Needle In A Haystack — Pressure Testing LLMs. The repository that turned NIAH into a standard stress test; every major lab now ships a version of this plot.

• Hsieh, C.-P., et al. (2024). RULER: What's the Real Context Size of Your Long-Context Language Models?. COLM 2024. Introduces the multi-task long-context benchmark and the "effective length" metric used throughout this post.

• Munkhdalai, T., et al. (2024). Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention. arXiv:2404.07143. Discusses the working-memory view of attention and motivates compressive approaches; a useful reference for the context-rot intuition.

• OpenAI. (2024). Prompt caching in the API. Documentation for the caching API that changed the economics of long stable prefixes.