Rerankers and Cross-Encoders

If you've followed Arc 9 so far, you already know how a dense retriever works. You encode the query into a vector, do an ANN search over pre-encoded document vectors, and return the top-$k$ by cosine similarity. That's the bi-encoder recipe, and it's the reason you can search a million documents in a few milliseconds.

The problem is that the ranking you get isn't very good. It's good enough for recall (the right doc is almost always somewhere in the top-100), but it's pretty rough at the fine-grained ranking inside that top-100. The exact relevant passage is often buried at rank 37 behind four confidently-worded near-misses.

This post is about the move that fixes that. Keep the cheap retriever for recall, then run a slow, careful model over the small candidate set to get the order right. In practice, adding a reranker on top of a decent dense retriever buys you 10-20 points of nDCG almost every time. It's one of the best-value moves in the whole retrieval stack.

What this post assumes

This is Arc 9, post 5. If you've read embeddings and vector search, dense vs sparse hybrid retrieval, and chunking strategies, you already have everything you need here. The core ideas are: a bi-encoder turns a document into a single vector, cosine similarity is a stand-in for relevance, and the first-stage retriever's job is to maximize recall at some moderate $k$ (typically 50–200). If any of that sounds unfamiliar, back up one post. The rest of this post is about what you do after the first stage returns a candidate set.

Why a bi-encoder can't tell you the real order

A bi-encoder encodes the query and the document separately. Each ends up as a single fixed-length vector, and you compare them with a cosine or a dot product. The whole doc, however long and structurally complicated, is being squashed into one vector that has to be useful for every possible future query.

This is a severe bottleneck. The document vector has to compress everything the doc might ever be relevant for. It can't know which part of itself the query is about. There's no opportunity for the query and the document to interact during scoring; they're frozen into vectors first, then compared.

The upside, which is enormous, is that you can pre-encode the entire corpus once and reuse those vectors across every future query. That's what makes sub-millisecond retrieval over a million documents possible.

The downside is what we're about to see: the cheap compressed comparison misses a lot of the fine-grained matching that a human (or a careful model) would catch.

One way to think about the bottleneck: the bi-encoder has to decide "is this doc relevant to this query" using only a single dot product at the very end. All the reasoning about topic overlap, specificity, and negation has to be crammed into the geometry of the vector space. And the vector for a document gets written down before anyone knows what the query is going to be. So the document vector is basically the doc's average self-description, averaged over every query it might ever be asked about. That's a lot to ask of 768 floats.

What a cross-encoder does differently

A cross-encoder feeds the query and the document together into a single encoder, usually as [CLS] query [SEP] doc [SEP], and reads off a single relevance score from the top.

Because the model sees both sides at once, its self-attention layers can do full token-to-token interaction. Every query token can attend to every doc token and vice versa. "paged attention" in the query can light up "KV", "blocks", and "memory" in the doc in a way a single-vector comparison just can't.

Nogueira and Cho were the first to show, back in 2019, that fine-tuning BERT as a cross-encoder on MS MARCO and asking it to produce a single relevance logit per (query, doc) pair blows past every classical IR baseline by a wide margin. Their MonoBERT model became the prototype. Most production rerankers today (Cohere Rerank, Jina Reranker, the various ms-marco-MiniLM models on the Hugging Face hub) are direct descendants of that setup, with better base models and more training data.

The price you pay: a cross-encoder has to re-run the encoder for every (query, doc) pair, because the doc representation now depends on the query. You cannot pre-index anything. A bi-encoder does one query-side forward pass and an ANN lookup. A cross-encoder does one forward pass per candidate.

Here's the napkin math. A modern bi-encoder produces a query vector in a few milliseconds, and the ANN search itself is well under a millisecond per query. A small cross-encoder (MiniLM scale) takes a few milliseconds per (q, d) pair; a base-sized model takes tens of ms. If you want to rerank 100 candidates with a base cross-encoder, that's a few hundred milliseconds of pure GPU time just for the rerank. That's roughly ~100× slower than the first stage.

The two-stage trick exists exactly to make this tradeoff tolerable. The bi-encoder narrows 1,000,000 docs down to 100. The cross-encoder only has to read those 100. So.

One more runtime note that matters in practice. Cross-encoder inference is embarrassingly parallel across candidates, every (q, d) pair is independent, so in practice you batch them. A batch of 100 pairs through a MiniLM cross-encoder on a single A100 is latency-bounded by one forward pass, not 100. This is why the real-world latency is usually much better than "100× a bi-encoder" implies: you're paying for one big forward pass over a long batch, not 100 sequential ones. It still dominates the overall query latency, but it's a single fat step, not a stampede of tiny ones.

The worked example: five candidates, one query

Let's make this concrete. Imagine a query: how does paged attention manage GPU memory?

A dense retriever pulls five candidates. Each gets a cosine score against the query embedding. Then we feed each (query, candidate) pair through a cross-encoder to get a second set of scores. Watch what happens.

The bi-encoder ranks doc D2 at the top because it contains "attention" and looks similar in the embedding space, but D2 is actually from the original transformer paper and has nothing to do with paged attention's memory management. The cross-encoder reads it and gives it a strongly negative logit. Meanwhile, D5 and D1 — the two docs that actually answer the query — float up from the middle to the top.

This is the pattern that shows up again and again across retrieval benchmarks. The bi-encoder's top-1 is frequently wrong in a way that's fixable with a cross-encoder pass. Recall is fine; it's the ordering inside the top-$k$ that needs help.

The cross-encoder output is a raw logit, not a probability, and the magnitude is not directly comparable across different models or even across different queries for the same model. What matters is the ordering of the scores for a single query. The absolute numbers are only useful once you've calibrated them (usually by sigmoid-ing and checking thresholds against a labeled set). If you try to use the raw logit as "how relevant is this doc on a scale of 0 to 10," you'll get confused quickly.

Minimal math: what the scores actually are

A bi-encoder score:

where $f$ is a shared encoder. The query and the doc never see each other inside $f$. They only meet at the dot product.

A cross-encoder score:

where $g$ is a transformer encoder over the concatenated sequence and $w$ is a small output head. There's no separation between query and doc inside $g$. It's one sequence, and the self-attention layers mix information across the [SEP] freely. The whole point is that $g(d)$ doesn't exist on its own, because the doc representation depends on which query it was fed with.

That single difference, separate encoding vs joint encoding, is the architectural pivot of the whole post. It's why a cross-encoder scores better, and it's why a cross-encoder can't be pre-indexed.

I like to keep a compact handle for this in my head: the bi-encoder does early interaction (basically no interaction; a single dot product at the end), the cross-encoder does full interaction (every layer of self-attention crossing freely between query and doc). The ColBERT idea we're about to see is called late interaction because the per-token vectors are computed independently and only meet in one cheap MaxSim step at the end. Once you have that three-way vocabulary (no, full, late) basically every retrieval architecture in the literature slots into one of the three.

ColBERT: can we have both?

An obvious question: can you somehow get the quality of a cross-encoder without paying the joint-encoding cost at query time? ColBERT's answer is: almost.

The trick is called late interaction. ColBERT encodes the query and the document separately, like a bi-encoder. But instead of collapsing each into a single vector, it keeps one vector per token. At query time, you compute the similarity between every query token and every doc token, take the max over doc tokens for each query token, and sum those row-maxes:

Each query token gets to "vote" for its best-matching doc token. Then the votes are summed. No cross-attention, but a much richer signal than a single-vector dot product.

The computational profile is really clean. Doc encoding happens once at index time, just like a bi-encoder. Query encoding happens once per query, also like a bi-encoder. The only query-time cost beyond that is the per-token dot products, which are fast and embarrassingly parallel. The end result is usually within a few nDCG points of a full cross-encoder, at something like 5–10× the cost of a bi-encoder rather than 100×.

The catch, and this was a real limitation of the original paper, is that now you're storing one vector per token in your index instead of one per doc. For long documents that blows up your storage budget fast.

ColBERTv2 fixed this. It quantizes the per-token vectors using residual compression: every token embedding gets clustered to a nearby centroid, and only the tiny residual plus the centroid id are stored. Index size drops by 6–10× with almost no quality loss. This is why you see ColBERT-style architectures in production now, rather than just as academic baselines.

The latency–quality pareto

The range of options in 2026 gives you a proper pareto frontier, not a single answer.

A few things worth saying out loud:

• The gap between bi-encoder and any reranker is big. Even a tiny MiniLM cross-encoder pulls nDCG@10 up meaningfully over a strong bi-encoder baseline on most BEIR datasets.
• The gap between a base-sized cross-encoder and an LLM reranker is often smaller than it feels like it should be. The LLM reranker adds something, especially on hard, reasoning-heavy queries, but the cost delta is huge.
• ColBERTv2 is genuinely in the pareto-interesting region. It's the one option on the chart that doesn't force you to choose between "fast" and "actually good."

In most RAG systems, the reranker is where the best ROI lives. Swapping in a stronger embedding model gives you a couple of points. Switching from no reranker to any reranker at all gives you 10+. If a RAG pipeline doesn't have one yet, that's usually the first thing worth adding.

The cost side of the tradeoff

Pareto plots hide the absolute numbers. Let's look at what this costs you per thousand queries as you vary the top-$N$ you feed into the reranker.

The shape of this curve is why practitioners almost never run a cross-encoder over more than a couple of hundred candidates. Past that, cost blows up and you get diminishing returns anyway: the bi-encoder had already found the relevant docs, you just needed to reorder the top ~50.

A reasonable default I'd reach for in 2026: retrieve top-100 with a strong dense retriever (or a hybrid dense+BM25 fusion), rerank with a managed cross-encoder over those 100, return the top-10. You'll land in the three-cents-per-thousand-queries range and pick up most of the quality available to you.

If you're latency-constrained, the first knob to turn is $N$, not the reranker. Dropping from 100 to 50 roughly halves the rerank cost and usually loses you less than a point of nDCG, because the genuinely relevant docs are almost always in the top-50 of a good first stage anyway. Going the other way (increasing $N$ to chase that last doc) has sharply diminishing returns and is almost never worth it. The exception is when your first stage is weak, and if your first stage is weak, fix it before you touch the reranker.

A tiny implementation

Here's what the full pipeline looks like in code. No frameworks, just sentence-transformers and the same small example from earlier.

from sentence_transformers import SentenceTransformer, CrossEncoder
import torch
 
# --- Stage 1: bi-encoder over a pre-encoded corpus ---
bi = SentenceTransformer("BAAI/bge-base-en-v1.5")
 
docs = [
    "vLLM paged attention fragments KV into blocks so GPU memory is reused.",
    "Attention is softmax(QK^T / sqrt(d_k)) V, from the original paper.",
    "GPU memory allocators use best-fit or buddy systems for VRAM.",
    "A KV cache stores per-layer K and V so decode skips recomputation.",
    "Paged attention removes contiguous KV reservations, cutting waste.",
]
 
# Pre-encoded once, reused across queries.
doc_vecs = bi.encode(docs, normalize_embeddings=True, convert_to_tensor=True)
 
query = "how does paged attention manage GPU memory?"
q_vec = bi.encode(query, normalize_embeddings=True, convert_to_tensor=True)
 
cos = torch.matmul(doc_vecs, q_vec)       # shape (5,)
top_n = torch.topk(cos, k=5).indices.tolist()
print("bi-encoder order:", [docs[i][:40] for i in top_n])
 
# --- Stage 2: cross-encoder rerank over the candidates ---
ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
 
pairs = [(query, docs[i]) for i in top_n]
ce_scores = ce.predict(pairs)             # shape (5,)
 
# Reorder by cross-encoder score, descending.
reranked = sorted(zip(top_n, ce_scores), key=lambda p: -p[1])
print("reranked order:", [docs[i][:40] for i, _ in reranked])

Two models, maybe 25 lines. The shape of every production reranking system is this. Whatever you build on top (Cohere's managed API, a self-hosted ColBERTv2, an LLM judge at the end) is a variation on this two-stage skeleton.

The bi-encoder's doc_vecs gets computed once, up front, and in a real system it lives in a vector database indefinitely. The cross-encoder's pairs get built at query time with the live query spliced into each one, and the forward pass runs over that fresh batch. That asymmetry is the whole architectural story of the post, compressed into five lines of Python.

Misconceptions

"A reranker can rescue a bad retriever." No. A reranker can only reorder the candidates it's given. If the right doc wasn't in the top-100 returned by the first stage, no amount of cross-encoder scoring brings it back. Reranker quality is bounded above by first-stage recall. In practice this means the recall@100 of your retriever matters more than its precision@10, because the reranker handles precision and only recall feeds it. It's easy to spend weeks tuning a reranker when the actual bottleneck is the first stage missing half the relevant docs.

"Cross-encoders are a specific model architecture." They're not; "cross-encoder" describes how you use any sequence-to-scalar model. You can fine-tune BERT, a T5 (monoT5 does exactly this), a Gemma, an LLM, anything that takes a concatenated sequence and produces a relevance score. The architecture is just "encoder that reads (query, doc) jointly." The tricks are all in the training data and the scoring head.

"ColBERT is a reranker." ColBERT can be used as a reranker, but its original pitch was as a first-stage retriever. The late-interaction score is cheap enough at query time that you can run it directly over a large doc pool, assuming you've got the storage for all those token vectors. Most production setups use ColBERTv2 as a retriever instead of a bi-encoder, not as a second stage on top of one.

"More rerankers stacked is better." You can stack a fast reranker after a slow one (e.g. MiniLM over top-200, then a base cross-encoder over top-50, then an LLM judge over top-10). Occasionally this helps. More often it adds latency and surface area for bugs without moving the needle, because the second rerank stage is rescoring an already-good list. Start with one reranker and a sensible $N$.

What's next

This post set up the two-stage retrieval spine: cheap first-stage recall, expensive second-stage precision, and the architectural reason cross-encoders dominate bi-encoders on the ranking side. ColBERT-style late interaction is the interesting middle path when you own your serving stack, and ColBERTv2's residual compression finally made that path practical at index scale.

The next post covers RAG end-to-end, where we wire retrieval, chunking, reranking, and generation together, look at what breaks in production, and trace a query through the whole pipeline.

Additional reading (and watching)

• Thakur, N., et al. (2021). BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS Datasets and Benchmarks 2021. The standard cross-domain retrieval benchmark that shows how much headroom rerankers have on top of dense retrievers.

• Nogueira, R. & Cho, K. (2019). Passage Re-ranking with BERT. arXiv:1901.04085. The first demonstration that a BERT cross-encoder blows past classical IR on MS MARCO. The blueprint for every modern reranker.

• Reimers, N. & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP 2019. Establishes the bi-encoder vs cross-encoder distinction cleanly and motivates the two-stage pipeline.

• Lin, J., et al. (2022). Pretrained Transformers for Text Ranking: BERT and Beyond. Synthesis Lectures. Book-length treatment of ranking models; the cost/quality tradeoffs are analyzed in depth.

• Bajaj, P., et al. (2016). MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. arXiv:1611.09268. The dataset most cross-encoders are fine-tuned on; the baseline for comparing reranker quality.

• Khattab, O. & Zaharia, M. (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. SIGIR 2020. Introduces MaxSim late interaction; the bridge between bi-encoders and cross-encoders.

• Santhanam, K., et al. (2022). ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. NAACL 2022. Residual compression shrinks the ColBERT index by 6–10× with almost no quality loss.

• Nogueira, R., Jiang, Z., & Lin, J. (2020). Document Ranking with a Pretrained Sequence-to-Sequence Model. EMNLP Findings 2020. MonoT5 reframes ranking as generating "true"/"false"; shows the cross-encoder idea isn't architecture-specific.

• Jina AI. (2024). Jina Reranker v2 release notes. Multilingual reranker built for RAG and agentic pipelines.

• UKP Lab. sentence-transformers cross-encoder models. The ms-marco-MiniLM and friends, the go-to self-hostable rerankers.