Rerankers and Cross-Encoders

A dense retriever encodes the query into a vector, searches an ANN index over pre-encoded document vectors, and returns the top-$k$ by cosine similarity. That is the bi-encoder recipe: precompute document representations, then search them for each new query.

A relevant passage may be in the candidate set without appearing near the top. Reranking gives a second model a smaller, more expensive comparison problem. First measure candidate recall: no reranker can score a passage it never receives.

What this post assumes

This builds on embeddings and vector search, dense vs sparse hybrid retrieval, and chunking strategies. The first-stage retriever returns a candidate set; here we score those candidates again with the query available during the comparison.

What a bi-encoder compresses

A bi-encoder encodes the query and the document separately. Each ends up as a single fixed-length vector, and you compare them with a cosine or a dot product. The document vector is computed before the query arrives and reused across queries.

The encoder cannot adapt its document representation to the current query. Relevance must be recoverable from the two independently computed vectors.

Query-time work is encoding one query plus index search, rather than running the encoder jointly over every query–document pair.

What a cross-encoder does differently

A cross-encoder feeds the query and the document together into a single encoder, usually as [CLS] query [SEP] doc [SEP], and reads off a single relevance score from the top.

In an encoder-style cross-encoder, self-attention over the concatenated sequence permits query and document tokens to interact. This is joint self-attention, not a separate encoder–decoder cross-attention block.

Nogueira and Cho demonstrated BERT passage reranking on MS MARCO in 2019. The relevant design is joint query–passage encoding followed by a relevance head. Later rerankers use a range of architectures and training objectives.

The final document representation depends on the query, so it cannot be precomputed once for all queries. Tokenization and some input processing can still be cached. Each candidate requires joint scoring, often in a batch.

With one million indexed passages and a shortlist of 100, the second stage scores 100 query–passage pairs. Sequence lengths, model size, hardware, batching and queueing determine latency.

Pairs are independent within a pointwise reranker and can be batched. Batching improves utilization but does not make 100 pairs cost the same as one. Measure throughput and tail latency under the intended load.

The worked example: five candidates, one query

Consider the query: how does paged attention manage GPU memory?

The following constructed example assigns two sets of scores and graded relevance labels to five candidates. It computes nDCG from those labels; it does not call a model.

In the authored scores, D2 starts first even though it explains the general attention formula rather than memory management. D5 and D1 rise under the second ranking.

A real evaluation asks how often such corrections occur and how often the reranker makes a good ranking worse. Use relevance judgments across many queries, not this one designed example.

Some rerankers output a raw logit; others apply an activation. Read the model and library contract. A sigmoid maps a logit into 0–1 but does not by itself calibrate the result as a probability. Scores are safest as within-query ranking signals unless calibration has been tested.

Minimal math: what the scores actually are

A bi-encoder score:

where $f$ is a shared encoder. The query and the doc never see each other inside $f$. They only meet at the dot product.

A cross-encoder score:

where $g$ is a transformer encoder over the concatenated sequence and $w$ is a small output head. There's no separation between query and doc inside $g$. It's one sequence, and the self-attention layers mix information across the [SEP] freely. The document representation depends on the query, so there is no reusable $g(d)$ to precompute.

Joint encoding lets the model condition its representation on the query at every layer. That adds expressive capacity and query-time work; improved ranking still depends on training and evaluation.

ColBERT: can we have both?

This is late interaction. ColBERT encodes the query and the document separately, like a bi-encoder. But instead of collapsing each into a single vector, it keeps one vector per token. At query time, you compute the similarity between every query token and every doc token, take the max over doc tokens for each query token, and sum those row-maxes:

Each query token gets to "vote" for its best-matching doc token. Then the votes are summed. No cross-attention, but a much richer signal than a single-vector dot product.

Document encoding happens at indexing time, and the query is encoded once. The MaxSim step then compares token vectors. This adds work over one-vector scoring but avoids joint encoding of every pair.

Storing one vector per token instead of one per document increases the index’s storage requirement.

ColBERTv2 uses residual compression to reduce token-vector storage: approximate each vector with a centroid plus a quantized residual. It also changes supervision. Its paper reports results for that combined system; compression does not make the index equivalent in size or cost to one vector per passage.

Compare the work before claiming a frontier

A latency–quality frontier requires a common dataset, candidate set, hardware and serving setup.

Record the candidate recall before reranking, ranking quality after it, and total service latency. This separates an upstream miss from a scoring error.

A small reranker may be sufficient, a larger one may help, or the existing ordering may already be good. Include the no-reranker baseline.

The cost side of the tradeoff

The calculator below assumes a price per scored pair. It makes the candidate-count multiplication explicit; it is not a managed-service price list.

Scoring more candidates gives the reranker more opportunities to recover a good result and also more work. A larger shortlist only helps if it brings useful passages into consideration.

Choose several shortlist depths, evaluate their relevance gains, and time them under realistic load. Keep the final prompt budget separate from the number of candidates scored.

Reducing N reduces pair count exactly for this pointwise setup. Wall-clock latency may change less predictably because batches, padding and queueing also matter.

A tiny implementation

This sentence-transformers example retrieves and reranks the passages shown earlier.

from sentence_transformers import SentenceTransformer, CrossEncoder
import torch
 
# --- Stage 1: bi-encoder over a pre-encoded corpus ---
bi = SentenceTransformer("BAAI/bge-base-en-v1.5")
 
docs = [
    "vLLM paged attention fragments KV into blocks so GPU memory is reused.",
    "Attention is softmax(QK^T / sqrt(d_k)) V, from the original paper.",
    "GPU memory allocators use best-fit or buddy systems for VRAM.",
    "A KV cache stores per-layer K and V so decode skips recomputation.",
    "Paged attention removes contiguous KV reservations, cutting waste.",
]
 
# Pre-encoded once, reused across queries.
doc_vecs = bi.encode(docs, normalize_embeddings=True, convert_to_tensor=True)
 
query = "how does paged attention manage GPU memory?"
# BGE v1.5 recommends this instruction for short-query passage retrieval.
instruction = "Represent this sentence for searching relevant passages: "
q_vec = bi.encode(instruction + query, normalize_embeddings=True, convert_to_tensor=True)
 
cos = torch.matmul(doc_vecs, q_vec)       # shape (5,)
top_n = torch.topk(cos, k=5).indices.tolist()
print("bi-encoder order:", [docs[i][:40] for i in top_n])
 
# --- Stage 2: cross-encoder rerank over the candidates ---
ce = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
 
pairs = [(query, docs[i]) for i in top_n]
ce_scores = ce.predict(pairs)             # shape (5,)
 
# Reorder by cross-encoder score, descending.
reranked = sorted(zip(top_n, ce_scores), key=lambda p: -p[1])
print("reranked order:", [docs[i][:40] for i, _ in reranked])

This is a pointwise two-stage example. It downloads weights and runs real inference when executed; its output need not match the authored scores in the visual.

The bi-encoder computes doc_vecs before the query arrives. Those vectors can be reused until the documents or encoder change. The cross-encoder’s pairs are built and scored at query time.

Misconceptions

"A reranker can rescue a bad retriever." It can improve the ordering of retrieved candidates. It cannot recover a relevant passage absent from that set. Measure candidate recall before attributing a final ranking failure to the reranker; a larger shortlist may recover evidence at additional scoring cost.

"Cross-encoders are a specific model architecture." They're not; "cross-encoder" describes how you use any sequence-to-scalar model. You can fine-tune BERT, a T5 (monoT5 does exactly this), a Gemma, an LLM, anything that takes a concatenated sequence and produces a relevance score. The architecture is just "encoder that reads (query, doc) jointly."

"ColBERT is only a reranker." Its token-level score can support retrieval over a full index or reranking of candidates. Those are different deployment choices.

"More rerankers stacked is better." A cascade can run a fast stage over many candidates and a slower stage over fewer. Each stage can also discard relevant evidence. Compare the whole cascade with a simpler baseline.

What's next

The next post covers RAG end-to-end, where we wire retrieval, chunking, reranking, and generation together, look at what breaks in production, and trace a query through the whole pipeline.

Additional reading (and watching)

• Thakur, N., et al. (2021). BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS Datasets and Benchmarks 2021. The standard cross-domain retrieval benchmark that shows how much headroom rerankers have on top of dense retrievers.

• Nogueira, R. & Cho, K. (2019). Passage Re-ranking with BERT. arXiv:1901.04085. An early BERT cross-encoder reranking result on MS MARCO.

• Reimers, N. & Gurevych, I. (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. EMNLP 2019. Establishes the bi-encoder vs cross-encoder distinction cleanly and motivates the two-stage pipeline.

• Lin, J., et al. (2022). Pretrained Transformers for Text Ranking: BERT and Beyond. Synthesis Lectures. Book-length treatment of ranking models; the cost/quality tradeoffs are analyzed in depth.

• Bajaj, P., et al. (2016). MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. arXiv:1611.09268. A widely used passage-ranking training set and benchmark, including for the cross-encoder in this example.

• Khattab, O. & Zaharia, M. (2020). ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. SIGIR 2020. Introduces MaxSim late interaction; the bridge between bi-encoders and cross-encoders.

• Santhanam, K., et al. (2022). ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. NAACL 2022. Residual compression shrinks the ColBERT index by 6–10× with almost no quality loss.

• Nogueira, R., Jiang, Z., & Lin, J. (2020). Document Ranking with a Pretrained Sequence-to-Sequence Model. EMNLP Findings 2020. MonoT5 reframes ranking as generating "true"/"false"; shows the cross-encoder idea isn't architecture-specific.

• Jina AI. (2024). Jina Reranker v2 release notes. Multilingual reranker built for RAG and agentic pipelines.

• UKP Lab. sentence-transformers cross-encoder models. Pretrained cross-encoders, including the ms-marco-MiniLM family.