Rerankers for RAG: open models you can run yourself

A reranker for RAG is a cross encoder model that reads the question and each retrieved passage together, scores how relevant the passage is, and reorders the shortlist so the best few reach the language model. It sits between retrieval (vector, keyword or hybrid) and the prompt. It helps most when the right passage is retrieved but ranked too low, and it costs one extra model pass over the shortlist for every question.

This is a Labs note built from model cards and library docs. We quote no benchmark of our own.

What does a cross encoder reranker do?

Vector search uses a bi-encoder: each passage becomes one embedding at indexing time, the question becomes another, and the search compares the two. That is fast because the passages were encoded in advance, but the model never sees the question and the passage side by side.

A cross encoder takes the question and the passage as one input and outputs a single relevance score. The Sentence Transformers docs explain why that cannot replace search: a cross encoder produces no embedding, so nothing can be indexed, and every pair must be scored at question time. Their example: clustering 10,000 sentences needs about 50 million pair scores, about 65 hours, against about 5 seconds for a bi-encoder. Their advice is to retrieve around 100 candidates with the fast model and rerank only those.

Where does it sit in the pipeline?

1. Filter by the user's permissions, so restricted text never enters the shortlist.
2. Retrieve a wide shortlist with vector search, keyword search, or both merged, as in our post on hybrid search with BM25.
3. Rerank the shortlist with the cross encoder.
4. Keep the top few and put them in the prompt.

Anthropic's contextual retrieval tests fetched the top 150 chunks, reranked them, and passed the top 20 to the model. Adding the reranker took their top 20 retrieval failure rate from 2.9% to 1.9%, on their own datasets.

When does a reranker help, and when not?

It helps when the right passage is somewhere in the top 50 but not in the top 5. That happens with large collections, near duplicate documents such as several versions of a policy, and questions phrased very differently from the source text.

It cannot fix a passage that was never retrieved. If the answer is missing from the shortlist, look at chunking, metadata and hybrid search first. To tell the two problems apart, measure recall at 50 and hit rate at 5 on a test set of real questions: a large gap between them is what a reranker closes.

Open reranker models you can self host

Licences, sizes and limits below are from each model card, checked on 10 November 2026.

Read the licence before you build. Jina's multilingual reranker v2, for example, is published under CC BY-NC 4.0 for research and evaluation, and its card points commercial users to Jina's API or cloud marketplaces.

To serve one, Hugging Face's Text Embeddings Inference lists bge-reranker-base, bge-reranker-large and gte-multilingual-reranker-base among supported rerankers, on CPU or NVIDIA GPUs. The bge-reranker-v2-m3 and Qwen3-Reranker cards also show how to load them with the CrossEncoder class in Sentence Transformers.

How much latency does reranking add?

Anthropic's write-up puts it plainly: reranking adds an extra step at run time, so it adds latency, and reranking more chunks trades speed and cost for quality. Three things decide how much:

• Number of passages. Each one is a separate pair to score, so 150 candidates cost about three times as much as 50.
• Passage length. Long chunks take longer to read; trimming to the model's limit also caps the cost.
• Model size. On the MiniLM card's V100 test, the 6 layer model scored 1,800 passages a second and the 12 layer one 960, with nearly the same ranking quality. Billion parameter rerankers such as the 4B and 8B Qwen3 models need more memory and more time per pair, so plan on a GPU.

A practical start is 30 to 50 candidates, batched in one call, on the same GPU as your embedding model.

How to choose one

1. English only documents: start with a small English model such as MiniLM or the Granite reranker.
2. Hindi, Tamil or mixed language documents: try gte-multilingual-reranker-base, bge-reranker-v2-m3 and Qwen3-Reranker-0.6B on your own questions.
3. Run the test set without a reranker, then with each candidate, and record the top 5 hit rate and the slowest response times.
4. Keep the smallest model whose quality you cannot tell apart from the best.

Our RAG chatbot builds combine vector and keyword search and rerank the results; where documents must stay on your premises, the whole stack runs on your own servers through private AI deployment.

If your assistant finds the right document but quotes the wrong passage, send us a few failing questions. Book a consultation.