# Reranking

Re-scoring an initial candidate set with a more expensive, more accurate model before passing results to the generator.

**Rests on:** Test-Time Compute

**Leads to:** Agentic Retrieval

*LLMs and Generative AI · Retrieval-Augmented Generation*


## Why learn this

Reranking is the single highest-return addition to a basic RAG pipeline. It turns a cheap, slightly sloppy first search into a precise final list — the difference between the generator pulling in irrelevant chunks and getting exactly the evidence it needs. Master this and you unlock: Context Precision and Recall (the metrics that prove your retrieval is good), then Multi-Document Retrieval (weighing several sources), then Admissible Evidence Spans (the regulated-world requirement that answers trace to specific verified passages). If you want to build a production-grade RAG system, reranking is the step that separates a demo from something you'd ship.


## The intuition

When you search a large collection of documents for a query, the first pass is deliberately fast but rough. It might bring back a hundred candidate chunks — many are on-topic, some are close but not quite right, others are clear misses. Reranking is the second, slower pass: it takes that pile of candidates, re-reads each one against your exact query, and produces a refined, ordered list — maybe just the top five — that you actually hand to the language model. The key idea is that you don't ask the cheap first pass to be perfect; you ask it to be fast and recall as many plausible candidates as it can, then you let a more careful, more expensive model decide which of those candidates are truly most relevant. This is the 'cheap wide retrieval, expensive narrow rerank' pattern. Think of it as a two-stage filter: the first stage is a rough sieve that catches everything that might be relevant, and the second stage is a fine mesh that keeps only the best.


## An analogy

Imagine you're a hiring manager at a large company with thousands of résumés for a single software developer position. Your first pass is a quick keyword scan: does the résumé mention 'Python' or 'API design'? That scan is fast and covers every single résumé, but it's inaccurate — it might flag a résumé because the word 'Python' is mentioned in a hobby, or miss a brilliant candidate who didn't use that exact keyword. You now have a shortlist of fifty résumés. The second pass is a manual, careful read of each one, comparing it directly against the job description. You check not just 'mentions Python' but 'has built a Python web service in production' and 'has designed REST APIs'. This careful read is slow — you can't do it for thousands — but it's far more accurate. After this, you have your final five candidates to interview. Reranking works exactly this way. The first retrieval is the keyword scan: fast, covers the whole corpus, but rough. The reranker is the careful human reader: it looks at each chunk *together with* the query and decides, with high precision, which ones are truly most relevant. Where the analogy breaks down: the reranker isn't human, it's another machine-learning model, so it doesn't have human judgment — it has been trained on millions of query-document pairs to *predict* what a human would find relevant. And 'interviewing' the final chunks is the generator's job, not the reranker's.


## Definition

Reranking is the process of taking an initial set of candidate documents (or chunks) retrieved by a fast but rough method, and re-scoring them with a more expensive, more accurate model that considers the query and document together, in order to select and order the most relevant subset before passing it to the generator.


## How it connects

You don't know embeddings or vector search yet, so think of the first retrieval stage as a black box that returns a list of candidates. Your library notes say that reranking is a key part of the RAG pipeline: ingest, chunk, embed, retrieve, rerank, generate. This concept pairs naturally with Context Precision and Recall — reranking improves precision by removing irrelevant chunks that would otherwise eat the model's context budget and distract it. It also supports Admissible Evidence Spans: when answers must cite specific verified passages, the reranker ensures the generator only sees passages that are truly on-topic, making citation and compliance much easier to enforce. It is distinct from Agentic Retrieval, where the model decides to search again in a loop — that is a different layer. Reranking is a one-shot refinement of the initial set.


## The mathematics

Reranking does not introduce new mathematics; it reuses the concept of scoring. To score something means to give it a number that reflects how good it is. In reranking, each candidate chunk gets a score reflecting how relevant it is to the query.

The crucial difference between the first retrieval and the reranker is that the first retrieval often scores the chunk *on its own*, using an embedding — a fixed list of numbers that represents the chunk's meaning. The reranker, however, scores the query and the chunk *together*, as a pair. It looks at the actual words of both and asks: 'Does this chunk directly answer this query?'

The mathematics behind this is simple arithmetic: the model computes a number — the score — for each pair, and we sort the chunks by that number, highest first. The model itself has been trained to produce high scores for genuinely relevant pairs and low scores for irrelevant ones, but you don't need to know its internals — just that it outputs one number per pair. Then you select the top N chunks by that score. The only operation you need is sorting a list of numbers.

Suppose you have three chunks, scores 0.8, 0.3, and 0.95. Sort them descending: 0.95, 0.8, 0.3 — you know which is best.

For each candidate document d in the initial set D, the reranker computes a score s = f(q, d), where f is the reranking model that takes the query q and document d together. The reranked list is the documents sorted by s descending, and you select the top k: R = top_k( sort_by_score(D, f) ). The symbol f is just the reranker model; you don't need to write its internal formula.


## A worked example

``` You have four candidate chunks from a first-pass search for the query 'What is the capital of France?'. The first-pass retrieval returned those chunks, but their order might be based on a similarity score that isn't perfect. Now you pass each chunk together with the query to a reranker that outputs a relevance score between 0 and 1 (1 = perfect match).

Step 1: Query + Chunk 1 ( "Paris, the capital, has the Eiffel Tower" ) -> score: 0.92   # The reranker gives a high score because the chunk directly mentions Paris and the capital.
Step 2: Query + Chunk 2 ( "France is a country in Europe" ) -> score: 0.45   # Related but not specific to the capital.
Step 3: Query + Chunk 3 ( "The Louvre is a museum in Paris" ) -> score: 0.88   # High score because it mentions Paris, though not explicitly the capital.
Step 4: Query + Chunk 4 ( "The weather in London is rainy" ) -> score: 0.05   # Clearly irrelevant.
Step 5: Sort by score descending: 0.92 (Chunk 1), 0.88 (Chunk 3), 0.45 (Chunk 2), 0.05 (Chunk 4)   # We reorder the list from most to least relevant.
Step 6: If you want top 2, select Chunk 1 and Chunk 3.   # The final set passed to the generator.

Result: The reranked, final list is [Chunk 1, Chunk 3, Chunk 2, Chunk 4]. With a top-2 cutoff, the generator receives only Chunk 1 and Chunk 3. ```


## From scratch, in Python

```python
import math

# Some candidates from a first-pass retrieval (just strings)
candidates = [
    "Paris, the capital, has the Eiffel Tower",
    "France is a country in Europe",
    "The Louvre is a museum in Paris",
    "The weather in London is rainy"
]

query = "What is the capital of France?"

# A toy reranker: counts how many words from the query appear in the chunk
# (In real life, a machine-learning model does this, but this shows the idea)
def toy_rerank(query, candidate):
    query_words = set(query.lower().split())
    candidate_words = set(candidate.lower().split())
    overlap = query_words.intersection(candidate_words)
    return len(overlap)

# Score each candidate
scored = []  # a list of (score, candidate) pairs
for c in candidates:
    s = toy_rerank(query, c)
    scored.append((s, c))  # a tuple (score, candidate)

# Sort descending by score (the first element of the tuple)
scored.sort(reverse=True)

# Take the top 2
k = 2
top_k = scored[:k]

print("Reranked list:")
for s, c in top_k:
    print(f"{s:.2f} - {c}")

```


## The same thing, with a framework

```python
In a real RAG system, you wouldn't write the scoring logic yourself — you'd use a library like `sentence-transformers` or `ragas`. Here's how you'd use a cross-encoder, which is a pretrained model that scores query-document pairs:

from sentence_transformers import CrossEncoder

# Load a pretrained reranking model
model = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

# The candidates and query
query = "What is the capital of France?"
candidates = [
    "Paris, the capital, has the Eiffel Tower",
    "France is a country in Europe",
    "The Louvre is a museum in Paris",
    "The weather in London is rainy"
]

# Create pairs (query, candidate) and score them all at once
pairs = [(query, candidate) for candidate in candidates]
scores = model.predict(pairs)

# Sort by scores and take top 2
scored = sorted(zip(scores, candidates), reverse=True)
top_k = scored[:2]

for score, candidate in top_k:
    print(f"{score:.3f} - {candidate}")

Underneath, this library does exactly what your `toy_rerank` did — it takes each query-candidate pair, runs them through a neural network, and outputs a single number. The only difference is that the network has been trained on millions of examples to be much better at judging relevance.
```


## What the code does

The first version, `from_scratch`, is deliberately simple. It breaks down the idea of scoring into three steps. First, we build a list (`candidates`) of strings we got from a first-pass retrieval. A list is a Python container that holds items in order — here, four short texts. We create an empty list called `scored` that will hold pairs of numbers and texts. A tuple `(s, c)` is like a list but can't be changed after creation; we use it to keep each score tied to its candidate. The `for` loop goes through each candidate one by one. For each, we call the `toy_rerank` function. A function is a reusable block of code that takes inputs (here, query and candidate) and returns a value (the score). Inside `toy_rerank`, we use `set()` and `split()`: `split()` divides a string into words, and `set()` makes a collection where each word appears only once — order doesn't matter. We then count how many words are in both sets using `intersection`. This gives a crude score: more shared words = more relevant. Then we add the `(score, candidate)` tuple to the `scored` list. After the loop, we sort the list with `scored.sort(reverse=True)` — sorting by the first element (the score) and reversing to get highest first. Finally, we take the first `k` items using `scored[:k]`, which is a slice that takes items from the start up to index k. The output prints the top two.


## Common misconceptions

**Reranking is the same as the first retrieval.** — The first search already returns the best chunks, so reranking is just a waste of time.

The first retrieval (e.g., vector search) scores each document *on its own* using an embedding — a fixed numerical summary. It cannot see the query's specific wording, so it can rank a chunk that is semantically similar but not directly answering the query high, and a perfect match lower. In fact: Reranking is a second, different stage: it scores the query and each candidate *together*, directly comparing their words. This joint scoring is far more accurate, and it's why reranking usually gives a big quality boost.

**Reranking is too expensive to be worth it.** — A reranker model is slow and costly, so it's better to skip it and just use the first retrieval results.

The reranker is expensive only because it runs on every candidate, but that's precisely why you first retrieve a small candidate set (like 20–50 chunks) with a cheap method. The reranker only processes those few, not the whole corpus, so the extra cost is tiny and the quality gain is large. In fact: The two-stage pattern exists to make this trade-off work: cheap broad retrieval followed by expensive narrow reranking on a small set. The total latency and cost remain low.

**The reranker's score is the final answer's confidence.** — If the reranker gives a high score, the generation will be correct.

The reranker only judges relevance of the chunk to the query — it does not verify factual accuracy or whether the chunk actually answers the question. It is possible for a chunk to be on-topic but contain outdated or wrong information. In fact: The reranker's score is a relevance signal, not a truth signal. The generator still needs to produce the answer, and citation or verification steps may be needed to ensure correctness.

**A reranker is a single monolithic model that works for everything.** — One reranker model is good enough for all types of queries and documents.

Rerankers are trained on specific data. A cross-encoder trained on general web text may perform poorly on specialised legal or medical documents. Domain-specific rerankers often work better. In fact: Choose a reranker that matches your domain, or fine-tune one on your data. It's a common optimisation to evaluate several models on your specific corpus.

**First retrieval must be perfect for reranking to work.** — If my first search misses the relevant document, reranking can't help.

This is true about recall, not precision. Reranking cannot recover documents that were never retrieved. That's why you usually retrieve a generous candidate set (e.g., 20-50) to ensure high recall, and then rerank for precision. In fact: The two stages have complementary roles: retrieval maximises recall (getting possible candidates), reranking maximises precision (selecting the best). To fix low recall, improve retrieval, not the reranker.


## In short

Reranking is the second, more careful pass over a set of candidates retrieved by a fast first stage. It scores each query-document pair together, sorts by that score, and keeps only the top-k most relevant. This pattern — cheap wide retrieval, expensive narrow rerank — is the standard in RAG and also in recommendation and search systems. It dramatically improves the quality of what the generator (or downstream consumer) sees, and it is often the single highest-return addition you can make to a basic RAG pipeline.


## Where this is used

Reranking is not a niche trick — it is a standard production pattern across almost every RAG system. Here's what actually happens in each context.

**In large language models (LLMs) and RAG pipelines:** When you use a foundation model (like GPT-4 or Claude) as a generator, you typically start with a retrieval step — you search a vector database or a search engine to get candidate passages. If you feed the model the top 20 raw candidates, you're wasting its context window and confusing it with noise. Reranking is the step that narrows that to the top 3–5 passages that are genuinely relevant. Practitioners use libraries like `LlamaIndex` or `LangChain`, which have built-in reranker integrations. For example, in LlamaIndex you'd write:

```python
from llama_index.postprocessor.cohere_rerank import CohereRerank

reranker = CohereRerank(top_k=3)
query_engine = index.as_query_engine(
    similarity_top_k=20,
    node_postprocessors=[reranker]
)
```

This says: retrieve 20 chunks first, then rerank to 3. If you skip reranking, the generator sees too many irrelevant chunks and its answers get diluted or hallucinated. If you do it wrong — say, with a badly tuned top_k — you might keep too few and lose the actual answer. The mechanism: the reranker scores each of the 20 against the query, keeps the 3 highest, and the generator reads only those.

**In small language models running on phones or laptops:** When the model must run on a device with limited memory and compute, you can't afford a huge reranker. The same pattern applies, but you use a smaller cross-encoder (like a distilled version) or even a simple lexical scorer. The key is that the cheap first pass does the heavy lifting of candidate recall, and the expensive narrow pass is only run on a handful of candidates — so even a slow model is feasible because it only runs on, say, 10 chunks, not on the whole corpus. On-device frameworks like ONNX Runtime or TensorFlow Lite can run a compact reranker in milliseconds.

**In retrieval-augmented generation (RAG) for enterprises:** The library notes emphasise that chunking and retrieval quality matter more than model choice. Reranking sits on top of that. In a production RAG system, you often combine hybrid search (keyword + vector) to get the first set, then rerank. The reranker can be a cross-encoder that reads the query and each document together, which is far more accurate than the bi-encoder that first retrieval uses. This is the difference between a demo and a reliable corporate assistant.

**In agentic systems that plan, call tools, and act in loops:** An agent might retrieve multiple batches of documents over a multi-step process. Each time, it can use a reranker to filter the most relevant evidence before deciding what to do next. This prevents the agent from being distracted by a huge context and lets it focus on the key facts. Frameworks like LangGraph let you insert a reranker as a step in the agent's loop.

**In everyday applied ML (recommendation, vision, forecasting):** The two-stage retrieval-rerank pattern is everywhere. In recommendation systems (e.g., Netflix), you first retrieve a thousand candidates with a fast similarity search, then a more expensive model (like a neural network) reranks those to the final 100. In ad ranking, you similarly retrieve then rerank. In search engines, the same idea. The principle is universal: cheap broad recall, then expensive precise ordering.

**In building software:** A developer typically uses a framework like LlamaIndex, LangChain, or Haystack, which provide reranker wrappers. They write a few lines of configuration to plug in a reranker model, and optionally tune the `top_k`. The failure mode is when the developer skips this step entirely and passes the raw first-pass results to the model — results that are often noisy. Another failure mode is using a reranker model that is too large for the latency budget, slowing down the user experience. Proper use: always measure retrieval quality with Context Precision and Recall before and after reranking to prove its value.


---

Read this on the web: https://basavapurushottam.com/learn-ai/reranking
