# Retrieval Quality

Measuring the retrieval stage independently of generation, so you know which half of a RAG system is failing.

**Leads to:** Context Precision and Recall

**Rests on nothing.** A place to start.

*LLMs and Generative AI · Retrieval-Augmented Generation*


## Why learn this

When a system gives you a wrong answer based on searched information, you need to know whether the search failed or the writing failed. Measuring the quality of the search separately tells you exactly that. If the search missed the right documents, no amount of clever writing can fix it. If the search brought back too much irrelevant text, it confuses the writer. By mastering this, you can diagnose and fix your system systematically, rather than guessing. This skill lets you ensure that when a system cites a source, that source actually supports the answer it just gave.


## The intuition

Think of the system as a two-person team: a researcher who finds documents and a writer who drafts the answer. If the researcher does a poor job, the writer has nothing good to work with. But how do you know whose fault the bad answer is? You measure the researcher's work alone, before the writer touches it. You look at the documents the researcher brought and ask two questions: Did they bring the right ones? And did they miss any essential ones? That gives you two numbers: precision and recall. Precision is 'of everything brought, how much was actually relevant?' Recall is 'of everything relevant, how much was actually brought?' If recall is low, the researcher left key documents in the archive. No writer can compensate for missing evidence. If precision is low, the researcher dumped a pile of irrelevant papers on the desk, distracting the writer and crowding out the useful ones.


## An analogy

Imagine you are searching a library for information to write a report on the history of the Indian space programme. You ask a librarian for help. The librarian returns with a stack of books. Precision asks: of those books, how many are actually about the Indian space programme? If half are about NASA, precision is 50%. Recall asks: of all the books in the library that are about the Indian space programme, how many did the librarian bring? If the library has twenty such books and the librarian brought five, recall is 25%. The analogy holds well in retrieval: you always have a fixed set of documents to search, and you retrieve a subset. The total relevant set is usually unknown, but in experiments it is known, so recall is computable. Where the analogy breaks down: in a real library, the librarian can be asked follow-up questions, but in basic retrieval, the search is a single step. Also, relevance is not always black and white here — some documents are more useful than others.


## Definition

Retrieval quality measures how well a system finds the right documents from a collection, evaluated by checking both the relevance of what was found (precision) and whether anything important was missed (recall).


## How it connects

This concept sits beside retrieval process, which is simply the act of finding documents. It also relates to vector database, which is the specialized storage engine that allows you to find documents based on meaning rather than just matching keywords word-for-word.


## The mathematics

This concept draws on simple arithmetic — counting and division. We are measuring fractions: how many of the retrieved items were relevant, and how many of the relevant items were retrieved. There is no algebra, no calculus. We just count and divide. Let's define two numbers. First, precision at k: imagine we retrieve k items (say, the top 5 from a search). Among those, count how many are relevant (on-topic). Precision is that count divided by k. So if we retrieve 5 items and 3 are relevant, precision is 3/5 = 0.6 or 60%. Second, recall at k: imagine all the relevant documents in the entire corpus (say, there are 10). We retrieve k items (say, 5). Among those 5, count how many are relevant (say, 2). Recall is that count divided by the total number of relevant documents (2/10 = 0.2 or 20%). So recall tells us what fraction of the good stuff we managed to get.

Precision@k = (number of relevant items in top k) / k
Recall@k = (number of relevant items in top k) / (total number of relevant items in corpus)

Where:
- 'k' is the number of retrieved items we are examining
- 'relevant' means on-topic with respect to the query
- 'corpus' is the entire set of documents we search over


## A worked example

```
Imagine a corpus of 100 documents. We know that exactly 10 of them are relevant to our query (we know this because we are testing on a benchmark where the relevant set is pre-labelled). We run a retrieval system and ask it for the top 5 documents (k = 5). The system returns documents A, B, C, D, E. We check each: A is relevant, B is relevant, C is not, D is not, E is not.

Step 1: Relevant in top 5 = 2 (A and B)   # We count how many of the returned documents are relevant.
Step 2: Precision at 5 = 2 / 5 = 0.4   # Precision = relevant retrieved / total retrieved.
Step 3: Recall at 5 = 2 / 10 = 0.2   # Recall = relevant retrieved / total relevant in corpus.

Result: Precision at 5 is 0.4 (40%), meaning 40% of the retrieved documents are useful. Recall at 5 is 0.2 (20%), meaning we only retrieved 20% of all the relevant documents. The retrieval system did not find most of the good material.
```


## From scratch, in Python

```python
# Simple retrieval quality calculation without any libraries

# Let's set up a tiny corpus of documents, each with an ID and some text.
corpus = {
    1: "The Indian space programme began in 1962 with INCOSPAR.",
    2: "NASA's Apollo missions landed humans on the Moon.",
    3: "ISRO launched Chandrayaan-1 in 2008.",
    4: "The Soviet Union launched Sputnik 1 in 1957.",
    5: "Mangalyaan reached Mars orbit in 2014.",
    6: "The European Space Agency studies comets.",
    7: "India's first satellite was Aryabhata, launched in 1975.",
    8: "The Chinese space station Tiangong orbits Earth.",
    9: "ISRO's PSLV has delivered many satellites to orbit.",
    10: "The Hubble Space Telescope sees distant galaxies."
}

# The ground truth: which documents are actually relevant to our query?
# Query: "history of Indian space programme"
relevant_ids = {1, 3, 5, 7, 9}  # five relevant documents

# Pretend our retrieval system returned these top 5 documents (by ID)
retrieved_ids = [1, 2, 3, 4, 5]

# Now compute precision at k (k = 5)
k = 5
relevant_and_retrieved = [doc_id for doc_id in retrieved_ids if doc_id in relevant_ids]
precision_at_k = len(relevant_and_retrieved) / k

# Compute recall at k
total_relevant = len(relevant_ids)
recall_at_k = len(relevant_and_retrieved) / total_relevant

print(f"Precision@{k} = {precision_at_k:.2f}")
print(f"Recall@{k} = {recall_at_k:.2f}")
```


## The same thing, with a framework

In practice, you would use a library like `ranx` or `pytrec_eval` to compute these metrics. For example, with `pytrec_eval`:

```python
import pytrec_eval

# qrels: ground truth relevance, doc_id -> relevance score (1 = relevant)
qrels = {
    "query1": {
        "1": 1, "3": 1, "5": 1, "7": 1, "9": 1
    }
}

# results: retrieved docs with rank score (lower is better rank)
results = {
    "query1": {
        "1": 5.0, "2": 4.0, "3": 3.0, "4": 2.0, "5": 1.0
    }
}

evaluator = pytrec_eval.RelevanceEvaluator(qrels, {"P.5", "recall.5"})
scores = evaluator.evaluate(results)
print(scores)
```

Underneath, `pytrec_eval` does exactly what we did by hand: it counts the relevant ones in the top k and divides by k or by the total relevant. The library just wraps that calculation for many queries at once.


## What the code does

First, we build a tiny corpus as a dictionary. A dictionary in Python looks up a value by a key — here, the document ID maps to its text. We also define a set of relevant IDs. A set holds each item once and forgets the order; we use it to quickly check membership. We then simulate the retrieval output as a list, which keeps items in order. The line `[doc_id for doc_id in retrieved_ids if doc_id in relevant_ids]` is a list comprehension — it creates a new list by going through each retrieved ID and keeping only those that are also in the relevant set. The `len()` function gives the count. We then divide to get precision and recall. Finally, `print` with an f-string formats the numbers. The framework version uses `pytrec_eval`, which is built for standard retrieval evaluation; it takes a dictionary of relevance scores (qrels) and a dictionary of ranked results, and computes the metrics behind the scenes — the same arithmetic as our loop.


## Common misconceptions

**A high recall means the answer will be good** — If recall is high, the system has all the evidence, so the answer must be correct.

Retrieval quality only measures the retrieval stage. Even with perfect recall, the generator can still make a poor answer. Also, high recall without precision means the context is full of noise, which can distract the generator. In fact: Recall tells you whether the evidence was present, but it says nothing about how it is used. It is a necessary condition, not a sufficient one.

**High precision means we can stop working** — If every retrieved document is relevant, the system is perfect.

Precision measures the proportion of retrieved items that are relevant, but it does not tell you whether you missed relevant items. You could retrieve one perfect document and miss nine essential ones — precision would be 100%, but recall would be 10%. In fact: Precision and recall must be looked at together. 100% precision can still mean complete failure if recall is low.

**Retrieval quality is the same as answer quality** — If the answer is wrong, the retrieval must be wrong too.

The generator can make mistakes even with perfect retrieval. Conversely, good answers can occasionally be produced from poor retrieval through luck or memorisation. In fact: The two are separate. Measuring each independently is the only way to know which half to fix.

**Metrics like precision@k are always the right choice** — I should always use precision@k and recall@k; they are the standard.

These metrics assume a binary relevant/not-relevant judgement. In many real-world tasks, relevance is graded (some documents are more useful than others), and different metrics like nDCG handle that. Also, the choice of k matters: k=1 might be good for a factoid question, but k=10 might be needed for a complex research question. In fact: Choose metrics based on the use case. Understand what they measure and what they ignore.


## In short

Retrieval quality measures the retrieval stage alone, using precision@k (of what was retrieved, how much was relevant) and recall@k (of what was relevant, how much was retrieved). It is independent from generation quality. Low recall is unfixable downstream; low precision wastes context. Always measure retrieval before blaming the LLM.


## Where this is used

Retrieval quality is the diagnostic heartbeat of any RAG system. In large language models, when you build a RAG pipeline on top of an LLM like GPT or Llama, the retrieval stage decides what goes into the context window. If recall is low, the LLM cannot answer correctly even if it is capable — the evidence simply is not there. If precision is low, the LLM gets distracted by irrelevant passages and may hallucinate. A developer tunes retrieval quality by adjusting chunk sizes, embedding models, and the number of retrieved chunks, often measuring precision@k and recall@k on a validation set before even looking at generated answers. For small models running on devices, such as a phone or a laptop, the context window is tiny, so precision becomes critical — you cannot afford to waste even one token on noise. Retrieval quality is the gatekeeper. In retrieval-augmented generation specifically, the standard pattern is: retrieve the top k chunks, then optionally rerank them with a cross-encoder. Retrieval quality metrics are applied at two points: after the initial retrieval and after reranking. The reranker often improves precision, but cannot improve recall — if the initial retrieval missed relevant chunks, reranking cannot bring them back. That is why retrieval quality must be measured at the earliest stage. Agentic systems take this further. An agent decides what to retrieve, evaluates whether it has enough evidence, and performs multiple searches in a loop. Retrieval quality is used at each step: the agent checks recall to decide if it needs another search, and checks precision to decide if it should discard noise. This is 'agentic retrieval' — retrieval as a loop, not a single step. In everyday applied machine learning, such as enterprise knowledge assistants, retrieval quality determines whether the assistant can point to admissible evidence. In regulated settings, answers must trace to verified passages, so recall must be high — missing a critical document is a compliance failure. Developers in this space spend most of their effort on the retrieval pipeline because a wrong answer is often a retrieval failure wearing a generation costume. To build a good retrieval system, a developer might use a vector database like Pinecone or Weaviate, write an embedding model call to convert the query into a vector, perform a similarity search, and then evaluate the results with a library like `pytrec_eval`. They would also implement hybrid search — combining keyword (BM25) and semantic (vector) scores — and reranking with a cross-encoder. Each of these steps is tuned by measuring retrieval quality. If the metrics degrade, the developer knows to blame the retriever, not the LLM.


---

Read this on the web: https://basavapurushottam.com/learn-ai/retrieval-quality
