# Multi-Document Retrieval

Assembling evidence from several sources for a single answer, including deduplication and ordering of the retrieved set.

**Rests on:** Chunking

**Leads to:** Agentic Retrieval, Citations and Evidence, Enterprise Knowledge Assistants

*LLMs and Generative AI · Retrieval-Augmented Generation*


## Why learn this

This is the step that decides whether a system that answers from outside sources uses real evidence or just guesses from memory. When you ask a question that touches several documents — a policy that changed between years, a product spec that spans two manuals, a medical guideline that references three studies — the system needs the right pieces, in the right order, without duplicates. Get this wrong and the model either misses the answer entirely or is distracted by repetition and contradiction. Mastering this step lets you build tools that answer from a specific collection of documents rather than from general knowledge. Without it, your tool is just a search engine with a chatbot stuck on top.


## The intuition

When you ask a system a question, you cannot feed it an entire library — the space available for its reasoning (the context window) is like a small desk, not a warehouse. So you must first *retrieve* the few pages that might contain the answer, then hand those to the model. Multi-document retrieval is the step of gathering those pages from several books at once, making sure you do not grab the same paragraph twice, and arranging them in the order the model should read them. The key insight is that the model does not weigh every word equally: it tends to read the beginning and the end of its available space carefully, and skims the middle. So the order you place the retrieved segments (chunks) matters — the most relevant evidence should go where the model is most attentive. Think of it as preparing a briefing folder for a busy decision-maker: you would not hand them fifty pages, nor would you hand them the same page twice, and you would put the most important memo on top, not buried in the middle.


## An analogy

Imagine you are a research assistant for a lawyer preparing a case. The lawyer asks: 'What is the statute of limitations for a breach of contract in our state?' You do not dump the entire legal library on her desk. You go to the shelves, pull down the contract law book, the civil procedure code, and an annotated casebook. You flip through each, find the relevant passages, and photocopy them. But you notice that the contract law book and the casebook both quote the same statute — you do not hand her two copies of the same page, you give her one, and you note that both sources agree. You also notice that one book says 'three years' and an older edition says 'two years' — you flag that conflict on a sticky note rather than silently picking one. Finally, you arrange the pages: the primary statute goes first, then the interpretive commentary, then the case law, because she will read the top pages first and with the most attention. This is exactly what multi-document retrieval does: it gathers from several sources, removes what is repeated, surfaces what contradicts, and orders what remains. Where the analogy breaks down: a human assistant can use judgment about what 'relevant' means and can read the entire book if needed. A retrieval system must decide relevance by numerical similarity, works on fixed chunks, and has a hard limit on how many pages fit on the desk — the context window. And a human assistant would notice a duplicate instantly by reading it; a retrieval system must detect it by checking if two chunks are too similar in meaning, which is a separate step called deduplication.


## Definition

Multi-document retrieval is the part of a system that takes a user's question, searches across a collection of documents, selects the most relevant segments from possibly different sources, removes redundant ones, resolves or surfaces conflicts, and orders the final set so the language model can use it effectively.


## How it connects

This sits inside the parent concept of Retrieval-Augmented Generation — the process is to take in new information (ingest), split it into manageable pieces (chunk), convert text into numbers for comparison (embed), find matches (retrieve), improve the order of those matches (rerank), and generate an answer. Multi-document retrieval handles the 'retrieve' step plus the initial 'rerank' step in miniature. It depends on two things you have notes on: Chunking (how the documents were split into retrievable units) and Embeddings (how each chunk and query is turned into a list of numbers that lets a computer measure similarity). Your notes on Hybrid and Semantic Search feed into how you retrieve in the first place, and Reranking is what happens after you have an initial set. The output of multi-document retrieval becomes the 'evidence' that your note on Citations and Evidence wants to point at, and your note on Retrieval Quality gives you the metrics to check whether this step did its job.


## The mathematics

The mathematics that underlies multi-document retrieval borrows from two branches: *linear algebra* and *probability*. Linear algebra is the branch that deals with lists of numbers and how to combine them; here, every chunk and query is turned into a long list of numbers called a vector, and the similarity between them is measured by an operation called the dot product. Probability comes in when we decide how confident we are that two chunks are duplicates, or when we score and rank multiple candidates. But before any of that, the core operation is the **dot product**, which you have not met yet. Let me teach it from nothing. Take two short lists of numbers, say [1, 2] and [3, 4]. The dot product is found by multiplying the first number in the first list by the first number in the second list, and the second by the second, and then adding those products: 1×3 = 3, 2×4 = 8, 3+8 = 11. The dot product is a single number that tells you how much the two lists 'point in the same direction' — if they are similar, the number is large; if they are opposite, it is negative; if they are unrelated, it is near zero. In retrieval, each chunk is represented by a vector of many numbers (often hundreds), and the query is also a vector. Retrieval is, in essence, computing the dot product between the query vector and every chunk vector, then picking the chunks with the highest scores. Deduplication uses the same idea: two chunks that are near duplicates will have a very high dot product (or cosine similarity, which is a normalised version), so you can set a threshold above which you keep only one. Ordering is then simply sorting the remaining chunks by their similarity score, with the caveat that models attend unevenly across the context, so you may place the top-scoring chunk first and then the others, possibly with a special 'best' position.

For each chunk i, its similarity to the query is s_i = q · c_i = Σ_j q_j × c_ij, where q is the query vector, c_i is the chunk vector, and j runs over all dimensions. Deduplication: for any two chunks a and b, if c_a · c_b > θ (a threshold), treat them as duplicates and keep the one with the higher s_i. Ordering: sort the surviving chunks in descending order of s_i; optionally, place the single best chunk at the very start (or use a 'bento' ordering to account for positional attention bias).


## A worked example

Suppose we have a query about 'apple fruit nutrition' and three chunks from different documents. After embedding (which you will learn about in the Embeddings concept), each is a vector of just 3 numbers for this example: Query: [1, 0, 1], Chunk A (from a nutrition manual): [0.9, 0.1, 0.8], Chunk B (a recipe for apple pie): [0.2, 0.9, 0.3], Chunk C (a duplicate of A, from a different source, slightly rephrased): [0.85, 0.12, 0.78]. We will compute the dot product between the query and each chunk, then deduplicate, then order.

Step 1: Query · Chunk A = (1×0.9) + (0×0.1) + (1×0.8) = 0.9 + 0 + 0.8 = 1.7   # Dot product step 1: multiply corresponding numbers and add.
Step 2: Query · Chunk B = (1×0.2) + (0×0.9) + (1×0.3) = 0.2 + 0 + 0.3 = 0.5   # Chunk B is about pie, not nutrition, so its score is lower.
Step 3: Query · Chunk C = (1×0.85) + (0×0.12) + (1×0.78) = 0.85 + 0 + 0.78 = 1.63   # C is a near-duplicate of A, so its score is close to A's.
Step 4: Similarity between A and C (dot product) = (0.9×0.85) + (0.1×0.12) + (0.8×0.78) = 0.765 + 0.012 + 0.624 = 1.401   # To deduplicate, we compute the dot product between A and C themselves, not against the query. The high score indicates they are near-duplicates.
Step 5: Threshold = 1.3, so 1.401 > 1.3 → keep only one of A or C, say A   # If the similarity between two chunks exceeds a threshold, they are considered redundant.
Step 6: Final set, ordered by query score: [A (1.7), B (0.5)]   # A is placed first, B second, because A has the higher dot product with the query.

Result: The retrieved set has just two chunks, with A first, and B second. A and C were deduplicated because they were near-identical, saving context space. The ordering reflects relevance to the query.


## From scratch, in Python

```python
We'll write a small Python program that takes a query and a list of chunks, computes similarity using a simple dot product (we'll assume each chunk is already a list of numbers), deduplicates by comparing chunks to each other, and orders the result.

# Step 0: Teach Python basics as we go.
# A list in Python is a collection that keeps items in order and can be changed.
# For example, my_list = [3, 1, 2] is a list of three numbers.
# We'll use lists to hold vectors and a list of chunks.

# Step 1: Define a function to compute the dot product of two lists of numbers.
def dot_product(a, b):
    total = 0
    for i in range(len(a)):  # loop over each position in the lists
        total = total + a[i] * b[i]
    return total

# Step 2: Set up our data.
# Each chunk is a tuple (a list that cannot be changed) containing the vector and a label.
# Tuples are like lists but fixed once created; we use them here to keep the chunk's
# vector and label together without worrying about accidental changes.

query = [1, 0, 1]  # the query vector

chunks = [
    ([0.9, 0.1, 0.8], 'A: nutrition manual'),
    ([0.2, 0.9, 0.3], 'B: apple pie recipe'),
    ([0.85, 0.12, 0.78], 'C: duplicate of A')
]

# Step 3: Compute similarity scores.
# We'll store them in a list of tuples (score, vector, label)
scored = []
for vector, label in chunks:
    score = dot_product(query, vector)
    scored.append((score, vector, label))

# Step 4: Deduplication.
# We'll compare each chunk to every other chunk. If their dot product is above a
# threshold, we consider them duplicates and remove the one with the lower query score.
threshold = 1.3
final_chunks = []
for i in range(len(scored)):
    is_duplicate = False
    for j in range(len(scored)):
        if i != j:
            # Compute similarity between chunk i and chunk j
            sim = dot_product(scored[i][1], scored[j][1])
            if sim > threshold and scored[i][0] < scored[j][0]:
                is_duplicate = True
                break
    if not is_duplicate:
        final_chunks.append(scored[i])

# Step 5: Order final chunks by descending score (highest first).
# We'll sort using Python's sorted with a key function that returns the score.
def get_score(item):
    return item[0]
final_chunks_sorted = sorted(final_chunks, key=get_score, reverse=True)

# Step 6: Print the result.
print("Retrieved and ordered chunks:")
for score, vector, label in final_chunks_sorted:
    print(f"Score: {score:.2f} - {label}")

# Output:
# Score: 1.70 - A: nutrition manual
# Score: 0.50 - B: apple pie recipe
```


## The same thing, with a framework

```python
In practice, you would never write the dot product by hand. A library like NumPy does it in one line. NumPy is a library that provides fast operations on lists of numbers, calling them 'arrays'. Under the hood, numpy.dot(a, b) is doing exactly the same multiply-and-add loop you wrote above, but in optimised C code. So you would use something like:

import numpy as np

# Vectors as NumPy arrays
query = np.array([1, 0, 1])
chunk_matrix = np.array([[0.9, 0.1, 0.8],
                         [0.2, 0.9, 0.3],
                         [0.85, 0.12, 0.78]])

# Compute dot product of query with every chunk (matrix-vector product)
scores = np.dot(chunk_matrix, query)  # same as multiply and add for each row

# Dedup: compute pairwise similarities (matrix multiply)
sim_matrix = np.dot(chunk_matrix, chunk_matrix.T)  # T is transpose, to align dimensions

# Apply threshold and keep only unique ones (pseudo-code; real code would use indexing)
# Order by score (simple sort)
order = np.argsort(scores)[::-1]  # indices from highest to lowest
print(scores, order)
```


## What the code does

We start with a query vector and a list of chunks, each represented by a vector and a label. The first function, dot_product, takes two lists and loops over them, multiplying corresponding entries and adding to a running total. That is the dot product you did by hand. Then we compute the score of each chunk against the query using that function. For deduplication, we compare every chunk to every other chunk: if the dot product between their vectors is above a threshold (1.3 in this example), and the current chunk has a lower query score than the other, we skip it. Otherwise we keep it. Finally we sort the kept chunks by their query score, highest first, using Python's sorted with a key function. The key function returns the score, and reverse=True gives descending order. The result is just the two unique chunks, A and B, with A first because it is more similar to the query. In the framework version, NumPy's dot product does the same thing across all rows at once, and argsort gives the order of indices from highest to lowest, which you can use to reorder the chunks.


## Common misconceptions

**Retrieval is just 'search, then copy-paste'.** — I just need to retrieve the top-k documents and shove them into the prompt. What else is there?

If you just take the top-k, you may get duplicates (same fact repeated) and miss a conflicting source that is actually important. Ordering also matters — the model reads the beginning and end more carefully, so where you place a chunk changes the answer. In fact: Multi-document retrieval is a deliberate pipeline: retrieve candidates, deduplicate, surface conflicts, and order the final set deliberately, not just take the first few matches from a search engine.

**Deduplication is only about exact duplicate text.** — If two chunks are not the same string, they are not redundant.

Two chunks can say the same thing in slightly different words — a summary and a full passage, or two translations. They are semantically redundant, and both would waste context if both are included. In fact: Deduplication uses semantic similarity, often measured by embedding similarity, to detect near-duplicates, not just exact text identity.

**The most relevant chunk should always be the first one.** — I should just sort by similarity score and put the highest first — that is the best order.

The model's attention is not uniform; it may read the middle of the context more carefully in some settings. The optimal order may involve putting the best chunk at the very start, but also possibly at the end, or interleaving different types of evidence. Simple sorting is a baseline, not the best. In fact: Ordering should account for the model's positional attention bias; you may intentionally place the top evidence at the beginning and end, and put less critical evidence in the middle.

**Conflicting sources should be merged silently.** — If two docs say different things, I should just pick the one that seems more accurate.

The retrieval system has no way to know which is more accurate; silently choosing can hide a real conflict and mislead the user. In fact: Multi-document retrieval should flag conflicts to the language model or the user, so the answer can say 'sources disagree' rather than fabricate a single truth.

**More retrieved chunks are always better.** — We have a 128k context, so let's fill it with as many relevant chunks as possible to give the model more info.

The context window is finite, and each extra chunk can dilute attention, introduce noise, and increase the chance of distraction. Also, processing more tokens is more expensive and slower. In fact: The goal is to retrieve the smallest set that contains all necessary evidence — precision matters as much as recall.

**Retrieval quality is only about the embedding model.** — If my retrieval is bad, I should just switch to a better embedding model.

Retrieval quality is also strongly affected by chunking strategy, deduplication, ordering, and the retrieval algorithm itself (hybrid search vs pure vector). As your notes say, chunking strategy affects retrieval quality more than most model choices. In fact: A robust retrieval system tunes chunking, deduplication, and re-ranking in addition to the embedding model.


## In short

Multi-document retrieval is the deliberate assembly of evidence from multiple sources for a single answer. It involves three core operations: deduplication (removing redundant chunks, whether exact or semantic), ordering (placing the most relevant evidence where the model will attend to it, not just sorting by score), and conflict surfacing (flagging contradictions rather than hiding them). The dot product is the fundamental operation that measures similarity between query and chunk, and between chunk and chunk. This step is the difference between a RAG system that answers from evidence and one that repeats itself and confuses the model. If you forget everything else, remember: gather from several sources, remove what is repeated, surface what clashes, and order what remains for the model's attention.


## Where this is used

**In large language models (LLMs):** Modern LLMs are trained on enormous text and can answer from their internal memory, but they are often hooked up to a retrieval system so they can answer from a specific document collection. The multi-document retrieval step assembles the set of chunks that go into the context window. Because a context window may hold, say, 128,000 tokens but a document collection could be millions of tokens, retrieval must pick the few thousand tokens that matter. Deduplication is crucial: if the same paragraph appears in three documents, feeding it three times wastes tokens and makes the model over-weight that information. Ordering matters because the model's attention is not uniform — it tends to focus on the beginning and end of its context; putting the most relevant evidence at the start (or using a 'bento' layout with the best in both first and last) improves answer quality. Conflicts between sources are also surfaced: if two chunks state contradictory facts, the model should be prompted to mention both, not silently choose one, which is why this concept matters for factual reliability. Developers using libraries like LangChain or LlamaIndex write code that calls a retriever object, gets back a list of documents, applies a deduplication step (often by embedding similarity), then passes the ordered list to the model, possibly with a system prompt that says 'cite evidence from the context only'.

**In small language models running on devices (like phones or laptops):** On a device, the model is smaller and has a shorter context window, so the retrieved set must be even more frugal. Deduplication becomes more critical because every redundant token eats into the tiny context. Ordering is also more important because small models are even more sensitive to where evidence appears. Also, on-device retrieval must be fast and memory-light; so, instead of running a huge embedding model, developers might use bm25 (a keyword-based scoring method) or a small embedding model, and deduplication may be done by exact hash comparison (if two chunks are identical text) rather than semantic similarity. The mechanism is the same, but the threshold and the cost of computation change.

**In retrieval-augmented generation (RAG):** This is the heart of RAG. The pipeline is: ingest, chunk, embed, retrieve, rerank, generate. Multi-document retrieval is what happens between 'retrieve' and 'rerank'. After you get a big set of candidate chunks from a vector database, you then deduplicate and order them. A common production stack, as noted in your library's source 'Large Language Models: From Foundations to Production AI,' is hybrid search (combining keyword and vector similarity) plus re-ranking. The deduplication step often happens before re-ranking because re-ranking is expensive (uses a cross-encoder that scores query-document pairs jointly). The output of multi-document retrieval is the final context passed to the generator, which must be grounded in those chunks. If you do not deduplicate, you waste re-ranking budget and context; if you do not order well, the generator might miss the best evidence.

**In agentic systems:** Agents that plan and call tools may perform multi-document retrieval repeatedly. For example, an agent asked 'What is the company's policy on remote work in India, and how does it differ from the UK?' will need to retrieve from India policy, UK policy, and possibly an FAQ. It will call a retrieval function, get back a set of chunks, deduplicate where the same policy is mentioned in two documents, and order them so the model can synthesise. The agent might then realise it needs more information and search again — this is agentic retrieval. Multi-document retrieval is the core function that the agent calls; its quality decides whether the agent can answer at all. If it returns contradictory chunks without flagging the conflict, the agent may confidently give a wrong answer.

**In everyday applied machine learning (recommendation, vision, forecasting, etc.):** Multi-document retrieval is the same as multi-source fusion. In recommendation, you retrieve candidate items from different catalogues and deduplicate the same product listed twice, then order by relevance. In vision, you might retrieve similar images from several databases and deduplicate duplicates (same image from different sources). In forecasting, you retrieve relevant historical reports and combine them. The underlying pattern is always: gather from multiple sources, remove redundancy, rank by relevance, and hand the top set to a decision model. A developer building such a system writes code that calls a database query, merges results by a key (e.g., product ID), and sorts by score. What breaks when done badly: redundant results waste the model's input budget, conflicting information leads to inconsistent predictions, and poor ordering means the best evidence is ignored. In enterprise settings, poor deduplication can also leak duplicate citations that look like multiple sources when they are one.

**In enterprise knowledge assistants:** Access control must be enforced at retrieval time: you cannot retrieve a document the user is not allowed to see, even if it is highly relevant. Multi-document retrieval must combine permissions with relevance, and deduplication must be done within the user's allowed set. Conflicts between stale and current documents are common in internal knowledge bases; the retrieval step must surface both versions or the newer one, depending on policy. As your notes say, 'stale and contradictory internal documents are the normal case' — so the deduplication and conflict-surfacing parts of multi-document retrieval are not optional extras, they are core to the assistant being safe and trustworthy.


---

Read this on the web: https://basavapurushottam.com/learn-ai/multi-document-retrieval
