# Bi-Encoders and Cross-Encoders

Bi-encoders embed query and document separately for fast retrieval; cross-encoders score them jointly for accuracy.

**Rests on:** Autoencoders

*Deep Learning · Natural Language Processing*


## Why learn this

This concept is the key to building search systems that are both fast and accurate. It unlocks your ability to understand how retrieval-augmented generation (RAG) works, how question-answering systems find relevant documents, and how recommendation engines match items to users. Once you master this, you can move on to more advanced topics like embeddings, vector databases, and agentic systems that use search as a tool.


## The intuition

Imagine you have a library with thousands of books and you need to find the ones about 'Indian history'. You could do this in two ways. First, you could quickly scan the titles or maybe a short summary on the spine of each book. This gives you a 'shortlist' of maybe twenty books. Then, you could read the introduction of just those twenty books to pick the best three. The first method is fast but rough; the second is slow but precise. Bi-encoders and cross-encoders are the two halves of this approach, but for text. A bi-encoder is like the first method: it converts all the books into a short summary (an embedding) and then compares that summary to the summary of your query. It is very fast because it can precompute all the book summaries ahead of time. A cross-encoder is like the second method: it reads the full book and your full query together, in detail, to give a very accurate score. But it has to do this for every pair, which is too slow for a large library. The trick is to use a bi-encoder to find a shortlist, and then a cross-encoder to re-rank that shortlist.


## An analogy

Think of a speed-dating event. There are two stages. In stage one, you, the query, have a short, quick chat with each potential match (the documents). You can't learn much in five minutes, but you can quickly eliminate people who are clearly not a good fit. This is the bi-encoder. You generate a quick 'vibe' profile for each person, and you compare it to your own. If they don't match on basic interests, you move on. This stage is fast because you've done it for everyone, and you remember the profiles. Now, imagine you have a shortlist of five people. For stage two, you go out on a long, deep dinner date with each of them. This is the cross-encoder. You have a long, detailed conversation that looks at everything about that one person in the context of you. This is much more accurate, but you can't do it with hundreds of people—it takes too long. The analogy ends where the computational trade-off becomes extreme: with millions of documents, the 'dinner date' (cross-encoder) is impossible for all of them, so the 'vibe check' (bi-encoder) is the only way to get the shortlist. The cross-encoder then adds precision to the shortlist, making the final results much better.


## Definition

A bi-encoder is a system that transforms the query and each document into separate, independent numerical representations (embeddings) and then computes a similarity score between them, allowing for fast pre-computation and retrieval; a cross-encoder is a system that processes the query and a document together as a single unit to produce a joint accuracy score, which is much slower but more precise.


## How it connects

This builds on your understanding of how computers represent text as numbers, which you are learning about. It directly uses the idea of an 'embedding' or a 'vector' from your notes on *Embeddings*. It also uses the concept of a 'score' or 'similarity' that you see in *Sentiment Analysis* (classifying how positive or negative a text is) and *Fake News Detection* (where you score how likely something is to be false). This concept also connects to *NLP for Mobility and Transportation* where you might need to search through incident reports, and it's a core component in *Emotional Intelligence in NLP* for retrieving contextually relevant empathetic responses. It is a crucial bridge from basic classification to complex retrieval and generation, leading to your notes on *Large Language Models* and *RAG*.


## The mathematics

This concept rests on **linear algebra**, which is the mathematics of lists of numbers called vectors. A vector is simply an ordered list, like a shopping list where the order matters (first item, second item, etc.).

The core operation we use to compare two vectors is called the **dot product**. The dot product measures how much two lists of numbers 'point in the same direction'. You can think of it as a numerical agreement score.

Here is how the dot product works step by step:
1. Take two vectors of the same length. Let's call them Vector A and Vector B.
2. Multiply the first number in Vector A by the first number in Vector B.
3. Multiply the second number in Vector A by the second number in Vector B.
4. Continue this for every position in the lists.
5. Add all those products together.

If both lists have large positive numbers in the same positions, they agree strongly, and the final sum (the dot product) will be a large positive number. If one list has a large positive number where the other has a large negative number, they disagree, and that part of the sum will be negative, lowering the total score.

We use this dot product to compare the 'summary number list' (called an embedding) of a search query with the embedding of a document. The higher the score, the more relevant the document is to the query.

There are two main ways systems handle this comparison:

**1. Bi-Encoders**
In a bi-encoder system, the mathematics is straightforward. We take the query and turn it into one vector (let's call it q). We take the document and turn it into another vector (let's call it d). Then we calculate the dot product of q and d.

The formula is:
similarity(q, d) = the sum of (q_i multiplied by d_i) for every position i.

Here, 'i' stands for the position number in the list. We do this for every position and add them up.

**2. Cross-Encoders**
In a cross-encoder system, the mathematics is more complex because we do not compare pre-made summaries. Instead, we combine the raw query and the raw document into one single input stream. This combined input is fed into a very deep neural network. The network processes the interaction between every word in the query and every word in the document.

The output of this network is a single number between 0 and 1. We call this score 's'.
score(q, d) = f( [q; d] )

In this formula:
- '[q; d]' means we join the query vector q and the document vector d end-to-end to make one long list.
- 'f' represents the deep neural network. It is a complex function that takes that long list as input and squashes the result into a number between 0 and 1.
- A score of 1 means a perfect match. A score of 0 means no match at all.


## A worked example

We have a query about 'solar energy in India' and three documents. The bi-encoder has already turned each of these into a 3-number list (an embedding). Let's say the query embedding is [2, -1, 3]. The document embeddings are:

- Doc A: [1, -1, 2] (about solar plants)
- Doc B: [-2, 3, 1] (about coal mining)
- Doc C: [2, -1, 3] (identical to the query, about solar energy in India)

We will compute the dot product between the query and each document. The dot product requires us to multiply corresponding numbers and then sum the results.

**Step 1: Calculate similarity for Doc A**
Query: [2, -1, 3]
Doc A: [1, -1, 2]

First, multiply the first numbers: 2 × 1 = 2
Second, multiply the second numbers: (-1) × (-1) = 1
Third, multiply the third numbers: 3 × 2 = 6

Now, add these products together:
2 + 1 + 6 = 9

So, the similarity score for Doc A is 9.

**Step 2: Calculate similarity for Doc B**
Query: [2, -1, 3]
Doc B: [-2, 3, 1]

First, multiply the first numbers: 2 × (-2) = -4
Second, multiply the second numbers: (-1) × 3 = -3
Third, multiply the third numbers: 3 × 1 = 3

Now, add these products together:
(-4) + (-3) + 3 = -7 + 3 = -4

So, the similarity score for Doc B is -4. The negative score means strong disagreement between the query and this document.

**Step 3: Calculate similarity for Doc C**
Query: [2, -1, 3]
Doc C: [2, -1, 3]

First, multiply the first numbers: 2 × 2 = 4
Second, multiply the second numbers: (-1) × (-1) = 1
Third, multiply the third numbers: 3 × 3 = 9

Now, add these products together:
4 + 1 + 9 = 14

So, the similarity score for Doc C is 14.

**Result**
The dot product scores are:
- Doc A = 9
- Doc B = -4
- Doc C = 14

Doc C has the highest score because it is identical to the query. This is how a bi-encoder retrieves a shortlist of relevant documents (for example, we might take Doc A and Doc C and ignore Doc B).


## From scratch, in Python

```python
def dot_product(a, b):
    """Compute the dot product of two lists."""
    result = 0
    for i in range(len(a)):
        result = result + (a[i] * b[i])
    return result

# Example embeddings
query_embedding = [2, -1, 3]
doc1_embedding = [1, -1, 2]
doc2_embedding = [-2, 3, 1]

# Compute scores
score1 = dot_product(query_embedding, doc1_embedding)
score2 = dot_product(query_embedding, doc2_embedding)

print("Score with doc1:", score1)
print("Score with doc2:", score2)
```


## The same thing, with a framework

In PyTorch (a popular library for machine learning), the dot product would be written as:

```python
import torch

query_embedding = torch.tensor([2., -1., 3.])
doc1_embedding = torch.tensor([1., -1., 2.])
score1 = torch.dot(query_embedding, doc1_embedding)
print(score1)
```

`torch.dot` does exactly the same multiply-and-add loop as the `dot_product` function, but it's a pre-built and highly optimized operation on tensors (a tensor is like a list of lists, but it's designed for mathematical calculations).


## What the code does

The code defines a function called `dot_product`. A function is a reusable set of instructions. It takes two inputs, `a` and `b`, which are defined as lists (ordered, changeable collections of items). Inside the function, we initialize a variable `result` to 0, which will hold our final score. Then, we use a `for` loop to go through each position in the list (using `range(len(a))` to get the numbers 0, 1, 2...). For each position, we take the value from list `a` at that position (a[i]) and multiply it by the value from list `b` at the same position (b[i]), and add the product to `result`. After the loop finishes, the function returns the final `result`. We then define two lists called `doc1_embedding` and `doc2_embedding` (these are the document vectors). We call the `dot_product` function twice, once for each document, and store the results in `score1` and `score2`. Finally, we print these scores. This is the same arithmetic you did by hand, but in a program.


## Common misconceptions

**Cross-encoders are slow but better, so I should always use them.** — The cross-encoder is more accurate, so I should just use it for everything.

A cross-encoder processes every query-document pair from scratch. With 100,000 documents, that means running the model 100,000 times. This is computationally and time-intensive, taking seconds or minutes. A bi-encoder only needs to run the model once per document (pre-computed) and once for the query, resulting in a simple arithmetic operation per document, which is milliseconds. In fact: The standard and only practical approach for large collections is to use a bi-encoder for retrieval to quickly narrow down the candidates, and then use a cross-encoder for re-ranking only the top few candidates to maximize accuracy.

**The dot product's highest score is always the correct answer.** — If a document has the highest dot product with my query, it must be the best answer.

The dot product can be affected by the length of the vectors. A document with a longer vector (more words or numbers) might have a higher dot product simply because it has more terms, even if it is less semantically relevant. Also, the embeddings from a bi-encoder are not perfect; they can miss subtle meanings, especially in conversational or domain-specific language. In fact: The dot product is a heuristic. It is a fast filter. The final answer is usually decided by the cross-encoder, which looks at the full context. The shortlist from the bi-encoder is just a starting point.

**Both encoders are the same, just with different names.** — A bi-encoder and a cross-encoder both convert text to numbers, so they are essentially the same.

The key difference is in *what* they convert and *when*. A bi-encoder converts each piece of text independently into a single vector. A cross-encoder takes both the query and the document *together* as a single input and outputs a single score. This means the cross-encoder can see the interaction between the words, while the bi-encoder cannot. In fact: Think of it as a quick summary (bi-encoder) versus a full joint reading (cross-encoder). They are used for different stages in a pipeline.

**Bi-encoders are worse, so they are not useful.** — Since cross-encoders are more accurate, bi-encoders seem inferior and unnecessary.

Bi-encoders are the only reason web-scale search is possible. Without them, you could not search billions of webpages in milliseconds. Their speed is their superpower. In fact: Bi-encoders are not 'worse'; they are designed for a different job: fast retrieval. They are fast *because* they precompute document vectors. They sacrifice a little accuracy for massive efficiency, making them indispensable.


## In short

To find the best matches between a query and a huge amount of documents, use a two-stage process: first, a bi-encoder quickly scores all documents by comparing their simplified summaries (embeddings). Precompute all document vectors to make this super fast. Then, take the top results from this stage and use a more powerful but slower cross-encoder to re-rank them by reading both the query and the document in full detail. This gives you the best of both worlds: speed and accuracy.


## Where this is used

In practice, this is the standard architecture for modern search systems, especially in **Retrieval-Augmented Generation (RAG)**. A typical RAG pipeline first uses a bi-encoder (like a model from the Sentence-BERT family) to embed all documents into a vector database, pre-computing all the document vectors. When a user asks a question, the system embeds the query with the same bi-encoder and uses a fast similarity search (like a dot product or cosine similarity) to find the top-20 most similar documents from the database. This is the retrieval stage. Then, a cross-encoder (often a model like a small BERT-based ranker) takes each of those 20 documents along with the query, processes them together, and outputs a more precise relevance score. The system re-ranks the documents by this new score and shows the top-5 results to the user. This two-stage process is crucial for LLMs: without the bi-encoder, searching a large corpus would be impossible due to speed; without the cross-encoder, the results would be too noisy for the LLM to generate accurate answers.

For **large language models** themselves, bi-encoders are used to create semantic search for internal documents, and also in the input layer to embed tokens (though they are more complex). In **small language models** running on phones, the bi-encoder is critical because it is lightweight and can run quickly on-device, but cross-encoders are often too heavy. The developer will use a knowledge distillation technique to train a small cross-encoder for re-ranking on-device.

For **agentic systems** (AI agents that can plan and call tools), bi-encoders and cross-encoders are used to select which tools or functions to call. The agent has a list of tool descriptions. It embeds them with a bi-encoder, and when it needs to act, it embeds its intent and retrieval a shortlist of possible tools. A cross-encoder can then refine the list to pick the exact tool.

For **recommendation systems**, this concept is used to match user profiles (the query) to millions of items (the documents). A bi-encoder can pre-embed all items, so when a user logs in, the system can quickly find a shortlist of items, which a cross-encoder can then re-rank based on more complex user-item interactions.

A developer would use a library like `sentence-transformers` in Python to load a pre-trained bi-encoder and cross-encoder. They would write code to encode the documents into vectors, store them in a vector database like FAISS or Pinecone, and then write a loop to encode the query, search the database, and re-rank. A common mistake is to try to use a cross-encoder for the first stage, which leads to an unusably slow system. Another mistake is using a bi-encoder for the final ranking, which can miss nuanced semantic matches that only a cross-encoder can capture.


---

Read this on the web: https://basavapurushottam.com/learn-ai/bi-encoders-and-cross-encoders
