# Vector Search

Finding the nearest embeddings to a query vector, using approximate algorithms to keep the search fast at scale.

**Rests on:** Vector Native Storage

**Leads to:** Hybrid and Semantic Search

*LLMs and Generative AI · Retrieval-Augmented Generation*


## Why learn this

Vector search is the engine that lets a computer find the most relevant information by meaning, not just by exact words. This unlocks Retrieval-Augmented Generation (RAG) — the technique that lets a large language model answer questions using documents you give it, instead of only what it memorised. With vector search you can build: a chatbot over your company's manuals, a search box that finds a paragraph even when the query uses different words, and a recommendation system that finds similar movies based on description. It is also the foundation for agentic retrieval, where a model decides what to look up next. Master vector search, and you will understand the retrieval half of modern AI systems.


## The intuition

Vectors are lists of numbers. In vector search, every document and every query is turned into a vector by an embedding model. The vectors are placed in a space so that similar meanings are closer together. When you ask a question, its vector is placed in the same space, and the search finds the documents whose vectors are closest to it. Exact search checks every vector one-by-one, which is slow for millions of documents. Approximate search uses clever shortcuts — like grouping vectors into clusters or using a tree of regions — to look at only a fraction of the candidates, trading a little accuracy for a huge speed gain. The key mental model: the vector database is the retrieval engine, not an implementation detail.


## An analogy

Imagine you have a huge library of books. You want to find a book about 'how birds fly'. Exact word search would only find books containing the phrase 'how birds fly', missing books titled 'Aerodynamics of soaring'. Vector search is like having a librarian who, instead of matching words, understands the *meaning* of books and places them on shelves so that similar ideas are near each other. You ask your question, the librarian finds the idea's location on the shelf, and picks the few books closest to it. This works great for meaning, but the librarian is lazy: instead of walking the entire library to find the closest book, they only look at a few shelves that are most likely to contain it (the approximate part). This saves time, but occasionally they miss a great book that was far away. That is the trade-off: speed versus perfection.


## Definition

Vector search is the process of finding the embeddings (lists of numbers that represent meaning) in a database that are most similar to the embedding of a query, using approximate algorithms to keep the search fast even when the database has millions of entries.


## How it connects

Vector search builds on embeddings and vector databases. It is a core part of Retrieval-Augmented Generation (RAG), where documents are embedded and then retrieved at query time. It connects to Hybrid and Semantic Search, where vector similarity is combined with keyword matching for better results. It is also linked to Agentic Retrieval, where the search is repeated in a loop, and to Context Precision and Recall, which measure how good the retrieval is. You will reuse these ideas in the next topics: Reranking, GraphRAG, and Multi-Document Retrieval. Your library notes say: 'Retrieval quality is measurable separately from answer quality' — vector search is that retrieval quality. Also, your notes mention 'The vector database is the retrieval engine' — that is the mental model here.


## The mathematics

This part of the system uses geometry, but in many dimensions at once. A vector is simply a list of numbers. For example, the text '[0.2, -1.5, 3.1]' is a vector with three entries. The number of entries tells us the dimension. In machine learning, embeddings (lists that represent meaning) often have hundreds of dimensions, such as 384 or 1536.

To find which vectors are close to each other, we measure distance or similarity. Two common measures are Euclidean distance and cosine similarity.

Euclidean distance is the straight-line distance between two points in space. If you have two vectors, a and b, you subtract them component by component, square the results, add those squares together, and take the square root of the total. This gives you the length of the line connecting point a to point b.

Cosine similarity looks at the angle between two vectors rather than their distance from the origin. It tells us how parallel the two lists are. To calculate cosine similarity, we first compute the dot product, then divide by the lengths of both vectors.

The dot product is calculated by multiplying corresponding entries in the two lists and adding up all those products. If a = [a1, a2, ..., an] and b = [b1, b2, ..., bn], the dot product is:
a1 times b1 plus a2 times b2 plus ... plus an times bn.

The length of a vector (often called its norm) is found by squaring each entry, adding those squares together, and taking the square root. So for vector a, the length ||a|| is the square root of (a1 squared plus a2 squared plus ... plus an squared).

Cosine similarity is then:
dot product of a and b divided by (length of a times length of b).

If the cosine similarity is 1, the vectors point in exactly the same direction. If it is 0, they are at right angles to each other.

To find the nearest neighbors for a query vector, the simplest method is to compute this distance or similarity score against every single vector in the database and pick the best one. This is called an exact search. However, if the database has millions of vectors, doing this calculation for every item is very slow. It takes time proportional to N, where N is the number of vectors.

To make this faster, we use approximate algorithms. These methods organize the vectors into structures, such as trees or clusters, so that the system only needs to check a small subset of candidates rather than all of them. This trades a tiny bit of accuracy for a large gain in speed.


## A worked example

We have three document vectors and one query vector. Each vector has 3 numbers (3 dimensions). We will use the dot product as the similarity score — a higher dot product means more similar.

Documents:
D1 = [1, 0, 0]
D2 = [0, 1, 0]
D3 = [0.5, 0.5, 0]

Query:
Q = [0.4, 0.6, 0]

Step 1: Calculate the dot product of D1 and Q.
Multiply each component of D1 with the corresponding component of Q, then sum the results.
D1 has components 1, 0, 0.
Q has components 0.4, 0.6, 0.
Calculation: (1 times 0.4) plus (0 times 0.6) plus (0 times 0).
Result: 0.4 + 0 + 0 = 0.4.

Step 2: Calculate the dot product of D2 and Q.
D2 has components 0, 1, 0.
Q has components 0.4, 0.6, 0.
Calculation: (0 times 0.4) plus (1 times 0.6) plus (0 times 0).
Result: 0 + 0.6 + 0 = 0.6.

Step 3: Calculate the dot product of D3 and Q.
D3 has components 0.5, 0.5, 0.
Q has components 0.4, 0.6, 0.
Calculation: (0.5 times 0.4) plus (0.5 times 0.6) plus (0 times 0).
First part: 0.5 times 0.4 equals 0.2.
Second part: 0.5 times 0.6 equals 0.3.
Third part: 0 times 0 equals 0.
Sum: 0.2 + 0.3 + 0 = 0.5.

Step 4: Compare the scores.
D1 score: 0.4
D2 score: 0.6
D3 score: 0.5
The highest score is 0.6, which belongs to D2.

Result: The most similar document to the query is D2 with a dot product score of 0.6.


## From scratch, in Python

```python
# Vector search from scratch: exact and a simple approximate

# Step 1: A few document vectors (each is a list of numbers)
documents = [
    [1.0, 0.0, 0.0],   # doc A
    [0.0, 1.0, 0.0],   # doc B
    [0.5, 0.5, 0.0],   # doc C
    [0.0, 0.0, 1.0],   # doc D
]
# Each vector has 3 numbers (3 dimensions), but real embeddings have 300+.

# Step 2: A query vector
query = [0.4, 0.6, 0.0]

# Step 3: Define a function to compute the dot product (how much they align)
# Dot product = sum of products of each dimension.
def dot_product(a, b):
    result = 0
    for i in range(len(a)):
        result += a[i] * b[i]
    return result

# Step 4: Exact search - compute dot product with every document, keep the max
best_index = -1
best_score = -1  # assuming scores are positive
for i, doc in enumerate(documents):
    score = dot_product(query, doc)
    print(f"Score with doc {chr(65+i)}: {score}")
    if score > best_score:
        best_score = score
        best_index = i
print(f"Exact search: best doc is index {best_index} (doc {chr(65+best_index)}) with score {best_score}")

# Step 5: Simple approximate search - group vectors into a list of buckets by their first element
# This is a toy version of an index. In real life it's a tree or a graph.
buckets = {}
for i, doc in enumerate(documents):
    # Use the first dimension to decide a bucket (coarse partition)
    bucket = int(doc[0] * 2)  # round to 0 or 1
    buckets.setdefault(bucket, []).append(i)

# Choose a bucket for the query
query_bucket = int(query[0] * 2)
print(f"Query goes to bucket {query_bucket}")

# Only search within that bucket (plus neighbor to be safe)
candidates = buckets.get(query_bucket, []) + buckets.get(query_bucket + 1, [])
print(f"Candidates: {candidates}")

best_approx_index = -1
best_approx_score = -1
for i in candidates:
    score = dot_product(query, documents[i])
    if score > best_approx_score:
        best_approx_score = score
        best_approx_index = i
print(f"Approximate search: best doc is index {best_approx_index} (doc {chr(65+best_approx_index)}) with score {best_approx_score}")
```


## The same thing, with a framework

```python
# Using libraries: numpy for calculations, faiss for the approximate index
import numpy as np
import faiss

# Vectors: as a numpy array (a table of numbers)
docs = np.array([
    [1.0, 0.0, 0.0],
    [0.0, 1.0, 0.0],
    [0.5, 0.5, 0.0],
    [0.0, 0.0, 1.0],
]).astype('float32')  # faiss prefers float32

# Faiss index for exact search (L2 distance)
index = faiss.IndexFlatL2(docs.shape[1])  # 'Flat' means exhaustive, 'L2' is Euclidean
index.add(docs)  # add documents to the index

# Query
query = np.array([[0.4, 0.6, 0.0]], dtype='float32')

# Exact search: return 2 closest
D, I = index.search(query, 2)  # D = distances, I = indices
print(f"Exact distances: {D}")  # lower is closer
print(f"Exact indices: {I}")

# Approximate index: IVF (inverted file) with 2 clusters
nlist = 2  # number of clusters
quantizer = faiss.IndexFlatL2(docs.shape[1])  # how to compare when assigning
index_approx = faiss.IndexIVFFlat(quantizer, docs.shape[1], nlist)
index_approx.train(docs)  # learn the clusters
index_approx.add(docs)  # add documents
index_approx.nprobe = 1  # how many clusters to check. 1 is aggressive, 4 is safer

D2, I2 = index_approx.search(query, 2)
print(f"Approx distances: {D2}")
print(f"Approx indices: {I2}")
```


## What the code does

From scratch: We have a list of lists `documents`. Each inner list is a vector. `dot_product` multiplies corresponding entries and sums them; this is the same as numpy's `dot`. Exact search loops through all documents, computes the dot product with the query, and keeps track of the best. Approximate search: we make a simple dictionary `buckets` that groups document indices by the value of their first dimension. We then look only at the bucket matching the query (plus a neighbor) and search only there. This shows the principle: check few candidates, not all. In the library version, faiss does this for you: `IndexFlatL2` does exact search using Euclidean distance. `IndexIVFFlat` clusters the vectors during training, then at search time checks only the closest clusters (controlled by `nprobe`). The `search` method returns distances (lower is better) and indices. The approximate result may differ from the exact — that's the trade-off.


## Common misconceptions

**Vector search finds exact matches** — Vector search works like Ctrl+F – it finds the documents that contain the exact same words as my query.

Vector search operates on embeddings, which are numbers representing meaning. Two documents with no shared words can be close if they have similar meaning (e.g., 'How do birds fly?' and 'Aerodynamics of avian locomotion'). In fact: Vector search measures similarity of meaning, not word overlap. It can match paraphrases, but may miss exact terms that the embedding did not capture.

**Approximate search is exact search** — The approximate algorithm still returns the true nearest neighbour, just faster.

Approximate algorithms deliberately skip some candidates to save time, so they can miss the actual nearest neighbour and return a slightly less close one. In fact: Approximate search trades a small loss in accuracy for a large gain in speed. The trade-off is controlled by parameters like nprobe in FAISS.

**The vector database is just a storage detail** — A vector database is just a place to put embeddings; the important part is the neural network.

The index structure and the algorithm determine what can be retrieved and how fast. A bad index means useless retrieval, no matter how good the embedding model is. In fact: The vector database is the retrieval engine. Its index structure (e.g., IVF, HNSW) determines the speed and accuracy trade-off, and its ability to filter by metadata makes results usable.

**Higher similarity always means relevance** — If the cosine similarity is 0.9, the document is definitely relevant to the query.

Similarity numbers are relative and depend on the embedding space. A high score could mean the embedding captured a different sense of the word, or the document is topically close but answers the wrong question. In fact: Similarity is a heuristic. It should be combined with keyword search (hybrid) and re-ranking to improve precision. Also, similarity thresholds vary by embedding model and domain.

**You only need one search index** — One vector index for all documents is enough.

Different document types or languages may need separate indices or embeddings. Also, metadata filtering may require separate indexes per category for performance. In fact: Often you create multiple indexes or use metadata filters to restrict the search to relevant subsets, improving both speed and precision.


## In short

Vector search converts documents and queries into numbers (embeddings) and finds the closest ones by a distance measure. Exact search is slow for large collections, so approximate algorithms (like FAISS, HNSW, IVF) trade a little accuracy for speed. It is the core of RAG and powers recommendation, image search, and more. Hybrid search with keywords plus re-ranking is the production-grade choice. Remember: the vector database is the engine.


## Where this is used

Vector search appears in almost every modern AI system that needs to fetch relevant information.

**Large Language Models (LLMs):** LLMs have a fixed context window, so to answer questions about a large corpus (e.g., company documents), you cannot feed everything. Instead, you embed the query and use vector search to retrieve the top-K passages, then place those in the context. This is the retrieval half of RAG. Without vector search, the model would only answer from its memorised knowledge, which may be stale. In practice, a developer uses a library like FAISS or a vector database (Pinecone, Weaviate) and writes something like `index.search(query_embedding, top_k=5)`. The results then go into the prompt.

**Small language models (on-device):** When a model runs on a phone or laptop, memory and speed are limited. Vector search is often used to offload general knowledge. Instead of the model memorising a huge dataset, it is paired with a small on-device database of embeddings (e.g., a FAQ). The model retrieves a few entries and generates an answer. This is called retrieval-augmented generation on-device. The index is built once and queried in milliseconds.

**Retrieval-Augmented Generation (RAG):** This is the main use. The pipeline: ingest doc → chunk → embed → vector search → rerank → generate. Vector search is the core retrieval step. The library notes emphasise: 'Hybrid search plus re-ranking is presented as the default production retrieval stack, not pure vector similarity.' So in production, vector search is often combined with keyword search (BM25) to handle identifiers and rare terms, and then re-ranked with a cross-encoder. The vector search may be fast but not perfect; reranking corrects errors.

**Agentic systems:** An agent does not just retrieve once. It may ask a sub-question, get results, evaluate if they suffice, and search again. Vector search is used to get an initial set of candidates. The agent can then decide to refine the query or fetch more. For multi-hop questions, the agent may issue multiple vector searches at each hop. The iterative nature means the search index must be fast, because the agent may query it many times.

**Everyday applied ML:**
- **Recommendation systems:** Represent users and items as vectors, then vector search finds the items closest to the user's vector. This is used in Netflix, Amazon, YouTube.
- **Vision:** Reverse image search: embed images and query with a new image.
- **Forecasting:** Find similar historical patterns by embedding sequences (e.g., time series) and searching for the most similar past periods.
- **Fraud detection:** Embed transaction features and search for similar past fraudulent transactions.
- **Speech:** Retrieve similar audio clips or transcriptions.

**What a developer actually writes:**
- For a RAG pipeline, in Python:
  ```python
  import faiss
  import numpy as np
  # ... embed documents ...
  index = faiss.IndexFlatL2(dim)
  index.add(doc_embeddings)
  query_emb = embed(query)
  D, I = index.search(query_emb, top_k)
  context = [documents[i] for i in I[0]]
  ```
- Or using a vector database client (e.g., Pinecone): `results = index.query(vector=query_emb, top_k=5, filter={"department": "legal"})`. Metadata filtering is crucial: you often want to restrict to a category.

**What breaks when done badly:**
- **Low recall**: The search misses relevant documents, and the answer is incomplete. This is unfixable downstream.
- **Low precision**: The retrieved set contains irrelevant chunks, wasting context budget and distracting the generator.
- **Slow search**: If the index is not approximate, latency kills the user experience.
- **Poor embeddings**: If the embedding model is not suited to the domain, vectors are meaningless.
- **Approximate index too aggressive:** With too few candidates, you miss the best match. Balancing speed and recall is an engineering trade-off.


---

Read this on the web: https://basavapurushottam.com/learn-ai/vector-search
