# Chunking

Splitting documents into retrievable units.

**Rests on:** Tokenization

**Leads to:** Multi-Document Retrieval

*LLMs and Generative AI · Retrieval-Augmented Generation*


## Why learn this

Chunking is the first decision you make when building a system that answers questions from your own documents — like a company assistant that reads its manuals, or a lawyer searching case files. Get chunking wrong and no amount of clever model choice will fix it: the evidence is simply not there to retrieve. Get it right and the rest of the pipeline — embeddings, retrieval, generation — has something solid to work with. Mastering chunking unlocks the next steps in retrieval-augmented generation: embeddings, vector databases, and retrieval quality. It also connects directly to your neighbouring notes on admissible evidence spans and citations: if a chunk is too messy, you cannot trace an answer to a defensible passage.


## The intuition

Imagine you have a huge pile of loose papers — your company's entire policy handbook, say. Someone asks you: "What is the leave policy for new mothers?" You cannot hand them the whole pile; they would drown. You need to find the relevant page or paragraph and give them that. But the pile is not organised into helpful sections — it is one continuous stream of text. Chunking means cutting that stream into manageable pieces, each of which could be a coherent answer to some possible question. The size of each piece is a trade-off: cut too small, and you lose the context — a single sentence about 'maternity leave' might not mention the '12 weeks' that appear two sentences later. Cut too large, and you dilute the meaning — a 50-page chapter contains too many topics, so when you search for 'leave policy', the whole chapter matches, but the relevant part is buried. The art of chunking is choosing boundaries that respect the text's own structure — paragraphs, sections, tables — so each piece is self-contained and meaningful.


## An analogy

Think of chunking like cutting up a large pizza for a group of people. You want each person to get a slice that has a bit of crust, a bit of topping, and is easy to hold. If you cut huge slices, some people get mostly cheese, some get mostly crust, and the toppings are uneven. If you cut tiny slivers, each person gets a bite but has to take multiple slices, and the toppings get spread so thin that no one tastes the pepperoni properly. The same with text: a chunk that is too big mixes many topics, so when you search for 'leave policy', the system retrieves a chunk that also talks about 'work-from-home' and 'performance bonuses' — the answer is diluted. A chunk that is too small might only contain 'maternity leave' without the crucial '12 weeks' detail — the answer is incomplete. The best slices are cut along the natural lines of the pizza — the slice separators, which are like paragraph and section boundaries in a document. But the analogy breaks down: pizza slices are all roughly the same size, whereas chunks can vary — a short paragraph might stand alone, while a long section might need to be split. The pizza also has no overlap, but chunks often deliberately overlap so that sentences that straddle a boundary are not lost. So chunking is not a fixed grid; it is a thoughtful division that follows the text's own structure, with some overlap to preserve continuity.


## Definition

Chunking is the process of splitting a document into smaller, self-contained units — typically a few sentences or a paragraph — so that each unit can be separately indexed, retrieved, and used as evidence for an answer.


## How it connects

You have not yet studied embeddings or vector databases, but chunking is the first stage of a RAG pipeline. Your library notes that the pipeline is 'ingest, chunk, embed, retrieve, rerank, generate' — chunking comes second, right after you bring a document in. It is the highest-leverage choice because every later stage depends on what you feed it. If you chunk poorly, your embeddings (which you will learn later) have nothing meaningful to represent, your retrieval (which you will learn as 'hybrid and semantic search') cannot find the right passages, and your generation (which you know from LLMs) has no grounded evidence to cite. Your neighbouring notes on 'admissible evidence spans' and 'citations' build directly on chunking: to cite a precise passage, the chunk must be small enough to be a defensible unit. And your notes on 'context precision and recall' will show you how to measure whether your chunks are any good.


## The mathematics

Chunking is a way of breaking a long document into smaller pieces so a computer can read them one by one. The math here is simple arithmetic: counting, dividing, and adding. We do not need algebra or advanced formulas. We only need to know how many words are in the document, how many words we want in each piece, and how much those pieces overlap.

The main question is: How many chunks do I get?

To answer this, we look at three numbers:
1. total_words: The total number of words in your original document.
2. chunk_size: The maximum number of words you want in each chunk.
3. overlap: The number of words that are repeated at the end of one chunk and the beginning of the next. This helps keep context connected.

If there is no overlap, the calculation is easy division:
number_of_chunks = total_words divided by chunk_size.

But if there is an overlap, the math changes slightly. When you add a new chunk, you do not add 'chunk_size' new words. You only add 'chunk_size minus overlap' new words, because some of those words are already in the previous chunk. The first chunk uses its full size. Every chunk after that adds only the non-overlapping part.

So the formula is:
number_of_chunks = ceil( (total_words - overlap) divided by (chunk_size - overlap) )

Here is what each symbol means:
- total_words: The count of all words in the document.
- chunk_size: The target length of each chunk in words.
- overlap: The number of shared words between chunks.
- ceil(x): This means 'round x up to the next whole number.' For example, if you calculate 12.5 chunks, you cannot have half a chunk. You need 13 full chunks to cover the text. So ceil(12.5) is 13.

Why does this matter? 
Smaller chunks mean more pieces. More pieces mean higher precision because a search query will match fewer words at a time, reducing noise. But if chunks are too small, they might cut sentences in half, losing meaning.
Larger chunks mean fewer pieces. Fewer pieces mean less precision because each piece contains more unrelated information (dilution). But larger chunks keep sentences and paragraphs intact, preserving context.
The overlap adds redundancy. It ensures that key phrases spanning the boundary between two chunks are captured in both. This costs more storage and processing time but improves accuracy.

There is no single correct number for chunk_size. It depends on your data. But you can use this arithmetic to compare options.


## A worked example

Let us work through a specific example with real numbers.

Scenario:
- total_words = 1000 words.
- We will test two chunk sizes: 100 words and 200 words.
- We will also test an overlap of 20 words.

Case A: No overlap, chunk_size = 100.
Step 1: Divide total_words by chunk_size.
1000 divided by 100 equals 10.
Result: You get exactly 10 chunks. Each chunk has 100 words. There is no repetition.

Case B: No overlap, chunk_size = 200.
Step 1: Divide total_words by chunk_size.
1000 divided by 200 equals 5.
Result: You get exactly 5 chunks. Each chunk has 200 words. The chunks are larger, so they contain more context, but there is less precision because each chunk covers more ground.

Case C: Overlap = 20 words, chunk_size = 100.
Here, the first chunk takes 100 words. Every subsequent chunk adds only (chunk_size - overlap) new words.
Step 1: Calculate the number of new words added per chunk after the first.
100 minus 20 equals 80. So, each new chunk contributes 80 unique words.

Step 2: Adjust the total count for the formula.
The formula subtracts the overlap from the total_words first to account for the fact that the last chunk does not need a full 'overlap' buffer at the end (or rather, the standard sliding window logic applies). Let us use the provided formula:
(1000 - 20) divided by (100 - 20).

Step 3: Perform the subtraction.
1000 minus 20 equals 980.
100 minus 20 equals 80.

Step 4: Perform the division.
980 divided by 80 equals 12.25.

Step 5: Apply the ceiling function.
ciel(12.25) means round up to the next whole number.
The next whole number after 12.25 is 13.
Result: You get 13 chunks.

Let us verify this manually to be sure:
Chunk 1: Words 1-100 (100 words)
Chunk 2: Words 81-180 (adds 80 new words, total covered: 180)
Chunk 3: Words 161-260 (adds 80 new words, total covered: 260)
...
Chunk 12: Covers up to word 940 (12 * 80 = 960? No. Let's trace carefully).
Actually, the formula (N - O) / (S - O) + 1 is often used for sliding windows where the last chunk might be shorter or full. The formula given in the text was ceil((total - overlap)/(size - overlap)).
Let's check the coverage of 13 chunks with this logic:
First chunk covers 100 words.
The remaining 900 words (1000-100) are covered by subsequent chunks adding 80 words each.
900 / 80 = 11.25.
So we need 11 more chunks after the first one? That would be 12 total.
Wait, let's look at the formula again: ceil((1000-20)/(100-20)) = ceil(980/80) = ceil(12.25) = 13.
Why 13?
If we have 13 chunks:
Chunk 1 ends at word 100.
Chunk 2 starts at word 81, ends at 180.
Chunk 13 starts at word (13-1)*80 + 1 = 961? No.
Let's use the standard sliding window definition: Chunk i starts at word (i-1)*(size-overlap) + 1.
Chunk 13 starts at 12*80 + 1 = 961. It ends at 961+99 = 1060. This covers the end of the document (word 1000).
Chunk 12 starts at 11*80 + 1 = 881. Ends at 980.
So Chunk 13 is needed to cover words 981-1000.
Thus, 13 chunks is correct.

Comparison:
Case A (10 chunks): High precision, no context loss across boundaries, but less context per chunk.
Case B (5 chunks): Low precision, high context per chunk.
Case C (13 chunks): Highest precision, good context preservation via overlap, but highest redundancy (more data to process).

The arithmetic shows that adding overlap increases the number of chunks significantly. You must weigh this cost against the benefit of better retrieval accuracy.


## From scratch, in Python

```python
def chunk_text(text, chunk_size, overlap):
    """Split a text into chunks of roughly chunk_size words, with overlap."""
    words = text.split()  # split on spaces to get a list of words
    chunks = []           # empty list to hold the chunks
    i = 0                 # start at the beginning

    while i < len(words):       # keep going until we pass the last word
        # Take a slice of words from position i, of length chunk_size
        chunk = words[i:i + chunk_size]
        # Join them back into a single string
        chunk_text = ' '.join(chunk)
        chunks.append(chunk_text)
        # Move forward by (chunk_size - overlap) words
        i += chunk_size - overlap

    return chunks

text = "This is a sample document about leave policies. " * 20  # just for testing
result = chunk_text(text, 10, 2)
print(len(result))  # prints the number of chunks
print(result[0])    # prints the first chunk
```


## The same thing, with a framework

In practice, you might use a library like `langchain` or `llama_index`, but the principle is the same. For example, with `langchain`:
```python
from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=100,  # target chunk size in characters, not words
    chunk_overlap=20
)
chunks = splitter.split_text(text)
```
Underneath, `RecursiveCharacterTextSplitter` tries to break at paragraph or sentence boundaries first, and only falls back to character-level splitting if a chunk is still too big. It does the same thing as our `chunk_text` function, but with smarter boundaries.


## What the code does

The `chunk_text` function starts by splitting the input text into a list of words using `.split()` — this is a list, which is a collection that keeps things in order and can be changed. We create an empty list `chunks` to store the output. The `while` loop repeats as long as the position `i` is less than the number of words. Inside the loop, we take a slice of the `words` list from position `i` up to `i + chunk_size` — this is like cutting out a piece of the list. The slice is a new list, so we join it back into a string with `' '.join()`. We add that string to the `chunks` list using `.append()`. Then we move `i` forward by `chunk_size - overlap` — this is the arithmetic we learned earlier: each new chunk overlaps the previous one by `overlap` words. The function returns the list of chunks. When we test it with a text that is 200 words long (20 copies of a 10-word sentence), with chunk_size=10 and overlap=2, we get about 24 chunks because each step advances by 8 words.


## Common misconceptions

**Chunking is just cutting text into equal-length pieces** — I can just split on spaces every N words, that's the easiest way.

Equal-length cuts ignore the text's structure. A sentence might be split mid-thought, and the two halves lose their connection. Paragraph boundaries exist for a reason. In fact: Chunking should respect existing boundaries — paragraphs, sentences, sections — so each chunk is a coherent unit. The goal is not uniform length but semantic coherence.

**Bigger chunks are always better because they have more context** — If I make my chunks really long, the model will have all the information it needs.

A long chunk contains many topics, so its embedding is a blur. A query for 'maternity leave' returns a chunk that is mostly about 'performance bonuses', and the relevant sentence is diluted. In fact: The right chunk size is small enough that each piece is about one thing, but large enough to contain full reasoning. There is a sweet spot that must be tested for your specific corpus.

**Overlap is cheating — you should never duplicate text** — Overlap means I am storing the same words twice, that's wasteful.

Overlap is a deliberate strategy to preserve context across chunk boundaries. If a sentence spans two chunks, without overlap you lose half of it. The cost is a little extra storage — usually worth it. In fact: Overlap (e.g., 10–20% of the chunk size) ensures that information at the boundaries is not lost, and it is standard practice in RAG.

**There is one universal chunk size that works for everything** — I saw a tutorial that used 256 words, so I will use that for my documents too.

The best chunk size depends on your documents and your queries. A manual of short commands should be chunked by command; a legal ruling should be chunked by paragraph. Copying a number blindly leads to poor retrieval. In fact: Chunk size and overlap are hyperparameters you should experiment with, using retrieval metrics like precision and recall to evaluate different settings.

**Chunking is only about word count** — I just count characters and split.

Character counts do not respect sentence or paragraph boundaries. A 500-character chunk might cut a sentence in half, while a 200-character chunk might contain a full paragraph. Natural language has structure. In fact: The best chunkers are 'structure-aware': they try to break at paragraph or sentence breaks first, and only fall back to character counts if the text is one long blob.


## In short

Chunking is the art of cutting documents into retrievable units. Small chunks are precise but lose context; large chunks are comprehensive but dilute meaning. The right strategy respects natural boundaries (paragraphs, sentences) and uses overlap to preserve continuity. There is no one-size-fits-all chunk size — you must test and measure. Chunking is the highest-leverage decision in a RAG pipeline, and getting it right is the foundation for useful retrieval, grounded answers, and defensible citations.


## Where this is used

Chunking is used everywhere in retrieval-augmented generation (RAG). In large language models (LLMs) — the kind behind ChatGPT — RAG is how they answer questions about documents they were not trained on. For example, a legal AI might be given a set of court rulings. The developer first chunks each ruling into paragraphs or sections, then embeds those chunks into a vector database. When a user asks a question, the system finds the most similar chunk and feeds it to the LLM as context. The chunking strategy determines whether the LLM has the right evidence to cite. In small models running on devices — like a keyword spotter on a phone — chunking helps because the model's context window is tiny. A small model can only read a few hundred tokens, so chunks must be very small and self-contained. For example, a voice assistant that connects to a smart home manual might chunk the manual by each command, so the phone can retrieve just the relevant instruction without loading the whole manual. In agentic systems — AI that plans and calls tools — chunking matters because the agent may need to retrieve multiple pieces of evidence to answer a multi-hop question. If chunks are too coarse, the agent cannot isolate the specific facts it needs to combine. A developer using an agent framework might set chunk sizes to 200 words with 20-word overlap to allow the agent to retrieve a fact from one chunk and a related fact from another, then stitch them together. In everyday applied machine learning — like recommendation systems or fraud detection — chunking is less common, but it appears in document classification: if you need to classify a long report, you might chunk it into sections and classify each separately, then aggregate. But for RAG, chunking is critical. A developer writing a RAG pipeline reaches for a library like `langchain` or `llama_index`, and the first thing they write is the splitter: `RecursiveCharacterTextSplitter` or `SentenceSplitter`. They set the chunk size based on the embedding model's max tokens (for example, if the embedding model only handles 512 tokens, they choose a chunk that fits that). What breaks when chunking is done badly? If chunks are too large, retrieval returns a wall of text that overwhelms the LLM's context window, so answers are unfocused or the model ignores the evidence. If chunks are too small, crucial context is missing, so the model guesses — and the answer may be wrong or hallucinated. If boundaries cut mid-sentence, the text is ungrammatical, and the embedding loses meaning. In production, teams often spend weeks tuning chunk size and overlap, and they measure the effect on context precision and recall — as your library notes say, retrieval quality is measurable separately from answer quality.


---

Read this on the web: https://basavapurushottam.com/learn-ai/chunking
