# Tokenization

Splitting text into the discrete units a model actually processes.

**Leads to:** BLEU and Reference Metrics, Byte-Pair Encoding, Chunking, Context Window, Instruction Tuning, Multi-Token Decoding, Reasoning Models, Temperature and Sampling, Text Mining at Scale, Token IDs and Special Tokens

**Rests on nothing.** A place to start.

*LLMs and Generative AI · Large Language Models*


## Why learn this

Tokenization is the first step every large language model takes with your text. Before a model can predict a next word, it must break your input into pieces it can look up in its vocabulary. This step silently decides three things that matter to you: the cost of every request to the service, how well the model handles languages other than English, and even whether it can do arithmetic correctly. Understanding tokenization helps you predict costs, debug unexpected model behavior, and make informed choices about prompts and model selection.


## The intuition

Imagine you are writing a message to a friend, but you are only allowed to send it in pieces, and each piece has a fixed label. The model works the same way: it does not read your text as characters or whole words. Instead, it has a fixed list of allowed pieces, and it must find a way to break your text into those pieces. This process is called tokenization. A token can be a whole word ('apple'), a part of a word ('app' and 'le'), or even a single character if nothing else fits. The model's vocabulary is the list of all allowed tokens, and each token has an ID number. When you send text to a model, it first converts it into a sequence of these IDs, and that sequence is what the model actually processes. This is not a detail you can ignore: the same text can be tokenized in many ways, and the choice affects how much the model 'sees' and how much it costs. For example, the sentence 'I love biryani' might become 5 tokens: 'I', ' love', ' bir', 'yani'. The model does not see spaces or punctuation; it sees these tokens in order. This is the first real step in how a model understands language.


## An analogy

Think of tokenization like packing a suitcase for a flight. You have a fixed set of luggage sizes the airline allows (these are your tokens). Some items fit in a single bag (common words like 'the'), but a bulky item (an uncommon word like 'supercalifragilistic') must be broken into smaller pieces that each fit. The airline's list of allowed bag sizes is the vocabulary, and you must use only those sizes. You cannot bring a bag that is not on the list. If a word is so rare that no piece of it fits, you might break it down to single letters, because those are always allowed (like a 'fragile' sticker) – this is the fallback to bytes. The packing process is the tokenizer. A good tokenizer is like an efficient packer: it uses the fewest bags possible for common words, but it never fails to pack anything, because single characters or bytes always fit. Where the analogy stops working: in a suitcase, the order of bags doesn't matter to the airline, but the model cares about the order of tokens because that order carries the meaning of the sentence. Also, a suitcase has a physical limit for the total weight, but a model's context window limits the number of tokens, not their total 'weight' – and that limit is where cost and memory come from.


## Definition

Tokenization is the process of breaking text into a sequence of discrete units (tokens) from a fixed vocabulary, which the model then maps to integer IDs for processing.


## How it connects

This is the entry point to understanding how large language models work. It connects directly to the Context Window (the maximum number of tokens a model can attend to), and Byte-Pair Encoding (the algorithm that chooses the vocabulary).


## The mathematics

Tokenization is the process of turning a string of characters into a list of smaller pieces called tokens. This is not arithmetic like addition or multiplication. It is a matching problem. You have a fixed list of allowed pieces, called the vocabulary. Your job is to chop the text into chunks that appear in that list.

The vocabulary is a finite set. Let us call its size V. This means there are exactly V different tokens allowed. We number them from 0 up to V minus 1. So if the vocabulary has 50,000 tokens, the IDs are the integers 0, 1, 2, ..., 49,999.

The tokenizer follows a specific rule to decide how to chop the text. The most common rule is called Byte-Pair Encoding (BPE). This rule looks for the longest possible chunk of characters that exists in the vocabulary. It takes that chunk out, records its ID, and then repeats the process on the remaining text until nothing is left.

Let T be the input text. Let u be the total number of tokens produced. The output is a sequence of IDs: id_1, id_2, ..., id_u. Each id_i is an integer between 0 and V minus 1. The value u depends on how the text is chopped. If you chop into many small pieces, u is large. If you chop into fewer, larger pieces, u is small. The goal of the vocabulary design is to make u as small as possible for common words, so that the model can process text faster and cheaper.

The key mathematical object is the sequence length u. This number determines how much memory the model needs and how long it takes to compute.


## A worked example

We have a vocabulary of 5 tokens: ['I', ' love', 'bir', 'yani', '.']. Each token has an ID.

ID 0 corresponds to the token 'I'.
ID 1 corresponds to the token ' love' (note the space at the beginning).
ID 2 corresponds to the token 'bir'.
ID 3 corresponds to the token 'yani'.
ID 4 corresponds to the token '.'.

The input text T is: 'I love biryani.'

Step 1: We look at the beginning of the text. The first character is 'I'. Does the token 'I' exist in our vocabulary? Yes, it is ID 0. We take this token.
The remaining text is ' love biryani.' (note the space after 'love').

Step 2: We look at the beginning of the remaining text. The first characters are ' love'. Does the token ' love' exist in our vocabulary? Yes, it is ID 1. We take this token.
The remaining text is 'biryani.'.

Step 3: We look at the beginning of the remaining text. The first characters are 'bir'. Does the token 'bir' exist in our vocabulary? Yes, it is ID 2. We take this token.
The remaining text is 'yani.'.

Step 4: We look at the beginning of the remaining text. The first characters are 'yani'. Does the token 'yani' exist in our vocabulary? Yes, it is ID 3. We take this token.
The remaining text is '.'.

Step 5: We look at the beginning of the remaining text. The character is '.'. Does the token '.' exist in our vocabulary? Yes, it is ID 4. We take this token.
The remaining text is empty.

Result: The sequence of IDs is [0, 1, 2, 3, 4]. The total number of tokens u is 5.


## From scratch, in Python

```python
# Let's build a tiny tokenizer by hand.
# This is a simplified version to show the idea.

# First, we need a vocabulary: a list of tokens.
# A list in Python is an ordered collection of items, written in square brackets.
vocab = ['I', ' love', ' bir', 'yani', '.']  # our tiny vocabulary

# We also need a mapping from token to ID.
# A dictionary in Python looks up values by keys, written in curly braces.
token_to_id = {'I': 0, ' love': 1, ' bir': 2, 'yani': 3, '.': 4}

# The input text we want to tokenize.
text = 'I love biryani.'

# We will tokenize by scanning the text and matching the longest token from vocab.
# This is a simplified greedy approach (not exactly BPE, but shows the idea).

def tokenize(text, vocab, token_to_id):
    tokens = []  # an empty list to hold the tokens we find
    i = 0  # index in the text
    while i < len(text):
        # Try to match the longest token starting at position i
        matched = None
        for token in vocab:
            if text.startswith(token, i):
                # If this token is longer than the current best match, keep it
                if matched is None or len(token) > len(matched):
                    matched = token
        if matched is None:
            # If no token matches, we'd normally fall back to characters or bytes.
            # For simplicity, we'll just raise an error.
            raise ValueError(f'No token matches at position {i}: {text[i:]}')
        tokens.append(matched)  # add the matched token to our list
        i += len(matched)       # move the index forward by the length of the match
    return tokens

# Tokenize the text
result = tokenize(text, vocab, token_to_id)
print(result)  # should print ['I', ' love', ' bir', 'yani', '.']

# Now convert to IDs
ids = [token_to_id[t] for t in result]
print(ids)  # should print [0, 1, 2, 3, 4]
```


## What the code does

The code defines a vocabulary as a list and a token-to-ID mapping as a dictionary. The tokenize function takes the text, the vocabulary, and the mapping, and returns a list of tokens. It uses a while loop to scan through the text. At each position, it tries to find the longest token in the vocabulary that matches the beginning of the remaining text. This is a simplified greedy longest-match approach, similar in spirit to the longest-match rule in Byte-Pair Encoding. If no token matches, in a real tokenizer it would fall back to bytes (single characters or byte values), but here we raise an error for simplicity. Once a token is matched, it is added to the list, and the index moves forward by the token's length. Finally, a list comprehension converts each token to its ID using the dictionary. The output is the list of tokens and then the list of IDs, which is what the model would actually receive.


## Common misconceptions

**Tokenization is just splitting on spaces** — I can just use text.split(' ') to tokenize, that's what the model does.

Real tokenizers do not split on spaces. They use subword algorithms like BPE that consider spaces as part of tokens and break words into smaller pieces based on frequency. Splitting on spaces would leave many words out of vocabulary. In fact: Tokenization is a learned, frequency-based process that produces subword tokens. A word like 'unhappiness' might become 'un', 'happi', 'ness'.

**Each word is one token** — Every English word is a single token, so token count equals word count.

Common words may be one token, but many words are split into multiple tokens. Verb forms, compound words, and rare words often split. On average, a word is about 1.3 tokens. In fact: Tokens are subword units; a single word can be one token or several. There is no guarantee of a one-to-one mapping.

**Tokenization is lossless** — Tokenization preserves all information, so the model sees exactly what I typed.

The tokenizer decides what constitutes a unit. It may drop or merge certain details (like exact spacing or case) depending on the vocabulary. Some information can be lost or represented ambiguously. In fact: Tokenization is a lossy compression: it maps text to a fixed vocabulary, and the model only ever sees the IDs. Minor variations in spelling might become the same token.

**Changing the tokenizer is easy** — If my model has trouble with Hindi, I can just swap in a better Hindi tokenizer.

The tokenizer is trained together with the model. The model's embeddings and predictions depend on the exact token IDs. Changing the tokenizer invalidates the model. In fact: The tokenizer is baked into the pretrained model. To get a different tokenizer, you typically need a different model or a fully new pretraining run.

**Tokenization determines meaning** — If two texts tokenize the same way, they must mean the same thing.

Tokenization only breaks text into units; it does not encode meaning. Two different sentences could share tokens but mean opposite things (e.g., 'not good' vs 'good'). In fact: Meaning comes from the sequence of tokens and the model's weights, not the tokenization alone.


## In short

Tokenization is the process of converting raw text into a sequence of tokens from a fixed vocabulary, each mapped to an integer ID. It is the first and unavoidable step for any language model, directly affecting cost, context window usage, multilingual quality, and arithmetic ability. The tokenizer is part of the pretrained model and cannot be changed. Remember: tokens are not words, and token count is not word count. Understanding tokenization is the key to estimating costs, debugging model output, and making informed choices about models and prompts.


## Where this is used

Tokenization is everywhere in AI systems. In large language models, every prompt and every generated response is tokenized. The tokenizer runs before the model sees any text, and it runs again on the output. In practice, a developer sending an API call to a model like GPT-4 does not manually tokenize; the SDK does it automatically. But the token count appears on the bill: you are charged per token, not per word. A typical English word is about 1.3 tokens, but this varies. For example, 'ChatGPT' might be one token, while 'biryani' might be two. This affects cost and also the context window: a longer tokenization means fewer tokens fit in the window. In small models running on a phone or laptop, tokenization must be fast and small. These models often use a compact vocabulary to keep the model size down. The tokenizer is part of the model bundle and cannot be swapped after training. Retrieval-augmented generation (RAG) also depends on tokenization: documents are chunked into pieces, and each chunk is embedded. The chunking (splitting text into segments) is a separate operation from tokenization, but they must work together. As your library notes from 'Large Language Models: From Foundations to Production AI' state: 'Tokenization and chunking are distinct operations that must work together — one is the model's alphabet, the other is the retrieval unit.' When a user asks a question, the question is tokenized, and relevant chunks are retrieved and then tokenized too, before being fed to the model. Agentic systems that plan and call tools also rely on tokenization: every tool call and result is added to the conversation, and the total token count is the main constraint on how much the agent can 'remember'. In everyday applied machine learning, tokenization appears in sentiment analysis, spam detection, and translation. For example, a spam filter might tokenize an email into words and then look for certain tokens. The choice of tokenizer can dramatically affect multilingual performance: a tokenizer trained mostly on English will split Hindi or Tamil words into many small pieces, making the model slower and less accurate. Developers working with non-English languages often report that the same text takes more tokens in their language than in English, which is both a cost and quality issue. When building software, a developer rarely writes a tokenizer from scratch (they use libraries like HuggingFace tokenizers or the tokenizer built into the model's SDK). What they do write is code that counts tokens to estimate cost, truncates text to fit the context window, and chooses a chunk size for RAG. What breaks when done badly: if you truncate text in the middle of a token, you might confuse the model; if you underestimate token count, you exceed the context window and get an error; if you use a tokenizer not aligned with your model, you get nonsense. Therefore, understanding tokenization helps you anticipate and avoid these failures.


---

Read this on the web: https://basavapurushottam.com/learn-ai/tokenization
