# Context Window

The maximum number of tokens a model can attend to at once — its entire working memory for a given call.

**Rests on:** Token IDs and Special Tokens, Tokenization

**Leads to:** Agent Memory Systems, Context Packs

*LLMs and Generative AI · Large Language Models*


## Why learn this

This concept explains exactly why your AI assistant sometimes ignores the instructions you gave it at the start of the conversation, or why asking it to read a 500-page report results in vague summaries. It also reveals why sending huge amounts of text into these systems costs real money and takes more time. Whether you are using ChatGPT, Claude, Gemini, or running software on your own computer, every one of them has this specific limit. Understanding it tells you how to structure your requests so the model actually sees what matters, helps you avoid surprising errors when texts get too long, and explains why models behave differently depending on their size and where they run.


## The intuition

Think of the context window as the model’s entire working memory for a single interaction. When you send a message to an AI, it does not have a filing cabinet full of notes from yesterday, nor does it remember your previous chats unless you explicitly paste them into the current conversation again. The only thing it can process is the specific block of text provided in this one request. The context window is the maximum size of that block, measured in 'tokens'. A token is roughly a chunk of text smaller than a word; for example, the word 'hello' might be one token, but 'unhappiness' might be split into 'un', 'happy', and 'ness'—three tokens. The model can look at, say, 8,000 of these tokens at once, but nothing beyond that limit exists to it. It is like being asked to read a book while only holding one page in your hands. You can read that page perfectly, but you cannot flip back to check a detail from page five if the context window is smaller than the whole book. If you feed the model a 200-page document (roughly 100,000 tokens) but its window is only 5,000 tokens, it can only 'see' the last part of the document or specific snippets you choose to include. The early details are effectively invisible. This window is also a resource budget: processing more tokens takes longer (latency) and costs more money. However, having a larger window does not guarantee the model understands everything in it equally well; it often struggles to find key information in the middle of very long inputs.


## An analogy

Imagine you are a chef working on a single order in a tiny kitchen. Your countertop is your context window. You can only hold as many ingredients and tools on that counter at one time as will physically fit—perhaps enough for one meal. The rest of the recipes, spices, and supplies are stored in the pantry, out of reach. If the order requires an ingredient you didn’t place on the counter, you cannot use it until you clear space to bring it over. If the order is so complex that it requires thirty different ingredients simultaneously, your counter will overflow; you will drop things, miss steps, and spend too much time just trying to organize the chaos. This limits how much work you can handle in a single pass. Where the analogy breaks down slightly is that a chef might remember simple recipes by heart without looking at them, whereas the AI has no memory outside of what is currently on its counter. Also, while you can buy a bigger countertop for your kitchen, having more space doesn't make you a smarter cook; it just allows for more complex dishes to be prepared at once, provided you have the skills to manage the extra ingredients.


## Definition

The context window is the maximum amount of text (measured in tokens) that an AI model can process and remember during a single conversation turn.


## How it connects

This concept is the foundational constraint for every large language model, which are computer programs designed to understand and generate human-like text. It directly relates to tokenization, the process of breaking text down into the tiny chunks (tokens) counted by the window, and to prompt engineering, the practice of crafting inputs that fit within these limits to get the best results.


## The mathematics

The context window is a hard limit on how much information a large language model can hold in its active memory at one time. Think of it as a bucket with a fixed capacity. The mathematics involved here is simple arithmetic: counting, adding, and comparing numbers. There is no algebra, no calculus, and no complex formulas. You only need to understand three things: what a token is, how to add them up, and how to compare the total to the limit.

First, we must define the unit of measurement. The model does not read words; it reads tokens. A token is a piece of text, which can be a whole word (like 'cat') or a fragment of a word (like 'ing' in 'running'). For English text, a useful rule of thumb is that one word equals approximately 1.3 tokens. This means if you have a paragraph with 100 words, it contains roughly 130 tokens.

Next, we define the variables for our calculation:
- Let W be the context window size. This is the maximum number of tokens the model can process in one go. It is a fixed number determined by the model's design (for example, 4,000 or 8,000).
- Let I be the number of tokens in your input. This includes every token in your prompt, whether it is your question, instructions, or pasted text.
- Let O be the number of tokens the model will generate as output. This is the length of the answer you are asking for.

The fundamental rule of arithmetic for this system is:
I + O must be less than or equal to W.

This inequality tells us two things. First, if your input I is already larger than W (I > W), the model cannot even begin to process the request because the bucket overflows before any calculation starts. Second, if the sum of your input and your desired output exceeds W (I + O > W), the model will run out of space. In many implementations, this results in truncation, where the model drops tokens from the middle or end of your input to make room for the output, often destroying the context needed for a correct answer.

There is also a cost component tied to this arithmetic. Most providers charge per token. Therefore, the total cost of a request is proportional to I + O. If you double the number of tokens in your prompt (I), you roughly double the cost of that single call.


## A worked example

Let us walk through a specific case with real numbers to see how the arithmetic works.

Scenario:
- The model has a context window W = 4,000 tokens.
- You want to send a prompt consisting of two parts:
  - Part A: 2,500 tokens of instructions.
  - Part B: 1,000 tokens of document text.
- You want the model to generate an output of O = 500 tokens.

Step 1: Calculate the total input tokens (I).
You must add all parts of your input together.
I = 2,500 + 1,000
I = 3,500 tokens.

Step 2: Calculate the total required space.
The model needs space for both the input it reads and the output it writes. We add them together.
Total Required = I + O
Total Required = 3,500 + 500
Total Required = 4,000 tokens.

Step 3: Compare the total required space to the window size (W).
We check if the Total Required is less than or equal to W.
4,000 ≤ 4,000.
The condition holds. The request fits exactly within the context window. The model can process this without truncation.

Step 4: Consider a change in input size.
Suppose you increase the document text (Part B) from 1,000 tokens to 1,500 tokens. The instructions (Part A) remain 2,500 tokens.
New I = 2,500 + 1,500
New I = 4,000 tokens.

Step 5: Recalculate the total required space with the new input.
Total Required = New I + O
Total Required = 4,000 + 500
Total Required = 4,500 tokens.

Step 6: Compare to the window size (W).
We check if 4,500 ≤ 4,000.
This is false. 4,500 > 4,000.
The condition fails. The total required space exceeds the context window by 500 tokens. In practice, the model will likely truncate part of your input (often the middle or end) to fit within the 4,000-token limit, which may cause it to lose important information from the document.


## From scratch, in Python

```python
# We'll count tokens and check whether they fit in a context window.
# This is pure Python — no libraries needed.

# A simple approximation: each word is about 1.3 tokens.
# We'll write a function that counts tokens in a string.

def count_tokens(text):
    # Split the text into words (by spaces) and count them.
    words = text.split()  # e.g. "hello world" -> ['hello', 'world']
    # Roughly 1.3 tokens per word, rounded to nearest integer.
    approx_tokens = round(len(words) * 1.3)
    return approx_tokens

# Example:
my_prompt = "Explain the theory of relativity in simple terms."
print(count_tokens(my_prompt))  # Should print 10 or 11 (7 words * 1.3 ≈ 9.1, but we round)

# Now a function to check if a prompt and a desired output fit in a window.

def fits_in_context_window(prompt, max_output_tokens, window_size):
    input_tokens = count_tokens(prompt)
    total_needed = input_tokens + max_output_tokens
    if total_needed <= window_size:
        return True, total_needed
    else:
        return False, total_needed

# Example:
window = 4000  # a model with a 4k context window
my_prompt = "Here is a long document..." * 100  # very long prompt
fits, total = fits_in_context_window(my_prompt, 500, window)
print(f"Fits? {fits}, total tokens needed: {total}")

# But our count is approximate. Real tokenizers are more complex.
# Here's a more realistic approach: we simulate a tokenizer that splits into subwords.
# We'll define a very simple tokenizer that splits on punctuation and spaces,
# and treats each piece as a token.

def simple_tokenize(text):
    # Replace punctuation with spaces, then split.
    import re
    tokens = re.findall(r"\w+", text.lower())  # e.g. "Hello, world!" -> ['hello', 'world']
    return tokens

# Example:
print(simple_tokenize("Hello, world!"))  # ['hello', 'world']

# But real tokenizers (like BPE) merge common pairs. Our simple one is fine for counting.

# The key concept: the model can only attend to a fixed number of tokens.
# We check if our input fits, and if not, we truncate the middle (as many systems do).

def truncate_to_fit(text, max_tokens):
    tokens = simple_tokenize(text)
    if len(tokens) <= max_tokens:
        return ' '.join(tokens)  # no truncation needed
    else:
        # Keep the first and last 25% of tokens, drop the middle.
        keep_front = int(max_tokens * 0.25)
        keep_back = int(max_tokens * 0.25)
        front = tokens[:keep_front]
        back = tokens[-keep_back:]
        return ' '.join(front + ['...'] + back)

# Example:
long_text = "The quick brown fox jumps over the lazy dog. " * 100
shortened = truncate_to_fit(long_text, 50)
print(shortened)

```


## The same thing, with a framework

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained('gpt2')
tokens = tokenizer.encode('Hello, world!')
print(len(tokens))  # The real token count

# Check if input fits in a context window
prompt = "Your long prompt here"
max_output = 500
window_size = 4096

if len(tokenizer.encode(prompt)) + max_output <= window_size:
    print("Fits")
else:
    print("Does not fit — need to truncate or reduce output")
```


## What the code does

The pure-Python version starts by defining a function that counts tokens approximately: it splits the text on spaces and multiplies the word count by 1.3. This is a rough approximation because real tokenizers are subword-based. The `fits_in_context_window` function takes a prompt, a maximum output token budget, and the window size, then adds the two token counts together and compares with the window size. This demonstrates the core arithmetic. The `simple_tokenize` function shows a more realistic tokenization: it splits on non-word characters and lowercases — that is closer to what a real tokenizer does, but still oversimplified. The `truncate_to_fit` function models what happens when input exceeds the window: it keeps some tokens from the beginning and end, and drops the middle (a common strategy in many systems). In the Hugging Face version, `AutoTokenizer.from_pretrained('gpt2')` loads a real tokenizer, and `encode` gives you the actual token IDs. You then do the same arithmetic: `len(input_ids) + max_output <= window_size`. This is exactly what a developer does when building an application — they compute whether their prompt fits, and if not, they either truncate or use a different strategy like summarisation or retrieval.


## Common misconceptions

**A bigger context window means the model remembers everything better** — If I have a 128k window, I can feed it an entire book and it will use every detail perfectly.

The model attends to all tokens, but its attention is not equal across the sequence. Empirical studies (like the 'lost in the middle' paper) show that models are much better at using the beginning and end of a long context than the middle. Also, a larger window slows down inference and costs more, so you rarely use the full capacity in practice. In fact: A larger window sets a higher ceiling, but it does not guarantee good performance across the entire span. You may still need to put key information at the start or end, or use retrieval to pull out the most relevant parts.

**The model forgets information when it falls outside the window** — The model must have a memory somewhere — it answered my earlier question correctly, so it should remember that when I ask a follow-up.

The model is stateless. It has no memory beyond the tokens currently in the context. If your follow-up prompt does not include the earlier conversation, it is as if that conversation never happened. In fact: To continue a conversation, you must include the previous dialogue in the prompt (that is what chat interfaces do). The context window is the only thing that carries the conversation forward.

**One token equals one word** — If I paste in 100 words, that is 100 tokens. I can fit 4,000 words into a 4,000-token context.

Tokens are subword units. A word like 'unbelievable' might become 'un', 'believable' — two tokens. A word like 'hi' might be one token, but a rare word might be three or four. On average, English text uses about 1.3 tokens per word, so 4,000 tokens are only about 3,000 words. In fact: Always count tokens with the model's actual tokenizer, not by word count. As a rough rule of thumb, multiply the number of words by 1.3 to get an approximate token count.

**The context window is a fixed resource that is fully used every call** — I can send any prompt, and the model will just use whatever it needs.

The window is a maximum, not a target. If you send a short prompt, the model fills only part of the window. But if you send too many tokens, you get an error or the input is truncated — your prompt is cut off without warning. In fact: You must design your prompts to fit within the window, leaving room for the model's output. Input plus output must be less than or equal to the window size.

**The context window is permanent across multiple calls** — I asked about the weather in Delhi yesterday, so today I can ask about the rain in Mumbai and the model will remember the weather context.

Each API call is independent. Unless you pass the entire conversation history, the model has no memory of previous calls. Chat applications achieve continuity by sending the whole history every time, but that history counts against the window. In fact: In your own applications, you must decide what to pass into the window. The window is a budget you allocate each call, not a persistent state.


## In short

The context window is the model's whole working memory for a single call: a fixed number of tokens that must hold everything you send and everything the model generates. Nothing outside it exists to the model. It is a budget — for cost, latency, and performance. A bigger window is not automatically better; you still need to place important information where the model will attend to it, and you must count tokens carefully because real token counts differ from word counts. The context window shapes every LLM application, from simple chat to RAG and agents, and mastering it is the first step to using AI systems well.


## Where this is used

The context window governs almost everything you do with a production LLM. In **large language models**, the window size is a headline specification: GPT-4-class models have 32k or 128k windows, while some newer models support a million tokens. But a huge window is not a panacea. When you have a 128k window and you paste in 100k tokens of legal documents, the model's attention spreads thin, and the middle sections are often ignored — a failure mode called 'lost in the middle'. Developers compensate by moving important content to the beginning or end of the prompt, or by using retrieval. In **small language models** that run on a phone or a laptop, the context window is much smaller — often 2k to 8k tokens — because memory and compute are scarce. A phone model cannot attend to a 100-page novel; it can only handle a few paragraphs. This forces architects to use smaller prompts, or to use external memory like a vector database. In **retrieval-augmented generation (RAG)**, the context window is the reason RAG exists. Instead of stuffing an entire book into a prompt, you first retrieve the most relevant passages (using embeddings and similarity search) and fit only those into the window. The choice of chunk size — how many tokens you insert per retrieved passage — is a direct tradeoff: bigger chunks use more window budget, and if you retrieve too many chunks, you exceed the window. In **agentic systems** — AI that plan, call tools, and act in loops — the context window is the system's working memory for the entire task. An agent that calls a web search, reads the result, calls a calculator, then reflects, must keep all that intermediate text within the window. If the window fills up, the agent must summarise previous steps or forget them — and that forgetting is a common cause of agents losing track of their goal. In **everyday applied ML** — recommendation, vision, forecasting, fraud detection, speech — the context window concept appears wherever a model processes sequences: a speech model has a window that limits how much audio it can transcribe at once; a forecasting model has a window that limits how much history it can see. In practice, a developer building an LLM app writes code that counts tokens, checks them against the window, and truncates or retrieves to fit. They might use a library like LangChain's `token_counter` or the model's own tokenizer. They also watch costs: every token in a prompt costs money, and a large window tempts you to paste everything, but that multiplies the bill. When done badly, the app silently truncates the user's input, producing answers that ignore crucial details — and the developer may not even realise the model never saw that part of the text.


---

Read this on the web: https://basavapurushottam.com/learn-ai/context-window
