# Sliding Window Sampling

Generating training examples by moving a fixed-size window across a corpus, with stride controlling overlap between consecutive samples.

**Rests on:** Temperature and Sampling

*LLMs and Generative AI · Large Language Models*


## Why learn this

This is how every large language model — ChatGPT, Gemini, Llama — is taught to read and write. When a model predicts the next word in a sentence, it learned that skill by training on millions of sliding windows. Understanding this concept unlocks: Transformer training → Context Window (how much the model remembers) → Temperature and Sampling (how it chooses words) → and eventually fine-tuning models for your own tasks. If you can build a sliding window, you can build your own tiny language model and watch it learn to speak.


## The intuition

Imagine you are reading a book to learn a story. You don't read the whole book at once; you read one paragraph at a time, moving your eyes forward. Each paragraph gives you a little bit of story, and you connect it to the next. A sliding window is the same idea for a computer: you give it a short chunk of text (say, three words) and ask it to guess the next word. Then you shift the chunk one word forward and repeat. The 'window' is the chunk of text the model sees at any moment. 'Sliding' means you move it along the text, like a magnifying glass you drag across a page. The 'stride' is how many words you move the window each step. If you move it one word at a time, you get a huge number of overlapping chunks — that's a lot of teaching examples. If you move it three words at a time, you get fewer chunks, but they don't overlap as much. The window size is how many words the model sees at once — that defines how much context it has to predict the next word. A window of three words is a small scope; a window of 1024 tokens is a big one (tokens are like words but not exactly — you'll learn that later). This process is the core of how a model learns language: it sees the pattern over and over, and adjusts its internal numbers (weights) to get better at prediction.


## An analogy

You are a chef learning to cook by watching a master chef. The master chef cooks a long dish — say, a 20-minute recipe. You can't watch the whole thing at once and remember every detail. Instead, you watch a short clip (like 5 seconds) of the chef's hands, then you try to guess what they do next. Then you move the clip forward a second and watch the next 5 seconds, and so on. The 'clip' is the sliding window, the '5 seconds' is the window size, and the '1 second step' is the stride. If you use a small stride (1 second), you get many overlapping clips — you see the same action many times, which helps you learn it well. If you use a large stride (5 seconds), you get fewer clips, but you cover more of the recipe per step. The window size determines how much context you see — a 5-second clip shows you only the immediate motion, but a 20-second clip shows you the whole technique. The chef analogy breaks down when you realize the model doesn't just watch — it predicts the very next second. Also, the model doesn't 'remember' past clips; each clip is independent. And the model doesn't improve by watching — it improves by adjusting its internal numbers based on whether its guess was right or wrong. So the window is not a memory; it's just the current view.


## Definition

Sliding window sampling is the technique of generating many small, overlapping training examples from a long text by moving a fixed-size window across the corpus, with a given stride (step size) controlling how much the windows overlap.


## How it connects

You have not studied any other topics yet. This is your first building block. Everything else in large language models — the context window, tokenization, attention — builds on the idea that the model is trained on these small, fixed-length snippets. Context Window is the concept that, at inference (when you ask the model something), the window is the maximum number of tokens the model can see at once. Sliding window sampling is the training-time version: you create many windows from a long text to teach the model. Later, when you learn about tokenization (how text is split into tokens), you'll see that the sliding window operates on token sequences, not raw characters. This foundational step is what makes next-token prediction possible.


## The mathematics

This section explains how to turn one long list of items into many shorter, overlapping lists. This is often called a 'sliding window'. You need three numbers:

1. The total length of the original list. Let us call this number L (for Length).
2. How many items you want in each short list. Let us call this number W (for Window size).
3. How many positions you move forward to start the next short list. Let us call this number S (for Stride).

The process works like this:

First, check if the window is too big for the list. If the window size W is larger than the total length L, you cannot make even one window. In that case, the number of windows is zero.

If the window fits (W is less than or equal to L), you start at the very beginning of the list. This is position 1. You take the next W items. That is your first window.

To find the start of the second window, you add the stride S to the previous start position. So the second window starts at position 1 + S. You take the next W items from there.

You keep doing this. To find the start of the next window, you add S to the current start position. You stop when the start position is so far along that there are not enough items left to fill a full window of size W.

Specifically, if the start position is k, you need items from k up to k + W - 1. If k + W - 1 is greater than L, you cannot make that window. You stop before that.

To calculate exactly how many windows you get without counting them one by one, use this formula:

Number of windows = floor((L - W) divided by S) plus 1.

Here is what the parts mean:
- (L - W): This subtracts the size of the window from the total length. It tells you how much 'extra' space is available to slide the window around.
- divided by S: This divides that extra space by the stride. It tells you how many full strides fit in that extra space.
- floor(...): This means round down to the nearest whole number. You cannot have a fraction of a step. If the division leaves a remainder, you drop it.
- plus 1: We add one because the very first window starts at position 1, which is not covered by the 'extra space' calculation. The formula counts the *steps* you take after the first window, so we add the first window back in.

Example of the logic:
If L = 10, W = 3, S = 2.
L - W = 7.
7 divided by 2 is 3.5.
Floor(3.5) is 3.
3 plus 1 is 4.
So there are 4 windows.

The windows start at:
1 (items 1,2,3)
1 + 2 = 3 (items 3,4,5)
3 + 2 = 5 (items 5,6,7)
5 + 2 = 7 (items 7,8,9)
The next start would be 9. Items 9 and 10 are only 2 items. We need 3. So we stop. The last valid start was 7.


## A worked example

Let us walk through a specific case with real numbers.

We have a sequence of words:
['The', 'cat', 'sat', 'on', 'the', 'mat']

Step 1: Identify the parameters.
- The total length L is 6. There are 6 words in the list.
- We choose a window size W of 3. Each short list will have 3 words.
- We choose a stride S of 1. We will move forward by 1 word each time.

Step 2: Check if the window fits.
Is W (3) less than or equal to L (6)? Yes. So we can make windows.

Step 3: Calculate the number of windows using the formula.
Formula: floor((L - W) / S) + 1
Substitute the numbers: floor((6 - 3) / 1) + 1
Calculate inside the parentheses: 6 - 3 = 3.
Divide by stride: 3 / 1 = 3.
Round down (floor): The number is already a whole number, so it stays 3.
Add one: 3 + 1 = 4.
So, we expect 4 windows.

Step 4: Verify by listing the windows.
Window 1 starts at position 1.
It takes items from position 1 to 1 + 3 - 1 = 3.
Items at positions 1, 2, 3 are: 'The', 'cat', 'sat'.

Window 2 starts at position 1 + 1 = 2.
It takes items from position 2 to 2 + 3 - 1 = 4.
Items at positions 2, 3, 4 are: 'cat', 'sat', 'on'.

Window 3 starts at position 2 + 1 = 3.
It takes items from position 3 to 3 + 3 - 1 = 5.
Items at positions 3, 4, 5 are: 'sat', 'on', 'the'.

Window 4 starts at position 3 + 1 = 4.
It takes items from position 4 to 4 + 3 - 1 = 6.
Items at positions 4, 5, 6 are: 'on', 'the', 'mat'.

Next potential start would be 4 + 1 = 5.
Items from 5 to 5 + 3 - 1 = 7.
But position 7 does not exist (L is 6). So we stop.

We have created exactly 4 windows, which matches our formula calculation.

Now let us try a different stride. Keep L = 6 and W = 3, but change S to 2.

Step 1: Calculate the number of windows.
Formula: floor((6 - 3) / 2) + 1
Substitute: floor(3 / 2) + 1
Divide: 3 / 2 = 1.5.
Round down (floor): 1.
Add one: 1 + 1 = 2.
So, we expect 2 windows.

Step 2: Verify by listing.
Window 1 starts at position 1.
Items 1 to 3: 'The', 'cat', 'sat'.

Next start is 1 + 2 = 3.
Window 2 starts at position 3.
Items 3 to 5: 'sat', 'on', 'the'.

Next start would be 3 + 2 = 5.
Items 5 to 7. Position 7 is out of bounds. Stop.

We have 2 windows. The formula worked.

Notice the difference:
With stride 1, we got 4 windows. They overlap heavily. Window 1 and 2 share two words. Window 2 and 3 share two words.
With stride 2, we got 2 windows. They overlap less. Window 1 and 2 share only one word ('sat').
A smaller stride (like 1) creates more training examples but they are very similar. A larger stride (like 2) creates fewer examples that are more distinct.


## From scratch, in Python

```python
# We'll write a simple function that takes a list of items (words)
# and returns a list of windows.

# First, let's understand lists in Python.
# A list is an ordered collection that can be changed (we can add, remove, or change items).
# Example: my_list = [10, 20, 30]  (a list of three numbers)
# You can access the first item with my_list[0] (which gives 10), the second with my_list[1], etc.

# Here's a text, as a list of words:
text = ['The', 'cat', 'sat', 'on', 'the', 'mat']

# We'll define a function that takes a list, a window size, and a stride.
# A function is a reusable block of code that takes inputs (arguments) and returns a result.
# Functions are defined with 'def', followed by a name and parentheses for the inputs.

def sliding_window(data, window_size, stride):
    windows = []  # we'll collect the windows here (an empty list)
    # The range function generates a sequence of numbers: range(start, stop, step)
    # Here, start = 0, stop = len(data) - window_size + 1, step = stride
    # For each start position, we take a slice of the list from start to start+window_size
    for start in range(0, len(data) - window_size + 1, stride):
        window = data[start:start+window_size]  # slicing: take items from start to start+window_size (not including the end)
        windows.append(window)  # add this window to our list
    return windows

# Let's call the function:
result = sliding_window(text, 3, 1)
print(result)
# Output: [['The', 'cat', 'sat'], ['cat', 'sat', 'on'], ['sat', 'on', 'the'], ['on', 'the', 'mat']]

# If we use stride=2:
result2 = sliding_window(text, 3, 2)
print(result2)
# Output: [['The', 'cat', 'sat'], ['sat', 'on', 'the']]

# Notice: the second window starts at position 2 (0-based), taking 'sat','on','the'. The next start would be 4, but 4+3=7 is beyond the end (6), so it stops.
```


## The same thing, with a framework

In PyTorch (a popular machine learning library), you can use the built-in Dataset and DataLoader, but for sliding windows, you often write a custom Dataset. Here's an example:

```python
import torch
from torch.utils.data import Dataset

class SlidingWindowDataset(Dataset):
    def __init__(self, data, window_size, stride):
        self.windows = []
        for start in range(0, len(data) - window_size + 1, stride):
            window = data[start:start+window_size]
            self.windows.append(window)
    
    def __len__(self):
        return len(self.windows)
    
    def __getitem__(self, idx):
        window = self.windows[idx]
        inputs = window[:-1]  # all tokens except the last
        target = window[-1]   # the last token is the target to predict
        return torch.tensor(inputs), torch.tensor(target)
```

Under the hood, this does the same as the plain Python: it loops over the data, takes slices, and stores them. The Dataset class just provides a standard interface so PyTorch can efficiently load examples in batches during training.


## What the code does

The plain Python code defines a function called `sliding_window`. It takes three inputs: `data` (which is a list of items, like words), `window_size` (how many items each window should have), and `stride` (how many positions to move forward each time). Inside, we create an empty list called `windows` to collect our results. The `for` loop uses `range` to generate start positions: we start at 0, go up to (but not including) the last possible start, and step by `stride`. For each start, we slice the data list from `start` to `start + window_size` — slicing in Python means we get a new list containing those items. We then append that slice to our `windows` list. After the loop, we return the list of windows. The PyTorch version wraps this in a Dataset class, which is a standard way to feed data into a model. It precomputes the windows in the __init__ method, then defines __len__ (how many examples there are) and __getitem__ (how to get a specific example). The inputs are the first `window_size - 1` tokens, and the target is the last token — that's how we teach the model to predict the next word. The framework version hides the loop, but it does exactly the same slicing and storing.


## Common misconceptions

**Sliding window and context window are the same thing** — Sliding window at training time is exactly the context window at inference time.

The context window is the maximum number of tokens the model can see when generating output; it is fixed by the model architecture (e.g., 2048). The sliding window is a data preparation technique that creates many small windows from a long text for training. They serve different purposes: the former is a limit on the model's memory, the latter is a way to create many examples. In fact: Sliding window is a training-time technique to create examples; the context window is an architectural limit at inference time. They are related (both involve window size), but they are distinct concepts.

**Stride must equal 1** — The only correct way to do sliding window is to move one token at a time.

Stride is a hyperparameter. You can choose any stride. A larger stride reduces overlap and dataset size, which can speed up training but may miss some patterns. A smaller stride increases overlap and dataset size, which can improve learning but takes more time. In fact: Stride is a choice you make. It trades off dataset size, redundancy, and training time. There is no one correct value; it depends on your goals.

**Window size is arbitrary and doesn't matter** — The size of the window doesn't affect the model's learning; any size works.

Window size defines how much context the model sees for predicting the next token. If the window is too small, the model can't learn long-range dependencies (e.g., in a sentence like 'The dog that chased the cat finally caught it', the word 'dog' is far from 'it'). If the window is too large, the model may have too much information and training becomes harder. The window size at training bounds the positions the model can attend to — it's a critical choice. In fact: Window size is a key hyperparameter. It limits the context the model can use. Larger windows allow capturing longer-range dependencies but increase complexity. The optimal size depends on the task and data.

**More windows always means better training** — Using a stride of 1 gives the most training examples, so it's always the best choice.

More examples isn't always better. With a stride of 1, adjacent windows overlap by almost the entire window, so the model sees the same tokens many times. This can lead to overfitting on common patterns and slower training without proportional benefit. Also, it increases dataset size, memory, and training time. In fact: The number of windows is a tradeoff. Too many overlapping examples may be redundant; too few may not capture all patterns. You should choose a stride that balances coverage and efficiency, often equal to the window size to avoid overlap in large corpora.

**Sliding window is only for text** — Sliding window sampling is only used for natural language processing.

Sliding windows are used in many domains: time series forecasting, speech processing, computer vision (sliding window over images for object detection), and any sequential data. The concept is general: any sequence can be sampled into fixed-size windows. In fact: Sliding window is a general technique for any sequential data, not just text. It is used in time series, audio, and even images (where it slides over pixels or patches).


## In short

Sliding window sampling is how you turn a long piece of text (or any sequence) into many short, fixed-length examples for training a language model. You choose a window size (how many tokens to look at) and a stride (how many positions to move each step). The stride controls how much the windows overlap: a small stride gives many overlapping examples, a large stride gives fewer, non-overlapping ones. The window size defines the maximum context the model can use for prediction. In practice, this is implemented with a simple loop that slices the sequence at each step, and it's used across LLM pretraining, RAG chunking, time-series forecasting, and more. The key tradeoffs are between dataset size, redundancy, and training efficiency. If you remember only one thing: sliding window creates many small, overlapping or non-overlapping windows from a long sequence, and the stride is the knob that controls how many examples you get and how much they overlap.


## Where this is used

Sliding window sampling is everywhere in modern AI, but its role varies by context. In large language models (like GPT-4, Llama, Gemini), pretraining involves taking massive text corpora (billions of tokens) and sliding a window of typically 1024 to 4096 tokens across the text, with a stride equal to the window size (so no overlap) to maximize coverage without redundancy. Each window becomes a training example where the model must predict the next token. In practice, a developer uses a data-loading library like HuggingFace's `datasets` or PyTorch's `DataLoader` to create these windows efficiently. If they use a stride smaller than the window size, they get overlapping windows, which increases dataset size but also causes the model to see the same tokens multiple times, which can help it learn patterns but also increase training time. In small language models that run on devices (e.g., mobile keyboards, smart speakers that run locally), the model is smaller (e.g., 1-7 billion parameters) and the training window is shorter (e.g., 512 tokens) because the model can't handle long context. The sliding window is still used to train, but the windows are shorter to match the device's memory constraints. In retrieval-augmented generation (RAG), sliding window sampling is not directly used for training the retriever, but it is used in chunking strategies: when you have a long document, you split it into overlapping chunks (using a sliding window) so that the retriever can find portions even if a relevant passage spans a chunk boundary. A developer might use a library like LangChain, which has a `RecursiveCharacterTextSplitter` that can be configured with a chunk size and overlap (which is like a window size and a stride, but the overlap is expressed as a percentage of the chunk size, so effectively stride = chunk_size - overlap). The goal is to ensure that no relevant information is lost at the edges of chunks. In agentic systems (AI agents that plan, call tools, and act in loops), sliding window sampling appears in the context of managing conversation history: when a conversation becomes too long, the agent must select a window of recent messages to keep as context. A simple approach is to take the last N messages, which is a fixed window with a stride of 1 (each new message replaces the oldest). More sophisticated agents use sliding windows over the entire conversation with a learning-based summarizer to compress older parts. This is not a training-time use but a runtime use of the same concept. In ordinary applied machine learning — such as forecasting time series (sales, traffic), anomaly detection, or signal processing — sliding windows are fundamental. For time series forecasting, you train a model to predict the next value given a window of past values. The window size is the lookback period, and the stride (often 1 for daily data, or the forecast horizon for multi-step forecasts) controls how many samples you generate. A developer using a library like `sktime` or `pandas` will apply a rolling window (e.g., `df.rolling(window=7).mean()`) to create features. In speech recognition, sliding windows over the audio waveform (e.g., 25ms windows with 10ms stride) are used to extract features like MFCCs. In fraud detection, a sliding window of transactions for a user is used to compute features like average amount or frequency. The developer writes code that iterates over the data, slices out the window, and uses it as input to a model. The library they use might hide this, but the principle is the same. When done badly — for example, using a window size that's too small (so the model lacks context) or a stride that's too large (so it misses important patterns between windows) — the model's performance degrades. Also, if you use overlapping windows incorrectly (e.g., using a stride of 1 with a window size of 100 on a dataset of 1 million tokens), you get nearly 1 million examples, which might be more than necessary and increase training time without proportional benefit. Understanding the tradeoff between window size and stride is a key decision when preparing data for any sequence model.


---

Read this on the web: https://basavapurushottam.com/learn-ai/sliding-window-sampling
