# Positional Encoding

Injecting order information into attention, which is otherwise permutation-invariant, through sinusoidal patterns, learned vectors or rotations.

**Rests on:** Multi-Head Attention

**Leads to:** Cross-Attention, GPT Architecture, RoPE Rotary Embeddings, Vision Encoders

*Deep Learning · Transformers*


## Why learn this

You are studying Transformers, the architecture behind ChatGPT and other modern large language models. These systems use a mechanism called self-attention to understand text. This mechanism allows every word in a sentence to look at every other word to understand context. However, this method treats words as an unordered collection, like a pile of scattered letters where 'cat' and 'act' are indistinguishable. Without a way to track order, the model cannot distinguish between 'dog bites man' and 'man bites dog'. Positional encoding is the solution that restores this sense of order. It is the foundational step that makes it possible for these models to read, write, and understand language structure.


## The intuition

Imagine a meeting where everyone is speaking at once. Each person can hear every other voice, but without knowing who spoke first, second, or last, the conversation is just noise. In a Transformer, the self-attention mechanism lets each word 'hear' every other word to gather context, but it has no natural sense of time or sequence. If the model sees the words 'river', 'bank', and 'loan', it knows their meanings individually but does not know that 'bank' follows 'river' and precedes 'loan'.

Positional encoding solves this by tagging every word with a specific marker of its position in the sequence as it enters the system. Think of it like giving each participant at the meeting a badge that displays their turn number. The model then uses two pieces of information for each word: what the word means (from the previous learning stage) and where it sits (from the positional encoding). By combining these, the model can distinguish that the sequence 'river bank' has a different meaning than 'bank river'.

For example, if you have the sentence 'The cat sat on the mat', the first word 'The' might be tagged with position 1, the second word 'cat' with position 2, and so on. The encoding ensures that the model knows 'cat' is the subject acting in the first half of the sentence, not the object at the end.


## An analogy

Imagine you are arranging a photograph album. The photos themselves represent the content of the words, but the order in which they appear tells the story. If you scatter the photos randomly on the floor, the narrative is lost. A positional encoding is like numbering each photo on the back before you put them in the album. These numbers tell you exactly where each photo belongs.

However, simple integer numbers (1, 2, 3) have a limitation: they do not naturally convey how close two items are relative to the total size of the set. For instance, the distance between photo 5 and photo 6 is physically one step, but if the album has 100 photos, that step is a small fraction (1/94) of the remaining space. If the album only has 10 photos, that same step is a much larger proportion of the journey. To handle this better, advanced systems use values that shift smoothly in a wave-like pattern instead of simple integers. These shifting patterns create structures where positions close together have similar values, and positions far apart have different ones. This allows the model to easily calculate relative distances—for example, knowing that position 5 is very close to position 6, regardless of whether the total sequence length is 10 or 100.

This analogy has one limitation: in a Transformer, the positional information is not just written on the back and kept separate; it is mathematically added to the word's meaning list, so the two are fused together from the start.


## Definition

A positional encoding is a set of numbers added to each word's representation (embedding) to indicate its specific location in the sequence, allowing the model to distinguish order despite processing all words simultaneously.


## How it connects

This concept builds directly on your understanding of embeddings, which are lists of numbers that capture a word's meaning. While embeddings provide the 'what', positional encodings provide the 'where'. It is also closely tied to self-attention, the mechanism where each word evaluates its relationship with every other word; without positional encodings, self-attention would be blind to sequence structure.


## The mathematics

Positional encoding adds information about where each word sits in a sentence to the word’s own meaning vector. Standard neural network layers treat all inputs as a set, ignoring order. To fix this, we attach a unique pattern of numbers to each position index.

The pattern uses sine and cosine waves. A sine wave starts at 0, rises to 1, falls back through 0 to -1, and returns to 0. A cosine wave starts at 1, falls to 0, drops to -1, rises back through 0, and ends at 1. By using waves of different speeds (frequencies), we create a unique signature for every position.

We define two formulas. One uses the sine function, one uses the cosine function. Both take the same input: the position index and the dimension index.

Let pos be the position in the sequence, starting at 0 for the first word.
Let i be the index of the specific number within the encoding vector, starting at 0.
Let d_model be the total width of the embedding vector (the total number of dimensions).

The formula for even-indexed dimensions (where i is 0, 2, 4...) uses sine:
PE(pos, 2i) = sin(pos / 10000^(2i / d_model))

The formula for odd-indexed dimensions (where i is 1, 3, 5...) uses cosine:
PE(pos, 2i+1) = cos(pos / 10000^(2i / d_model))

Note that the exponent in the denominator is always 2i divided by d_model. The base of the exponent is 10,000.

The number 10,000 stretches the wavelengths. For small i, the wavelength is short. For large i, the wavelength is very long. This allows the model to see both fine-grained local differences and coarse global positions.

Once we calculate the encoding vector for a position, we add it element-by-element to the word embedding vector for that position. The model then processes this combined vector.


## A worked example

We will calculate the positional encoding for the first four positions (0, 1, 2, 3) in a sequence. We assume a very small embedding width of d_model = 4. This means our encoding vectors will have 4 numbers.

The dimensions are indexed i = 0, 1, 2, 3.

First, we determine the denominator for each dimension pair. The formula uses 10000 raised to the power of (2i / d_model).

For i = 0 (the first number in the vector):
The exponent is (2 * 0) / 4 = 0.
The denominator base is 10000^0 = 1.
So the input to the sine/cosine function is pos / 1 = pos.

For i = 1 (the second number in the vector):
The exponent is (2 * 1) / 4 = 0.5.
The denominator base is 10000^0.5, which is the square root of 10,000 = 100.
So the input to the sine/cosine function is pos / 100.

For i = 2 (the third number in the vector):
The exponent is (2 * 2) / 4 = 1.
The denominator base is 10000^1 = 10,000.
So the input to the sine/cosine function is pos / 10,000.

For i = 3 (the fourth number in the vector):
The exponent is (2 * 3) / 4 = 1.5.
The denominator base is 10000^1.5 = 10000 * sqrt(10000) = 10,000 * 100 = 1,000,000.
So the input to the sine/cosine function is pos / 1,000,000.

Now we calculate the values for each position. We use radians for standard trigonometric calculations.

Position 0:
Dimension 0 (even, i=0): sin(0 / 1) = sin(0) = 0.
Dimension 1 (odd, i=1): cos(0 / 100) = cos(0) = 1.
Dimension 2 (even, i=2): sin(0 / 10000) = sin(0) = 0.
Dimension 3 (odd, i=3): cos(0 / 1000000) = cos(0) = 1.
Encoding for position 0 is [0, 1, 0, 1].

Position 1:
Dimension 0 (even, i=0): sin(1 / 1) = sin(1 radian). 1 radian is approximately 57.3 degrees. sin(1) ≈ 0.8415.
Dimension 1 (odd, i=1): cos(1 / 100) = cos(0.01 radians). cos(0.01) ≈ 0.99995.
Dimension 2 (even, i=2): sin(1 / 10000) = sin(0.0001 radians). This is very close to 0. sin(0.0001) ≈ 0.0001.
Dimension 3 (odd, i=3): cos(1 / 1000000) = cos(0.000001 radians). This is extremely close to 1. cos(0.000001) ≈ 1.0.
Encoding for position 1 is [0.8415, 0.99995, 0.0001, 1.0].

Position 2:
Dimension 0 (even, i=0): sin(2 / 1) = sin(2 radians). 2 radians is approximately 114.6 degrees. sin(2) ≈ 0.9093.
Dimension 1 (odd, i=1): cos(2 / 100) = cos(0.02 radians). cos(0.02) ≈ 0.9998.
Dimension 2 (even, i=2): sin(2 / 10000) = sin(0.0002 radians). sin(0.0002) ≈ 0.0002.
Dimension 3 (odd, i=3): cos(2 / 1000000) = cos(0.000002 radians). cos(0.000002) ≈ 1.0.
Encoding for position 2 is [0.9093, 0.9998, 0.0002, 1.0].

Position 3:
Dimension 0 (even, i=0): sin(3 / 1) = sin(3 radians). 3 radians is approximately 171.9 degrees. sin(3) ≈ 0.1411.
Dimension 1 (odd, i=1): cos(3 / 100) = cos(0.03 radians). cos(0.03) ≈ 0.99955.
Dimension 2 (even, i=2): sin(3 / 10000) = sin(0.0003 radians). sin(0.0003) ≈ 0.0003.
Dimension 3 (odd, i=3): cos(3 / 1000000) = cos(0.000003 radians). cos(0.000003) ≈ 1.0.
Encoding for position 3 is [0.1411, 0.99955, 0.0003, 1.0].

Result: The encodings are:
Position 0 → [0, 1, 0, 1]
Position 1 → [0.8415, 0.99995, 0.0001, 1.0]
Position 2 → [0.9093, 0.9998, 0.0002, 1.0]
Position 3 → [0.1411, 0.99955, 0.0003, 1.0]

Notice that the first two dimensions change significantly between positions because their wavelengths are short (1 and 100). The last two dimensions change very little because their wavelengths are long (10,000 and 1,000,000). This provides a multi-scale view of position.


## From scratch, in Python

```python
import math

# d_model is the size of the embedding. We'll use 4 for simplicity.
# But we only generate for positions 0 to 3, showing the pattern.

def positional_encoding(seq_len, d_model):
    # seq_len: how many positions (words) in the sequence
    # d_model: how many numbers in each position's encoding

    # Initialize a matrix (list of lists) with zeros
    pe = [[0.0] * d_model for _ in range(seq_len)]

    for pos in range(seq_len):
        for i in range(0, d_model, 2):  # even indices
            # Compute the sine value for this dimension
            angle = pos / (10000 ** (i / d_model))
            pe[pos][i] = math.sin(angle)
            # If there is an odd counterpart
            if i + 1 < d_model:
                pe[pos][i + 1] = math.cos(angle)
    return pe

# Test with a small example
seq_len = 4
d_model = 4
encodings = positional_encoding(seq_len, d_model)
for pos, vec in enumerate(encodings):
    print(f"Position {pos}: {vec}")
```


## The same thing, with a framework

```python
In PyTorch, you can use the built-in sinusoidal positional encoding from the nn.Transformer module or define your own. A common snippet:

import torch
import math

def positional_encoding_torch(seq_len, d_model):
    pe = torch.zeros(seq_len, d_model)
    position = torch.arange(0, seq_len, dtype=torch.float).unsqueeze(1)
    div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
    pe[:, 0::2] = torch.sin(position * div_term)
    pe[:, 1::2] = torch.cos(position * div_term)
    return pe

# This produces a tensor of shape (seq_len, d_model) that you can add to your embeddings.
```


## What the code does

The from-scratch code defines a function called positional_encoding. First, we create a matrix (a list of lists) called pe, with seq_len rows and d_model columns, all initialized to zero. Then we loop over each position (0 to seq_len-1). For each position, we loop over dimensions in steps of 2 (even indices). For each even index i, we compute the angle using the formula: angle = pos / (10000 ** (i / d_model)). We then set pe[pos][i] to the sine of that angle, and if there's a next dimension (i+1), we set it to the cosine. This fills the matrix. Finally, we return the matrix. The framework code does the same thing but using PyTorch tensors and vectorised operations: it creates a tensor of positions, computes the divisor term using exponentials, then computes sin and cos for even and odd columns respectively. Both produce the same encodings.


## Common misconceptions

**Positional encoding is optional** — I guess you can just skip adding positional encodings if the model is small enough?

Without any sense of order, self-attention treats the sentence as a bag of words, so a model without positional encodings cannot understand syntax at all—even for a two-word sentence. In fact: Positional encoding is absolutely essential in Transformers. Every practical Transformer includes it, because attention alone is permutation-invariant.

**The encoding is the position number itself** — The positional encoding is just the index (0, 1, 2...), so why not add that integer to the embedding?

A single integer lacks the capacity to represent fine-grained relative positions across long sequences and can cause huge magnitude differences, disrupting learning. In fact: Positional encodings are vectors of sine/cosine values (or learned high-dimensional vectors) that capture both absolute and relative position information smoothly.

**All positional encodings are sinusoidal** — Since the original Transformer paper used sines and cosines, that's the only way.

Many modern models use learned positional embeddings (e.g., BERT) or rotary embeddings (RoPE). In fact: Sinusoidal encodings are the original design, but learned embeddings and RoPE are now widely used, especially for long-context efficiency.

**The encoding is multiplied, not added** — I think you multiply the positional encoding with the embedding to combine them.

Adding preserves the original information while injecting position; multiplication would scale the word meaning and distort it. In fact: Positional encodings are added to the word embedding (element-wise). This keeps the content intact and adds a location signature.


## In short

Positional encoding is the trick that lets Transformers understand order. It assigns a vector (often sinusoidal) to each position in the sequence, added to the word's embedding, so self-attention can use both meaning and location. Without it, attention would treat every sentence as an unordered set, and language understanding would collapse.


## Where this is used

Positional encodings are used everywhere Transformers appear. In large language models like GPT-4, positional encodings (often learned or RoPE) are added to token embeddings before the self-attention layers. For small language models on devices, simpler positional encodings like relative or sinusoidal are used to save compute, but they still need them to understand word order. In retrieval-augmented generation (RAG), when documents are fetched and fed to the model, positional encodings help the model understand the order of sentences in the retrieved context. In agentic systems that plan and call tools, the model processes sequences of actions and observations, and positional encodings keep the events in temporal order. In everyday applied ML, any Transformer-based model—for recommendation, vision, forecasting, fraud detection—uses positional encodings. A developer writing a Transformer from scratch in PyTorch would add a line like `x = x + positional_encoding(seq_len, d_model)` where `x` is the embedding tensor. Getting this wrong—like forgetting to add positional encodings—results in a model that can't distinguish between 'dog bites man' and 'man bites dog', leading to nonsense outputs. In practice, choosing the right positional encoding (e.g., RoPE for long-context) can extend a model's ability to handle much longer inputs, which is why re-engineering positional encodings is a hot research area.


---

Read this on the web: https://basavapurushottam.com/learn-ai/positional-encoding
