# RoPE Rotary Embeddings

Encoding position by rotating query and key vectors by an angle proportional to position, so attention scores depend on relative distance.

**Rests on:** Positional Encoding, Self-Attention from Scratch

*Deep Learning · Transformers*


## Why learn this

This technique explains how modern language models like Llama and Mistral keep track of where words are in a sentence while still being able to understand very long documents. Without this, a model might know which words appear but forget who spoke when or which noun matches which verb because the connection got lost in the middle of the text. Mastering this will let you read technical discussions about AI architecture with clarity instead of confusion. You will understand why researchers moved away from older methods of marking position (like using simple sine waves) toward rotating numerical lists. This knowledge is essential if you ever need to adjust how far a model can 'read' at once, known as its context window, because extending that window requires understanding exactly how position information scales.


## The intuition

Imagine you are at a dinner table with six people. You want each person to be able to talk to everyone else, but you also need to respect that people who are sitting next to each other are more likely to have a natural conversation than people at opposite ends. The classic way to do this is to give each seat a number — seat 1, seat 2, seat 3 — but that only tells you 'this is seat 3', not how far it is from seat 5. RoPE does something smarter: instead of giving each seat a number, it turns each seat into a direction on a clock face. Seat 1 points at 3 o'clock, seat 2 points at 4 o'clock, seat 3 at 5 o'clock, and so on. Now, if you look at two seats, the angle between their hands tells you not just which seat is which, but exactly how far apart they are — one tick apart, two ticks, three. And that relative distance is what the model really needs.

In a language model, each word's meaning is stored as a list of numbers (a vector), which is simply an ordered collection of values. RoPE takes that list, treats those numbers as coordinates in a geometric space, and rotates the entire list by an angle that depends on the word's position in the sentence. When the model compares two words — say, the first word and the tenth word — it calculates a 'dot product' (a specific calculation that multiplies corresponding numbers in two lists and adds them up to produce a single score representing their similarity). Because the lists have been rotated by different amounts based on their positions, this dot product naturally includes a factor that depends on 10 − 1 = 9. The rotation encodes the distance itself, not just the absolute position.


## An analogy

Think of a compass that always points north, but the needle is weighted so that when you walk one step forward, the needle rotates by 10 degrees. Stand at the start: needle at 0°. Walk one step: needle at 10°. Walk two steps: 20°. The needle itself tells you how many steps you have taken — it is a step counter, but a special one: the angle between your current needle and your needle at any previous position is exactly proportional to the number of steps you took between them. Now imagine you are in a field with a friend. You both have these compasses. You want to know if you are standing close together or far apart. You don't compare the numbers on your compasses (that would be absolute position), you look at the angle between your two needles — if the angle is 30°, you know you are exactly 3 steps apart, regardless of where you both started. This is precisely what RoPE does: the query of a word at position i and the key of a word at position j are both rotated, and the dot product (a measure of how well they agree) ends up depending on (i − j). The analogy stops working when you note that the real vectors have many numbers, not just one angle — but the principle is identical: multiply by a rotation matrix that depends on position, and then the dot product sees relative distance. Also, the rotation is not a physical rotation in 3D space; it is a rotation in the high-dimensional space of the vector's coordinates — a mathematical move, not a real one.


## Definition

Rotary Position Embedding (RoPE) is a method that updates the position of each word in a sequence by rotating its numerical representation, ensuring that the comparison between any two words directly reflects how far apart they are in the text. It achieves this by applying a geometric rotation to the query and key vectors before calculating their similarity score.


## How it connects

This concept sits directly under 'Transformers' in your library, alongside 'Positional Encoding' and 'Causal Attention Masking'. You have mastered neither of those yet, but they are the immediate family: RoPE is one of the three main ways to inject order information into attention (the others being sinusoidal encodings and learned embeddings). The key difference from the classic sinusoidal encoding is that RoPE operates on the query and key vectors *after* they are computed, rotating them in a way that makes the dot product depend on relative distance, whereas sinusoidal encodings are added to the input embeddings before attention. Both are trying to solve the same problem — attention alone has no sense of order at all — but RoPE does it in a way that generalises better to longer texts, which is why it has become the default in recent open models.


## The mathematics

RoPE (Rotary Positional Embeddings) uses a branch of mathematics called **trigonometry**. Trigonometry is the study of triangles and angles. It gives us formulas for what happens to a point's coordinates when you rotate it around a center.

You have not met rotation formulas before, so we will build one from scratch. Start with a flat surface, like a piece of paper. Pick a dot on that paper. You can describe the dot's location by two numbers: how far right it is (call this x) and how far up it is (call this y). The center of rotation is the point where x and y are both zero.

Now imagine rotating that dot around the center by an angle, which we will call theta (theta). If theta is positive, we rotate in a counter-clockwise direction. Where does the dot end up? Let the new location be x-prime (x') for the horizontal position and y-prime (y') for the vertical position.

Trigonometry tells us exactly how to calculate x' and y'. We need two values from our angle theta:
1. The **cosine** of theta, written as cos(theta).
2. The **sine** of theta, written as sin(theta).

The new coordinates are calculated by mixing the old coordinates with these sine and cosine values:

x' = x multiplied by cos(theta) minus y multiplied by sin(theta)
y' = x multiplied by sin(theta) plus y multiplied by cos(theta)

These two lines are the entire mathematics of a single rotation. In RoPE, we do not just rotate once. We have a vector with many numbers (dimensions). We group these numbers into pairs. For each pair, we apply a rotation. The angle of rotation depends on which pair it is and where the word is in the sentence.

Let us define the variables:
- d is the total number of dimensions in the vector. This number must be even.
- m is an index that counts the pairs. We start at m = 0 and go up to m = d divided by 2 minus 1. So if d is 4, m takes the values 0 and 1.
- pos is the position of the word in the sequence (e.g., the first word is pos=0, the second is pos=1).
- theta-m (theta sub m) is the specific angle for the m-th pair. It is calculated as:

theta-m = pos divided by 10000 raised to the power of (2 times m divided by d)

This formula means that pairs with smaller m rotate more slowly, and pairs with larger m rotate faster. This creates a pattern where relative positions are preserved.

For each pair m, let the original numbers be x-sub-2m and x-sub-(2m+1). The rotated numbers, x'-sub-2m and x'-sub-(2m+1), are:

x'-sub-2m = x-sub-2m multiplied by cos(theta-m) minus x-sub-(2m+1) multiplied by sin(theta-m)
x'-sub-(2m+1) = x-sub-2m multiplied by sin(theta-m) plus x-sub-(2m+1) multiplied by cos(theta-m)

Why do we do this? The key insight comes from the **dot product**. The dot product is a way to measure how similar two vectors are. You multiply corresponding numbers in two vectors and add them all up.

If you take two vectors that have been rotated by RoPE, one at position i and one at position j, their dot product contains a term that is the cosine of (i minus j). This means the similarity score depends on the *difference* between their positions (the relative distance), not on their absolute positions. This allows the model to understand that 'cat' and 'sat' are related by a fixed distance, regardless of where they appear in the sentence.


## A worked example

We will rotate a tiny 2D vector to see the formulas in action. Take a vector v = (1, 0). This is a point on the x-axis, pointing directly right.

We will rotate this vector by two different angles: 30 degrees and 60 degrees. Then we will compute the dot product between the two rotated vectors to see how they relate.

**Step 1: Get the trigonometric values for 30 degrees.**
cos(30°) is approximately 0.866.
sin(30°) is exactly 0.5.

**Step 2: Rotate the vector (1, 0) by 30 degrees.**
Here, x = 1 and y = 0. Theta = 30°.
x'-sub-0 = 1 multiplied by cos(30°) minus 0 multiplied by sin(30°)
x'-sub-0 = 1 * 0.866 - 0 * 0.5
x'-sub-0 = 0.866

y'-sub-1 = 1 multiplied by sin(30°) plus 0 multiplied by cos(30°)
y'-sub-1 = 1 * 0.5 + 0 * 0.866
y'-sub-1 = 0.5

So the vector rotated by 30 degrees is v_30 = (0.866, 0.5).

**Step 3: Get the trigonometric values for 60 degrees.**
cos(60°) is exactly 0.5.
sin(60°) is approximately 0.866.

**Step 4: Rotate the original vector (1, 0) by 60 degrees.**
Here, x = 1 and y = 0. Theta = 60°.
x'-sub-0 = 1 multiplied by cos(60°) minus 0 multiplied by sin(60°)
x'-sub-0 = 1 * 0.5 - 0 * 0.866
x'-sub-0 = 0.5

y'-sub-1 = 1 multiplied by sin(60°) plus 0 multiplied by cos(60°)
y'-sub-1 = 1 * 0.866 + 0 * 0.5
y'-sub-1 = 0.866

So the vector rotated by 60 degrees is v_60 = (0.5, 0.866).

**Step 5: Compute the dot product of v_30 and v_60.**
The dot product is the sum of the products of corresponding components.
dot(v_30, v_60) = (x-component of v_30 * x-component of v_60) + (y-component of v_30 * y-component of v_60)
dot(v_30, v_60) = (0.866 * 0.5) + (0.5 * 0.866)

Calculate the first part:
0.866 * 0.5 = 0.433

Calculate the second part:
0.5 * 0.866 = 0.433

Add them together:
0.433 + 0.433 = 0.866

**Step 6: Check against the cosine of the angle difference.**
The angle between the two rotated vectors is the difference between their rotation angles: 60° - 30° = 30°.
cos(30°) is approximately 0.866.

**Result:**
The dot product (0.866) equals cos(30°). This confirms that the similarity between two rotated vectors depends only on the difference in their rotation angles (which corresponds to the relative distance in position), not on their absolute positions.


## From scratch, in Python

```python
import math

def rotate_pair(x0, x1, theta):
    """Rotate a single pair of numbers by angle theta."""
    c = math.cos(theta)
    s = math.sin(theta)
    new_x0 = x0 * c - x1 * s
    new_x1 = x0 * s + x1 * c
    return new_x0, new_x1

def rope_rotate(vec, pos):
    """Rotate all pairs of a vector by angles depending on position."""
    d = len(vec)
    rotated = []
    for i in range(0, d, 2):
        theta = pos / (10000 ** (i / d))
        new0, new1 = rotate_pair(vec[i], vec[i+1], theta)
        rotated.append(new0)
        rotated.append(new1)
    return rotated

# Example: a 4-dimensional query vector for position 2
vec = [1.0, 0.0, 0.5, -0.3]
pos = 2
rotated = rope_rotate(vec, pos)
print(rotated)  # Output around [0.9998, 0.02, 0.495, -0.306]
```


## The same thing, with a framework

```python
In PyTorch, you typically don't implement RoPE manually; you use a library like `transformers` that has it built into the model. But if you wanted to implement it in NumPy, you'd do the same thing with arrays:

import numpy as np

def rope_rotate_np(vec, pos):
    d = len(vec)
    rad = np.arange(0, d, 2)
    theta = pos / (10000 ** (rad / d))
    c = np.cos(theta)
    s = np.sin(theta)
    vec_pairs = vec.reshape(-1, 2)
    x0 = vec_pairs[:, 0]
    x1 = vec_pairs[:, 1]
    new0 = x0 * c - x1 * s
    new1 = x0 * s + x1 * c
    return np.stack([new0, new1], axis=1).reshape(-1)

# In a transformer, you'd apply this to both the query and key before the dot product.
# The library `transformers` uses it internally when you set `use_rope=True` or when the model architecture specifies it.
# A developer rarely writes RoPE by hand; they call `model(q, k, v)` and the framework does it.
```


## What the code does

The from-scratch code defines one function `rotate_pair` that takes two numbers (a pair) and an angle, and returns the rotated pair using the sine and cosine formulas we worked through by hand. You give it x0 and x1, and it returns new_x0 and new_x1. The `rope_rotate` function iterates over the vector in steps of 2 — that is, it looks at the vector as a list of pairs. For each pair, it computes the angle theta based on the position and the index of the pair, then calls `rotate_pair`. The `math.cos` and `math.sin` are built-in Python functions that return the cosine and sine of an angle in radians. We use a loop because we don't know how many pairs there are — we just step through the list. The framework version does the same thing but uses NumPy arrays to do all pairs at once, which is faster on GPUs. Developers don't usually write this themselves; they rely on a model class that includes RoPE as part of its forward pass.


## Common misconceptions

**RoPE only cares about absolute position** — RoPE just adds a number to each position so the model knows the word order.

RoPE rotates the vector, and the dot product between two rotated vectors depends on the *difference* of their angles, not their absolute values. That difference is what encodes relative distance. In fact: RoPE encodes position such that the attention score becomes a function of the relative distance between the two tokens, not where each token sits in absolute terms.

**The rotation angle is the same for all dimensions** — Every pair in the vector is rotated by the same angle, which would mean all dimensions rotate in lockstep.

The angle θ_m varies with the pair index m — earlier pairs get a larger angle, later pairs get a smaller one, which allows different dimensions to capture different frequencies of position information. In fact: Each pair has its own frequency, governed by the base 10000, so the model can represent both fine-grained and coarse-grained position differences.

**RoPE adds a constant to the vector** — You add position-related numbers to the query and key vectors, just like sinusoidal embeddings are added to the input.

RoPE multiplies the vector by a rotation matrix — it does not add anything. Adding would shift the vector, but multiplying by a rotation preserves the vector's length and only changes its direction. In fact: RoPE applies a rotation (a matrix multiplication) to the query and key vectors, which changes their orientation but not their magnitude.

**RoPE is just for generating text** — Only generative models use RoPE; it is irrelevant for BERT-style encoders.

Any transformer that needs positional information can use RoPE, and many encoders (like some recent multilingual models) also adopt it. In fact: RoPE is a general positional encoding method, used in both encoder and decoder architectures where relative distance matters.

**Longer context always works with RoPE** — If a model is trained with RoPE on 2,000 tokens, it will automatically work on 10,000 tokens with no changes.

The rotation angles become very large for positions far outside the training range, which can break the scale of the attention scores. The model needs to have been trained on long contexts or you need to adjust the base frequency. In fact: RoPE generalises better than absolute embeddings, but you still need to tune the frequency or train on the longer context to get good performance.

**RoPE is a separate network layer** — Rotary embeddings are a standalone neural network component that you train as a separate module.

RoPE is not trainable; it is a fixed mathematical operation applied on the fly. There are no parameters to learn. In fact: RoPE is a deterministic function of position — you compute the angle, you rotate, and that's it. It does not have its own weights.


## In short

RoPE is a way to bake position into attention by rotating each word's query and key vectors by an angle that grows with its position. The dot product between a query and a key then measures agreement minus the difference in their rotation angles — so the attention score naturally depends on how many positions apart they are. This is why almost every modern open-weight LLM uses it: it is simple, cheap, and it lets models handle longer texts than they saw in training. Remember: rotation, not addition; relative, not absolute; fixed, not learned.


## Where this is used

RoPE is everywhere in modern AI. **Large language models** — Llama 2, Llama 3, Mistral, Gemma, and many others — use it as their positional encoding. When you fine-tune such a model on a long document, the model's ability to handle long contexts is directly due to RoPE: because the attention score depends on *relative* distance, the model can generalise to sequences longer than those in its training data far better than with absolute position embeddings. A practitioner working with, say, a 2,000-token context window but wanting to infer on 10,000 tokens will often look for a model that uses RoPE and attempt 'context extension' techniques — like adjusting the base frequency (the 10000 in the formula) — which is only possible because RoPE's relative nature allows such interpolation. **Small language models running on phones or laptops** also rely on RoPE, because it is computationally cheap — a rotation per pair is a few multiplications and additions, far less expensive than, say, adding a learned embedding, and it works well with smaller context windows where the relative distance is small. **Retrieval-augmented generation** (RAG) does not directly use RoPE within the retriever, which often uses a bi-encoder, but the generator (the LLM) does, and when you are chunking documents to fit into the context window, the model's ability to handle relative positions within each chunk is a consequence of RoPE. **Agentic systems** that plan and call tools often interact with long conversation histories — RoPE is what lets the model remember which message is older and which is newer, which is crucial for maintaining a coherent plan. **Everyday applied machine learning** that uses transformers for, say, time-series forecasting, also inherits RoPE if the model is a transformer — a researcher building a transformer for stock prices will often use RoPE because relative time differences matter more than absolute timestamps. In practice, a developer reaches for the Hugging Face `transformers` library, loads a model family that uses RoPE (like Llama), and does not write any of this — but when they want to *modify* the RoPE parameters (like the base frequency) or debug why the model loses performance on long inputs, they find the code in the model's `rotary_emb` module and need to understand it. When done badly — for instance, blindly applying RoPE without adjusting the frequency for a much longer context than trained — the attention scores become noisy because the relative angle becomes large, and the model degrades to nonsense. Doing it well — knowing when to recalculate theta for longer sequences — is what separates a practitioner who can effectively use modern long-context LLMs from one who hits a wall at the training length.


---

Read this on the web: https://basavapurushottam.com/learn-ai/rope-rotary-embeddings
