# Self-Attention from Scratch

Computing queries, keys and values from the same sequence, scoring every position against every other, and mixing values by those scores.

**Rests on:** Automatic Differentiation

**Leads to:** Causal Attention Masking, Cross-Attention, Multi-Head Attention, RoPE Rotary Embeddings

*Deep Learning · Transformers*


## Why learn this

You know that when you read a difficult sentence, your brain doesn't process each word in isolation. It looks back at earlier words to clarify meaning, or looks ahead to understand context. Self-attention is the mathematical technique that gives artificial intelligence this same ability. It is the core engine behind ChatGPT, Claude, and other modern language models. Understanding it from scratch means you will never again feel lost when someone mentions 'transformers', 'context windows', or how AI handles long documents. It unlocks the next topics in your library: Causal Attention Masking (how a model writes text one word at a time without cheating), Multi-Head Attention (how models track several relationships at once), and ultimately the GPT Architecture. Without self-attention, none of these make sense.


## The intuition

Imagine you are reading this sentence: "The trophy did not fit in the suitcase because it was too big." Your brain immediately links the word "it" to "trophy," not "suitcase." How? By scoring how well each earlier word matches the current question. Self-attention does exactly this, but with numbers for every word at once.

Here is how the arithmetic works, step by step, using a tiny example with just three words: A, B, and C. We want to update the meaning of Word A.

1. **The Setup**: For each word, we have three lists of numbers (vectors). Think of them as simple tags:
   - Query for A: [1, 0] (What A is looking for)
   - Key for B: [2, 1] (What B offers)
   - Value for B: [5, 5] (The actual content B provides)

2. **Scoring Compatibility**: To see how relevant B is to A, we calculate a "score." We multiply corresponding numbers in the Query and Key lists, then add the results together.
   - Multiply first positions: 1 * 2 = 2
   - Multiply second positions: 0 * 1 = 0
   - Add them: 2 + 0 = 2. This is the score.
   
   If we did this for every other word, we would get a list of scores. Let's say the scores are: A vs A = 3, A vs B = 2, A vs C = 1.

3. **Weighting**: We want to combine information from all words into A. But we care more about high-scoring words. We turn these scores into weights by dividing each score by the total of all scores (3+2+1=6). So A gets a weight of 3/6 (50%) from itself, 2/6 (~33%) from B, and 1/6 (~17%) from C.

4. **Weighted Mixture**: Now we blend the "Value" lists. A takes 50% of its own Value, plus 33% of B's Value, plus 17% of C's Value.
   - If B's Value was [5, 5] and C's Value was [0, 10]:
   - Contribution from B: 0.33 * [5, 5] ≈ [1.65, 1.65]
   - Contribution from C: 0.17 * [0, 10] ≈ [0, 1.7]
   
   This blend is the "weighted mixture." The word "it" in the original example has a high score for "trophy" and a low score for "suitcase," so its final meaning is dominated by "trophy"'s value. In arithmetic terms, high agreement (score) means that word pulls a large share of its value into the current understanding.


## An analogy

Think of a team meeting where every member must write a summary of what they heard, but they are allowed to borrow ideas from others. Each member has a notepad with three columns: what they want to know (Query), what they offer (Key), and what they actually say (Value). To decide how much to borrow from any colleague, they look at their own Query and the colleague's Key—if the Key answers the Query well, they copy a large portion of that colleague's Value into their own summary. Each person does this for everyone else simultaneously, so the final summary is a blend of everyone's input, weighted by how relevant each person was. The analogy breaks in one key way: in a meeting, people can choose to ignore someone completely; in self-attention, every word always takes a tiny bit of every other word—some bits are just vanishingly small. Additionally, the "agreement" between Query and Key is calculated by multiplying corresponding numbers and adding them up, not by human judgment.


## Definition

Self-attention is a process that updates each item in a sequence by blending information from all items (including itself), where the blend ratio depends on how well each item's 'offer' matches the current item's 'search'.


## How it connects

This sits beside Causal Attention Masking, which adds a rule to this blending process so a model can only look at past words—this turns it into a generator. It also sits beside Multi-Head Attention, which runs several of these blending operations in parallel so different aspects of meaning can emerge simultaneously.


## The mathematics

Self-attention is a way for a computer to look at all the words in a sentence at once and decide which words are most important for understanding each specific word. It does this by comparing every word to every other word.

To do this, the computer turns every word into three lists of numbers (vectors). We call these lists:
- The Query (Q): What the current word is looking for.
- The Key (K): What information the other words are offering.
- The Value (V): The actual content or meaning carried by the other words.

The process happens in three main steps for every word in the sentence.

Step 1: Measuring Relevance (The Dot Product)
For a specific word, we take its Query list and compare it to the Key lists of all words in the sentence (including itself). We do this by multiplying the first number of the Query by the first number of the Key, then the second by the second, and so on. Finally, we add all those products together. This single number is called the 'score'.

A high positive score means the word being looked at has features that match what the current word is looking for. A low or negative score means they do not match.

Step 2: Turning Scores into Weights (Softmax)
After comparing the Query to all Keys, we have a list of raw scores. These numbers can be anything—positive, negative, large, small. We need to turn them into weights that behave like percentages: they must all be positive, and they must add up to 1 (or 100%).

We do this using a function called 'softmax'. It works in two parts:
1. Exponentiation: For each score, we calculate e raised to the power of that score. (e is approximately 2.718). This ensures every number becomes positive. If a score was negative, it becomes a small positive fraction. If it was large and positive, it becomes a very large number.
2. Normalization: We add up all these new positive numbers. Then, for each word, we divide its exponential value by that total sum. This forces the weights to add up to 1.

Step 3: Combining Information (Weighted Sum)
Now we have a weight for every word in the sentence. These weights tell us how much of each word's 'Value' list we should keep.

We multiply the Value list of every word by its corresponding weight. Then, we add all those weighted lists together. The result is a new list of numbers for our original word. This new list contains information from the whole sentence, but focused on the words that were most relevant.

The Scaling Factor
There is one important detail before Step 2. When we calculate the dot product in Step 1, if the lists are long (have many numbers), the resulting score can become very large. If the scores are too large, the softmax function becomes 'sharp'. It will give a weight of nearly 1 to the highest score and nearly 0 to everything else. This makes it hard for the computer to learn because small changes in input don't change the output much.

To fix this, we divide the raw scores by the square root of the length of the lists (the dimension). This keeps the numbers in a manageable range so the softmax function produces smooth, useful weights.

In summary:
1. Calculate dot products between Query and all Keys.
2. Divide those scores by the square root of the list length.
3. Apply softmax to turn scores into weights that sum to 1.
4. Multiply each Value list by its weight and add them up.


## A worked example

Let us walk through a tiny example with just two words: 'Apple' and 'Banana'.

We assume each word is represented by lists of length 2 (two numbers). In real applications, these lists are much longer, but the math is the same.

The Data:
Word 1 ('Apple'):
- Query Q1 = [1, 0]
- Key K1 = [1, 1]
- Value V1 = [0.5, 0.5]

Word 2 ('Banana'):
- Query Q2 = [0, 1]
- Key K2 = [0, 1]
- Value V2 = [1, 0]

We will calculate the new representation for 'Apple' first.

Part A: Calculate Raw Scores for Apple's Query
Apple's Query is Q1 = [1, 0]. We compare this to the Keys of all words.

1. Compare Q1 with K1 (Apple's own key):
   Multiply corresponding numbers and add:
   (1 * 1) + (0 * 1) = 1 + 0 = 1.
   Score for Apple is 1.

2. Compare Q1 with K2 (Banana's key):
   Multiply corresponding numbers and add:
   (1 * 0) + (0 * 1) = 0 + 0 = 0.
   Score for Banana is 0.

Raw scores for Apple are [1, 0].

Part B: Scale the Scores
The length of our lists (dimension) is 2. We need to divide by the square root of 2.
Square root of 2 is approximately 1.414.

Scaled Score for Apple = 1 / 1.414 ≈ 0.707.
Scaled Score for Banana = 0 / 1.414 = 0.

Our scaled scores are [0.707, 0].

Part C: Apply Softmax to Get Weights
We need to turn [0.707, 0] into weights that sum to 1.

1. Exponentiate each score (e^x):
   e^0.707 ≈ 2.028.
   e^0 = 1.

2. Sum the exponentiated values:
   Total = 2.028 + 1 = 3.028.

3. Divide each exponential value by the total to get weights:
   Weight for Apple (W_A) = 2.028 / 3.028 ≈ 0.670.
   Weight for Banana (W_B) = 1 / 3.028 ≈ 0.330.

Check: 0.670 + 0.330 = 1.0. The weights are valid.

Part D: Compute the Weighted Sum of Values
Now we combine the Value lists using these weights.

Apple's Value V1 = [0.5, 0.5].
Banana's Value V2 = [1, 0].

Weighted contribution from Apple:
   0.670 * [0.5, 0.5] = [0.335, 0.335].

Weighted contribution from Banana:
   0.330 * [1, 0] = [0.330, 0].

Add these contributions together:
   New Apple Representation = [0.335 + 0.330, 0.335 + 0]
   New Apple Representation = [0.665, 0.335].

Now let us calculate the new representation for 'Banana'.

Part E: Calculate Raw Scores for Banana's Query
Banana's Query is Q2 = [0, 1]. We compare this to the Keys of all words.

1. Compare Q2 with K1 (Apple's key):
   (0 * 1) + (1 * 1) = 0 + 1 = 1.
   Score for Apple is 1.

2. Compare Q2 with K2 (Banana's own key):
   (0 * 0) + (1 * 1) = 0 + 1 = 1.
   Score for Banana is 1.

Raw scores for Banana are [1, 1].

Part F: Scale the Scores
Divide by square root of 2 (1.414).

Scaled Score for Apple = 1 / 1.414 ≈ 0.707.
Scaled Score for Banana = 1 / 1.414 ≈ 0.707.

Our scaled scores are [0.707, 0.707].

Part G: Apply Softmax to Get Weights
Since the scores are identical, the weights will be identical.

1. Exponentiate:
   e^0.707 ≈ 2.028.
   e^0.707 ≈ 2.028.

2. Sum:
   Total = 2.028 + 2.028 = 4.056.

3. Divide:
   Weight for Apple (W_A) = 2.028 / 4.056 = 0.5.
   Weight for Banana (W_B) = 2.028 / 4.056 = 0.5.

Part H: Compute the Weighted Sum of Values
Weighted contribution from Apple:
   0.5 * [0.5, 0.5] = [0.25, 0.25].

Weighted contribution from Banana:
   0.5 * [1, 0] = [0.5, 0].

Add these contributions together:
   New Banana Representation = [0.25 + 0.5, 0.25 + 0]
   New Banana Representation = [0.75, 0.25].

Final Result:
The new representation for 'Apple' is [0.665, 0.335].
The new representation for 'Banana' is [0.75, 0.25].

Notice that Apple's new vector is a mix of its own value and Banana's value, weighted by how much it attended to each. Banana's new vector is an equal mix because it attended equally to both words.


## From scratch, in Python

```python
import math

# Softmax function: converts a list of numbers into weights that sum to 1
def softmax(scores):
    # Find the maximum score to avoid huge exponentials
    max_score = max(scores)
    exps = [math.exp(s - max_score) for s in scores]
    sum_exps = sum(exps)
    return [e / sum_exps for e in exps]

# Dot product: multiplies two lists of numbers elementwise, then adds them up
def dot_product(a, b):
    total = 0
    for i in range(len(a)):
        total += a[i] * b[i]
    return total

# Self-attention for one position, given all queries, keys, values
def self_attention_one_position(pos, queries, keys, values):
    # Query for this position
    q = queries[pos]
    # Compute raw scores against all keys
    scores = []
    for key in keys:
        scores.append(dot_product(q, key))
    # Scale by square root of dimension
    d_k = len(q)
    scaled_scores = [s / math.sqrt(d_k) for s in scores]
    # Softmax to get weights
    weights = softmax(scaled_scores)
    # Weighted sum of values
    output = [0.0] * len(values[0])
    for i, weight in enumerate(weights):
        for j in range(len(output)):
            output[j] += weight * values[i][j]
    return output

# Example with two words as in the mathematics section
queries = [[1, 0], [0, 1]]
keys = [[1, 1], [0, 1]]
values = [[0.5, 0.5], [1, 0]]

output0 = self_attention_one_position(0, queries, keys, values)
output1 = self_attention_one_position(1, queries, keys, values)
print('Output for Apple:', output0)  # Should be approximately [0.6345, 0.3655]
print('Output for Banana:', output1)  # Should be approximately [0.75, 0.25]
```


## The same thing, with a framework

```python
import numpy as np

# Same example, but using NumPy for vectorised operations
queries = np.array([[1, 0], [0, 1]])
keys = np.array([[1, 1], [0, 1]])
values = np.array([[0.5, 0.5], [1, 0]])

# Compute scores: dot product between every query and every key
scores = queries @ keys.T  # Shape: (2, 2)

# Scale by sqrt of dimension
d_k = queries.shape[1]
scaled_scores = scores / np.sqrt(d_k)

# Softmax along each row (axis=1)
# To keep it numerically stable, subtract the row max
exp_scores = np.exp(scaled_scores - np.max(scaled_scores, axis=1, keepdims=True))
weights = exp_scores / np.sum(exp_scores, axis=1, keepdims=True)

# Weighted sum of values
output = weights @ values
print('Outputs:')
print(output)
```


## What the code does

The `from_scratch` code defines two helper functions. `softmax` takes a list of numbers, subtracts the largest to avoid overflow, exponentiates each, and divides by the sum, so the result is a list of positive numbers that add up to 1. `dot_product` takes two lists of equal length, multiplies their elements pairwise, and adds the products—this is exactly the dot product from the mathematics section. Then `self_attention_one_position` computes the attention output for one position. It gets the query for that position, computes a dot product against every key to get raw scores, scales each by the square root of the dimension (the length of the query), applies softmax to obtain weights, and finally computes a weighted sum of all values. The example at the bottom uses the same numbers as the numerical example, so you can run it and verify the outputs. In the framework version, we use NumPy to do the same operations but on whole matrices at once: `@` does matrix multiplication, which for the scores is equivalent to a loop over all query-key pairs. The softmax is applied along each row to get weights, and then a matrix multiplication with values produces the final output. This is not magic—it is exactly the same arithmetic, just written in a more compact and faster way, because NumPy is optimised for operations on arrays.


## Common misconceptions

**Attention weights must sum to 1 and are like probabilities** — The softmax weights are the same as probabilities, so the model is choosing one word to focus on.

Softmax weights do sum to 1 and are all positive, but they are not probabilities in a statistical sense—they are just normalised importance scores. The model never samples from them; it computes a deterministic weighted average. Also, every position always takes some of every other position; it is not a choice of one. In fact: Attention weights are a set of non-negative numbers that sum to 1, used to compute a weighted sum of values. They represent a smooth distribution of attention across all positions, not a hard choice.

**Self-attention is about looking at the nearest words** — Attention weights are usually higher for nearby words, so distance is the main factor.

Attention weights are driven by content, not distance. A pronoun can attend strongly to a noun that is dozens of words away if they are related. The learned query and key vectors encode semantic compatibility, not just locality. While in practice some heads may learn to focus on nearby words, others specialise in long-range dependencies. In fact: Self-attention enables direct long-range dependencies. The weight between any two positions depends on their content, not on how far apart they are in the sequence.

**Queries, keys, and values are the same as the input embeddings** — The query, key, and value vectors are just the input word embeddings repeated.

Queries, keys, and values are computed from the input embeddings by multiplying with learned matrices (W_q, W_k, W_v). They are projections—each is a different learned transformation of the same input. This allows the model to derive distinct roles for 'what to look for', 'what I offer', and 'what I contribute'. In fact: Queries, keys, and values are learned projections of the input embeddings, each with a distinct role in the attention mechanism.


## In short

Self-attention computes, for every position, a weighted mixture of all positions. It uses three learned vectors per position: query (what it seeks), key (what it offers), and value (what it contributes). The compatibility between a query and a key is measured by a dot product, scaled by the square root of the dimension to keep softmax from saturating, then the dot products are turned into weights with softmax. The output is the weighted sum of values. This single mechanism is what lets transformers capture long-range dependencies and is the foundation of all modern LLMs, RAG, agents, and much of applied ML.


## Where this is used

Self-attention is the core of every transformer, the architecture behind all modern language models. Here is how it shows up in each area you care about.

**Large Language Models (LLMs):** The most visible use: ChatGPT, Claude, Gemini, Llama—all are transformers. They consist of many layers, each with a self-attention sublayer followed by a feed-forward block. In each layer, every token's representation is updated by attending to all other tokens in the context. A developer using a pre-trained LLM does not write the self-attention code; they call an API like `openai.ChatCompletion.create()`, passing a list of messages. But under the hood, the model's context window—often thousands or tens of thousands of tokens—is processed simultaneously through self-attention. For example, when you give a long document and ask for a summary, the model's self-attention can connect the first sentence to the last, forming a coherent summary. Without self-attention, the model would have to read sequentially and might forget earlier information. The cost is quadratic: if the context is 10,000 tokens, the model computes 100 million attention scores per layer, per head. This is why long-context models are expensive and why there is active research into making attention linear or sparse.

**Small Language Models on phones/laptops:** When a model runs on a device, like the keyboard on your phone or a local LLM on your laptop, the same self-attention is used, but at a smaller scale. Because self-attention is quadratic, small models keep their context window short—a few hundred to a couple of thousand tokens—to stay fast and memory-light. A developer using something like Llama-2-7B on a laptop would still call a high-level function like `model.generate()`, but the underlying inference computes attention scores for every token pair in the prompt. The efficiency of the device implementation matters: hardware accelerators are built to do many parallel multiply-adds, which is exactly what attention needs. If you were to write the attention loop in Python without optimisation, it would be far too slow; that is why frameworks like PyTorch have pre-optimised attention kernels.

**Retrieval-Augmented Generation (RAG):** RAG combines a retrieval system (like a search over a vector database) with a generative model. The retrieval finds relevant documents from a corpus, and the generator—an LLM—reads those documents plus the user's question. How does self-attention come in? First, the retrieval itself often uses embeddings derived from an encoder that uses self-attention (like a BERT-style model). The query is embedded into a vector, and documents are embedded into vectors; similarity is measured by cosine similarity, which is a normalised dot product—the same operation you learned here. Second, when the generator receives the retrieved documents, it puts them into the context, and self-attention lets the model weigh which parts of the documents matter for the question. For example, if you ask 'What is the capital of Australia?' and a retrieved document has a sentence 'Canberra is the capital' among other noise, self-attention will assign high attention from the question tokens to the relevant document tokens, pulling that information into the answer. A developer building a RAG pipeline might use a library like LangChain, write a query, and the framework handles the retrieval, but the generation step still relies on self-attention inside the LLM.

**Agentic Systems:** An AI agent is an LLM that can plan, call tools, and act in loops. For example, an agent told to 'Book a flight and a hotel for my trip' might call a flight API, then a hotel API, then confirm the booking. Self-attention is critical because every action is part of the same context: the original instruction, the tool results, and the agent's own reasoning all appear in the prompt. Self-attention lets the agent connect the result of the flight API call back to the original request, adjusting its plan. For instance, if the flight API returns a time, the agent needs to attend to that time when deciding which hotel to choose. Without self-attention, the agent would treat each tool call in isolation and lose track of the goal. A developer builds an agent in a framework like LangChain or a direct loop: they append the tool output to the message list and call the LLM again. The attention mechanism is what makes the conversation coherent.

**Everyday Applied ML:** Self-attention is not just for language—it is used in vision (ViT), speech, recommendation, and forecasting. In **recommendation systems**, self-attention can model the sequence of a user's past purchases to predict the next one. Each item in the history is a token, and attention learns which past items are relevant to the next choice. For example, if a user buys a coffee maker and then coffee beans, the attention weights might strongly link those two items. In **time-series forecasting**, self-attention can capture long-range dependencies, like a yearly seasonality, by attending to the same month last year. In **computer vision**, an image is split into patches, and self-attention is applied over the patches, so the model can relate distant parts of an image—for instance, noticing that a person is holding a ball in the bottom right and a dog in the top left, both relevant to the label 'playing fetch'. Developers in these areas typically use prebuilt transformer models from a library like Hugging Face Transformers, calling something like `model(pixel_values)` or `model(input_ids)`, but all of it is self-attention underneath. **Fraud detection** is another example: a sequence of credit card transactions is processed with self-attention, and the model learns to flag unusual correlations between a current transaction and much earlier ones. **Speech** uses self-attention over audio frames to relate phonemes across time. In all these cases, the mechanism is the same: queries, keys, values, dot products, softmax, and weighted sums—you now understand the core of all of it.


---

Read this on the web: https://basavapurushottam.com/learn-ai/self-attention-from-scratch
