# Multi-Head Attention

Running several attention operations in parallel on projected subspaces, then concatenating, so different heads can specialise.

**Rests on:** Self-Attention from Scratch, Tensors and Shapes

**Leads to:** Cross-Modal Fusion, Positional Encoding

*Deep Learning · Transformers*


## Why learn this

Multi-head attention is the engine inside every modern large language model — ChatGPT, Claude, Gemini, and the open models like Llama all run on it. Without this page, the phrases 'attention is all you need' and 'multi-head self-attention' are just noise. With it, you can read a model architecture diagram and see what each component does, understand why models can look at an entire sentence at once, and grasp why scaling up the number of heads helps. This concept unlocks the deeper study of Transformers, causal masking (the rule that lets a model only look backward when writing), positional encodings (how a model knows word order), and the GPT architecture — all the machinery that makes generative AI work. In practical terms, after this page you'll be able to explain to a colleague why a model can correctly link 'she' to 'Maria' in a long document, and why throwing more heads at a problem isn't always better.


## The intuition

Imagine you are a detective trying to solve a mystery in a room full of people. You have one spotlight. You point it at the person who just spoke (this is the query — the specific item you are currently focused on), then at the person they might be referring to (the keys — the items you are checking for relevance), then at the person who looks nervous (the values — the actual details or context attached to those relevant items). You can only look at one thing at a time, and have to decide where to point the light. That is single-head attention.

Now imagine you have a team of detectives, each with their own spotlight. One detective always focuses on who is speaking, another on who is being talked about, another on where people are standing, another on the timeline of events. They all search simultaneously, in parallel, and then they pool their findings. That is multi-head attention.

Each detective is a 'head', and each one specialises in a different kind of relationship. The key insight is that by running several attention operations in parallel, the model can capture different kinds of connections at the same time — syntax, position, meaning — without them interfering. In the original Transformer paper, they used 8 heads.

Each head looks at a 'projected subspace' — this means we take the original information, split it into smaller chunks, and rotate or transform those chunks so each detective sees a slightly different version of the facts. For example, one detector might see only grammatical structures, while another sees only emotional tones. This filtering allows each head to focus on its specialty without distraction.

The outputs from all heads are then concatenated — joined together into one long list — and passed to the next layer. The total computation stays roughly the same as a single larger attention operation, because each head works on a smaller dimension.


## An analogy

Think of a large tech company's board meeting. The CEO (the query) wants to know the status of every project. The project leads (the keys) present their updates. The actual progress reports (the values) are the details.

In single-head attention, the CEO listens to each project lead, scores how relevant each update is, and then takes a weighted average of the reports — focusing mostly on the most relevant one. That works, but the CEO can only pay attention to one aspect of the reports at a time.

What if the CEO could clone themselves? Clone A focuses exclusively on budget numbers. Clone B focuses on timeline delays. Clone C focuses on team morale. Each clone has a different 'lens' — they look at the same reports but filter them differently. After the meeting, all clones merge their notes into a single summary. That is multi-head attention. The clones are the 'heads', and each has its own projection of the queries, keys, and values — its own filter.

The important detail is that the clones don't interfere with each other; they work in parallel, and their combined report is richer than any single clone could produce. But this analogy breaks down in one important way: the heads are not manually assigned specialties. The model learns what each head should focus on during training. Sometimes heads specialise in syntax, sometimes in coreference (like linking 'she' to 'Maria'), sometimes in position. And we don't always control what they learn — some heads end up redundant and can be pruned without hurting performance.


## Definition

Multi-head attention is a mechanism that runs several attention operations in parallel, each on a different filtered version of the inputs (where the input data is split and transformed so each part focuses on a specific aspect), then concatenates the results, so that different heads can learn to specialise in different kinds of relationships.


## How it connects

Your library notes under Transformers say that self-attention computes a weighted mixture of all positions, with weights derived from content. Multi-head attention is a direct extension of that: instead of one weighted mixture, you compute several, each with its own filtered views (queries, keys, and values), and then combine them. You've also noted that causal masking is the single change that turns a bidirectional encoder into a generative decoder — multi-head attention is what gets masked in that case. And you have notes on positional encodings; the heads operate on the positions that those encodings inject. This concept builds on your notes about self-attention from scratch, and it leads directly to understanding the GPT architecture, where multi-head attention is stacked in layers.


## The mathematics

This concept rests on a branch of mathematics called linear algebra, and specifically on two operations: the dot product and matrix multiplication. Don't worry if those words are unfamiliar — we'll build them from scratch.

A dot product measures how much two lists of numbers agree. If you have two lists of the same length, you multiply the first pair, the second pair, and so on, then add all those products together. A large positive result means the lists point in the same direction; a large negative result means they point in opposite directions; a zero means they are unrelated. In attention, we use the dot product between a query and a key to score how relevant the key's position is to the query.

Matrix multiplication is just a compact way to do many dot products at once. A matrix is a table of numbers. When you multiply a matrix by a vector (a list of numbers), you compute the dot product of each row of the matrix with the vector, and the result is a new vector — one number for each row. So a matrix acts as a collection of filters, each looking for a different pattern.

Now, in multi-head attention, we have three sets of vectors: queries, keys, and values. For each head, we take the original queries, keys, and values and multiply them by a special matrix that projects them into a smaller subspace. Think of it as putting on tinted glasses — each head sees the same scene but through a different filter. Then we compute attention scores: for each query, we take the dot product with every key to get a score, divide by the square root of the head dimension (to keep the numbers stable), and turn those scores into probabilities using something called softmax. Softmax takes a list of numbers and converts them to probabilities that sum to 1, making larger numbers bigger and smaller numbers smaller.

Then we take the weighted sum of the values, where the weights are those probabilities. That gives us the output of one head. We do this for all heads in parallel, then concatenate the outputs — join them together side by side — and pass the combined result through one more projection. The entire process lets the model attend to different kinds of relationships simultaneously.

For each head i:  
Head_i = softmax(Q_i K_i^T / sqrt(d_k)) V_i  
Then output = Concat(Head_1, ..., Head_h) W_O  
where Q_i = Q W_Q^i, K_i = K W_K^i, V_i = V W_V^i.  
Each symbol: Q is the query matrix (all queries stacked), K is the key matrix, V is the value matrix. W_Q^i, W_K^i, W_V^i are projection matrices for head i, d_k is the dimension of each head, and W_O is the output projection matrix.


## A worked example

```
Let's work with a tiny example. Suppose we have a sequence of two words. Each word is represented by a 4-dimensional vector. We'll have two heads. For simplicity, we'll skip the input projections and just use the raw vectors as queries, keys, and values for each head. Word A: [1, 0, 1, 0]. Word B: [0, 1, 0, 1]. Each head will have a dimension of 2, so we'll take the first two numbers for head 1 and the last two for head 2. We'll compute attention scores without the scaling factor for simplicity.

Step 1: Head 1: Query_A = [1, 0], Key_A = [1, 0], Key_B = [0, 1]   # We split the 4-dim vector into two 2-dim parts for head 1.
Step 2: Score_A_A = dot([1,0], [1,0]) = 1*1 + 0*0 = 1   # Dot product of query A with key A.
Step 3: Score_A_B = dot([1,0], [0,1]) = 1*0 + 0*1 = 0   # Dot product of query A with key B.
Step 4: Softmax over [1, 0]: exp(1)/(exp(1)+exp(0)) ≈ 0.731, exp(0)/(exp(1)+exp(0)) ≈ 0.269   # Convert scores to probabilities.
Step 5: Output_A_head1 = 0.731*[1,0] + 0.269*[0,1] = [0.731, 0.269]   # Weighted sum of values.
Step 6: Head 2: Query_A = [1, 0], Key_A = [1, 0], Key_B = [0, 1] (same as head 1, but different projection)   # For this example, we use the same split, but in reality each head has different weights.
Step 7: Output_A_head2 = same as head 1? No, let's assume a different projection: Head 2 uses [0,1] as query and keys for a different pattern. For simplicity, we'll just say the outputs differ.   # In a real model, heads have different projections, so they compute different scores.
Step 8: Concatenate: [0.731, 0.269] (head1) and [0.5, 0.5] (head2) → [0.731, 0.269, 0.5, 0.5]   # Join the outputs side by side.
Step 9: Multiply by output projection matrix to get final output of multi-head attention.   # This mixes information from all heads into a single vector.

Result: The output is a vector that encodes the weighted mixture of values from all heads, capturing different relationships. The exact numbers depend on the projections, which are learned during training.
```


## From scratch, in Python

```python
import math

# We'll implement multi-head attention from scratch using plain Python.
# No libraries except math for exp.

def softmax(scores):
    """Convert a list of raw scores to probabilities."""
    max_score = max(scores)
    exps = [math.exp(s - max_score) for s in scores]
    total = sum(exps)
    return [e / total for e in exps]

def dot_product(v1, v2):
    """Multiply two lists of the same length, pair by pair, and add."""
    total = 0
    for i in range(len(v1)):
        total += v1[i] * v2[i]
    return total

def attention_scores_and_output(query, keys, values):
    """Single-head attention."""
    scores = []
    for key in keys:
        scores.append(dot_product(query, key))
    # Scale (we'll skip scaling for this example)
    probs = softmax(scores)
    # Weighted sum of values
    output = [0] * len(values[0])
    for i, value in enumerate(values):
        for j in range(len(value)):
            output[j] += probs[i] * value[j]
    return output, probs

def multi_head_attention(queries, keys, values, num_heads, head_dim):
    """Compute multi-head attention by splitting the last dimension."""
    # Assume each vector length is num_heads * head_dim
    all_head_outputs = []
    for head in range(num_heads):
        # For each head, slice the vectors
        start = head * head_dim
        end = start + head_dim
        head_queries = [q[start:end] for q in queries]
        head_keys = [k[start:end] for k in keys]
        head_values = [v[start:end] for v in values]
        head_outs = []
        for query in head_queries:
            output, _ = attention_scores_and_output(query, head_keys, head_values)
            head_outs.append(output)
        all_head_outputs.append(head_outs)
    # Concatenate across heads
    concatenated = []
    for pos in range(len(queries)):
        vec = []
        for head_out in all_head_outputs:
            vec.extend(head_out[pos])
        concatenated.append(vec)
    return concatenated

# Example usage
queries = [[1, 0, 1, 0], [0, 1, 0, 1]]  # two positions, each 4-dim
keys = [[1, 0, 1, 0], [0, 1, 0, 1]]
values = [[1, 0, 1, 0], [0, 1, 0, 1]]
result = multi_head_attention(queries, keys, values, num_heads=2, head_dim=2)
print(result)
```


## The same thing, with a framework

```python
In PyTorch, you would use `torch.nn.MultiheadAttention`. Underneath, it does exactly what the from-scratch code does: it projects the input to queries, keys, and values using linear layers, splits them into heads, computes scaled dot-product attention, concatenates, and applies a final projection. You don't have to write the loops yourself.
```


## What the code does

Let's walk through the from-scratch code:

- `softmax(scores)`: We take a list of raw numbers (like [1.0, 0.5]). We subtract the maximum for numerical stability, then exponentiate each, and divide by the sum. This turns the list into probabilities that sum to 1.
- `dot_product(v1, v2)`: This is the core of attention. It multiplies each corresponding pair of numbers and adds them up. For example, dot_product([1,0], [0,1]) = 1*0 + 0*1 = 0.
- `attention_scores_and_output(query, keys, values)`: This computes one head's output. For each key, it computes the dot product with the query, softmaxes all those scores, then takes a weighted average of the values using those probabilities.
- `multi_head_attention(...)`: This is the main loop. It splits the input vectors into `num_heads` parts, each of length `head_dim`. For each head, it runs the single-head attention on those slices. Then it concatenates the outputs from all heads — that is, it joins them end-to-end. So a position that had 4 numbers (for 2 heads of size 2) becomes a 4-number output again.

This is a simplified version: real multi-head attention also learns projection matrices that transform the inputs before splitting, and applies an output projection at the end. But the core idea is exactly this.


## Common misconceptions

**More heads always means better performance.** — If I add more heads, the model will always get better.

Beyond a certain point, heads become redundant and stop learning distinct patterns. The total parameter count grows, which can lead to overfitting if the dataset is small, and training becomes slower. In fact: More heads can help, but only up to a point. Many heads can be pruned without much loss, showing that redundancy is real. The right number of heads is a hyperparameter found by experimentation.

**Each head has a predetermined role like 'subject-verb agreement'.** — The model is designed so that head 1 always handles syntax and head 2 always handles coreference.

Nothing in the architecture forces a head to do one thing. The heads learn their roles purely from the training data, and often the roles are not cleanly separable — a head may combine several functions. In fact: Heads can specialise, but that specialisation is learned, not designed. We can inspect heads after training and find patterns, but they were never explicitly assigned.

**Multi-head attention is fundamentally different from single-head attention.** — Multi-head attention is a completely new mechanism, not just running single-head attention multiple times.

The only difference is that we run several attention operations in parallel on different projected subspaces and then concatenate the results. The core computation is identical. In fact: Multi-head attention is just several single-head attentions in parallel, with each head working on a smaller dimension, followed by a concatenation and a mix.

**The dimension must be split equally across heads always.** — You can't use heads of different sizes or an odd split.

In practice, the standard is to split equally, but it's not mathematically required. Some architectures use different head sizes, but the common choice is uniform for simplicity and efficiency. In fact: Equal splitting is standard, but not mandatory. The constraint is that the total model dimension must be divisible by the number of heads if you want equal sizes.

**Attention heads work independently with no interaction.** — Each head computes its output on its own, and they never mix until the very end.

They are independent in their computation, but the final output projection mixes them all together, and the attention scores in each head depend on the same input features. In fact: Heads operate in parallel but their outputs are combined through the concatenation and the output projection, so information from different heads is mixed.


## In short

Multi-head attention runs several attention operations in parallel on different projected subspaces, then concatenates and mixes the results. It lets a model capture many different relationships at once — like syntax, coreference, and position — without increasing the total computational cost much. It is the core of every transformer-based LLM. Remember: the heads are not manually assigned roles; they learn to specialise from data, and many are redundant and can be pruned.


## Where this is used

Multi-head attention is the backbone of virtually every modern large language model, including the GPT series, Claude, Gemini, and open models like Llama. In a transformer block, multi-head attention runs first, followed by a feed-forward network. The heads allow the model to simultaneously track different linguistic relationships: one head might specialise in subject-verb agreement, another in coreference resolution (like linking 'she' to 'Maria'), another in capturing positional relationships, and others in more abstract patterns. During training, the model learns which relationships matter for the task, and the heads naturally specialise.

In small language models that run on phones or laptops (like the ones in the TinyML ecosystem), multi-head attention is still present, but the number of heads and the hidden dimension are reduced to fit within memory and compute constraints. The key trade-off is that fewer heads reduce expressiveness, but the model can still capture many relationships because each head is learning a combination of features.

In retrieval-augmented generation (RAG), the retriever is usually based on embeddings (vector representations of text), but the generator, typically a transformer, uses multi-head attention to combine the retrieved documents with the user's query. The cross-attention mechanism in encoder-decoder models, which is a form of multi-head attention where queries come from one sequence and keys/values from another, allows the decoder to condition on the retrieved content. So multi-head attention is what lets the model blend external knowledge into its response.

In agentic systems that plan and call tools, multi-head attention is used in the underlying LLM to reason about the conversation history, tool results, and the next action. The attention heads help the model focus on relevant past messages and tool outputs. For example, when an agent decides to call a weather API, the attention mechanism weighs the user's question, the tool's response, and the dialogue history to generate the final answer or the next API call.

In everyday applied machine learning — beyond text — multi-head attention appears in vision models (like Vision Transformers, where image patches are treated as tokens), speech recognition (where audio frames are the tokens), and recommendation systems (where user history and item features are the tokens). In each case, the principle is the same: compute attention scores between all pairs of tokens, allowing the model to focus on relevant parts of the input regardless of distance.

When a developer works with multi-head attention in practice, they rarely write the from-scratch code. They reach for a library like PyTorch or TensorFlow, and instantiate a layer with something like `torch.nn.MultiheadAttention(embed_dim, num_heads)`. They must choose the number of heads and the dimension of each head. A common configuration is 12 heads with a head dimension of 64. The developer feeds in the queries, keys, and values (often the same tensor for self-attention) and receives the output. The hard part is debugging: if the model doesn't learn, it could be because the heads are redundant or the scaling was wrong. Breaking the implementation to inspect individual head outputs, as we did in the from-scratch code, is often the fastest way to find the issue.


---

Read this on the web: https://basavapurushottam.com/learn-ai/multi-head-attention
