# Cross-Attention

Attention where queries come from one sequence and keys and values from another, conditioning generation on separate content.

**Rests on:** Positional Encoding, Self-Attention from Scratch

*Deep Learning · Transformers*


## Why learn this

Cross-attention is the mechanism that lets a transformer read from one source while generating another. Imagine you are writing a description for a photo you see on your screen; you look at the image (the source) to decide what words to write next (the output). Mastering this concept explains how models perform tasks like turning an image into text, translating languages by looking back at the original sentence, or pulling specific facts from a large document to answer a question. It is the core technology that allows artificial intelligence to combine information from different types of data, such as matching words with pictures.


## The intuition

Imagine you are a professional translator sitting at your desk. On your left is a document written in French. On your right is a blank screen where you will type the English translation word by word. You do not just look at the last English word you typed; instead, for every new word you write, you glance back at the entire French document to find the best match. That French text acts as your reference or context.

Cross-attention is the mathematical process that enables this behavior in a computer model. It allows the model to pull specific, relevant information from a separate source each time it generates a new piece of output. It is called 'cross' because the request for information (the query) comes from one sequence—the part of the translation being built right now—while the information available to search through (the keys and values) comes from another sequence—the original French text.

Consider the specific numbers in the calculation. The model first measures how relevant each word in the source is by multiplying the request by the available data. Because these multiplication results can become very large numbers that are hard to work with mathematically, the system divides each result by the square root of the dimension size (a number representing the length of the vector representation). This step is called scaling and keeps the numbers stable. Next, the system uses a function called softmax to convert these scaled scores into probabilities. Softmax works by raising each score to the power of e (Euler's number), then dividing each result by the sum of all such results for that step. The output is a set of percentages that add up to 100%, showing exactly how much attention to pay to each part of the source.

Because these steps use information from two different sequences—asking about one thing while looking at another—there is a direct, content-based connection between them.


## An analogy

Think of a chef in a busy kitchen following a long, detailed recipe card placed on the counter. The recipe card is the source of truth, and the dish being plated is the output. Each time the chef needs to add an ingredient or describe a step, they glance at the recipe to see what comes next. The chef does not re-read the entire card from the top every time; instead, they focus their eyes on the specific line relevant to their current action. In this analogy, the chef’s immediate task—‘what should I do now?’—is the query. The lines on the recipe card are the keys and values: the headings or bolded words act as keys that tell the chef what topic each line is about, while the text below them are the values containing the actual instructions. The chef mentally matches their current task to the most relevant line in the recipe and follows those specific instructions.

Where this analogy breaks down: a human chef has common sense and understands that ‘simmer for 10 minutes’ implies time passing, whereas a model has no such innate understanding and must learn these associations purely through training. Also, while a human reads lines sequentially, a model can weigh the relevance of every line in the source simultaneously.


## Definition

Cross-attention is a process where a model generates a sequence by repeatedly looking at a separate source sequence for context. It calculates relevance by measuring the match between the current step of generation (query) and all parts of the source (keys), converting those matches into weights (values) that determine how much information to borrow from the source.


## How it connects

This sits next to Self-Attention, which is a process where a model looks at every part of a single sequence to understand its own internal relationships. In self-attention, the queries, keys, and values all come from the same text. In cross-attention, the queries come from the generating side (often called the decoder), while the keys and values come from the reference side (often called the encoder) or a different type of data (a modality, such as an image versus text). It is also closely related to Multi-Head Attention, which is simply the technique of running several cross-attention processes in parallel to capture different kinds of relationships at once.


## The mathematics

Cross-attention allows one sequence (the decoder) to look at another sequence (the encoder). It works by comparing a 'query' from the first sequence against 'keys' from the second sequence. If they match well, the query pays attention to the corresponding 'value' in the second sequence.

Here is the step-by-step process:

1. Compute raw scores: For each query vector q and each key vector k, calculate their dot product. The dot product of two vectors [a, b] and [c, d] is (a * c) + (b * d). This number tells us how much the directions agree.

2. Scale the scores: Divide every raw score by the square root of the dimension of the keys (d_k). We do this to keep the numbers from becoming too large, which would make the next step unstable. The formula for the scaling factor is 1 / sqrt(d_k).

3. Convert to weights: Apply the softmax function to the scaled scores. Softmax takes a list of numbers and turns them into positive weights that add up to exactly 1. It does this by taking the exponential (e raised to the power of each number) and then dividing each result by the sum of all exponentials. This ensures the output is a valid probability distribution.

4. Weighted sum: Multiply each value vector v by its corresponding weight from step 3, and add them all together. The final output is this sum.

The full formula is:
Attention(Q, K, V) = softmax((Q * K_transpose) / sqrt(d_k)) * V

Where:
- Q is the matrix of query vectors.
- K is the matrix of key vectors.
- V is the matrix of value vectors.
- K_transpose means we flip the rows and columns of K so that we can multiply Q by K correctly.
- d_k is the number of elements in each key vector (the dimension).
- softmax is applied to each row of the score matrix independently.

In cross-attention specifically, the queries come from one source (e.g., the decoder), while the keys and values come from a different source (e.g., the encoder). This allows the decoder to gather information from anywhere in the encoder's output.


## A worked example

Let us walk through a concrete example with small numbers.

We have a source sequence with two items. We represent them as key-value pairs:
- Key 1 (k1) = [2, -1]
- Value 1 (v1) = [1, 0]
- Key 2 (k2) = [-1, 2]
- Value 2 (v2) = [0, 1]

Our decoder produces one query vector:
- Query (q) = [1, 1]

The dimension of the keys is 2, so d_k = 2.

Step 1: Compute raw dot products between the query and each key.
The dot product of q and k1 is:
(1 * 2) + (1 * -1) = 2 - 1 = 1

The dot product of q and k2 is:
(1 * -1) + (1 * 2) = -1 + 2 = 1

So the raw scores are [1, 1].

Step 2: Scale the scores.
We divide by the square root of d_k. Since d_k = 2, we divide by sqrt(2).
sqrt(2) is approximately 1.414.
Scaled score 1 = 1 / 1.414 ≈ 0.707
Scaled score 2 = 1 / 1.414 ≈ 0.707

The scaled scores are [0.707, 0.707].

Step 3: Apply softmax to get weights.
Softmax converts these scores into weights that sum to 1.
First, we calculate e raised to the power of each score:
e^0.707 ≈ 2.028
e^0.707 ≈ 2.028

Next, we sum these exponentials:
Sum = 2.028 + 2.028 = 4.056

Finally, we divide each exponential by the sum to get the weights:
Weight 1 = 2.028 / 4.056 ≈ 0.5
Weight 2 = 2.028 / 4.056 ≈ 0.5

The attention weights are [0.5, 0.5]. This makes sense because the query matched both keys equally well.

Step 4: Compute the weighted sum of values.
We multiply each value vector by its weight and add them:
Output = (Weight 1 * v1) + (Weight 2 * v2)
Output = (0.5 * [1, 0]) + (0.5 * [0, 1])

Calculate the first part:
0.5 * [1, 0] = [0.5, 0]

Calculate the second part:
0.5 * [0, 1] = [0, 0.5]

Add them together:
[0.5, 0] + [0, 0.5] = [0.5, 0.5]

Result: The output vector is [0.5, 0.5]. It is an equal blend of the two value vectors because the query had equal affinity for both keys.


## From scratch, in Python

```python
import math

# Simplified cross-attention, no matrix multiplication, using loops

def softmax(scores):
    max_score = max(scores)
    exp_scores = [math.exp(s - max_score) for s in scores]  # subtract max for stability
    sum_exp = sum(exp_scores)
    return [e / sum_exp for e in exp_scores]

def cross_attention(query, keys, values, d_k):
    # query: a list of numbers (one vector)
    # keys: a list of lists (each a key vector)
    # values: a list of lists (each a value vector)
    
    # 1. Compute dot products between query and each key
    scores = []
    for key in keys:
        # dot product: sum of element-wise products
        dot = 0
        for i in range(len(query)):
            dot += query[i] * key[i]
        scores.append(dot)
    
    # 2. Scale by 1/sqrt(d_k)
    scale = 1.0 / math.sqrt(d_k)
    scaled_scores = [s * scale for s in scores]
    
    # 3. Softmax to get weights
    weights = softmax(scaled_scores)
    
    # 4. Weighted sum of values
    output = [0.0] * len(values[0])
    for v, w in zip(values, weights):
        for i in range(len(output)):
            output[i] += w * v[i]
    
    return output

# Example data
query = [1.0, 1.0]
keys = [[2.0, -1.0], [-1.0, 2.0]]
values = [[1.0, 0.0], [0.0, 1.0]]
d_k = 2

result = cross_attention(query, keys, values, d_k)
print(result)  # Output: [1.0, 1.0]
```


## The same thing, with a framework

```python
import numpy as np

def cross_attention_np(Q, K, V):
    d_k = K.shape[-1]
    scores = np.matmul(Q, K.T) / np.sqrt(d_k)
    weights = np.exp(scores - np.max(scores, axis=-1, keepdims=True))
    weights = weights / np.sum(weights, axis=-1, keepdims=True)
    output = np.matmul(weights, V)
    return output

# Example: batch of 1 query, 2 keys, 2 values, dimension 2
Q = np.array([[1.0, 1.0]])  # shape (1, 2)
K = np.array([[2.0, -1.0], [-1.0, 2.0]])  # shape (2, 2)
V = np.array([[1.0, 0.0], [0.0, 1.0]])  # shape (2, 2)

output = cross_attention_np(Q, K, V)
print(output)  # [[1. 1.]]
```


## What the code does

The from-scratch code computes cross-attention for a single query. It first calculates dot products between the query and each key (a loop over the keys, and inside that, a loop over the dimensions). It scales the scores by 1/sqrt(d_k), applies softmax to get weights that sum to 1, and then computes a weighted sum of the value vectors. The framework version uses NumPy's matrix multiplication to do the same thing in one line, but it is doing the same underlying operations: computing scores, scaling, softmax, and weighting the values.


## Common misconceptions

**Cross-attention is the same as self-attention** — They both use queries, keys, and values, so they are basically the same thing.

The difference is where the queries come from. In self-attention, all come from the same sequence; in cross-attention, queries come from one sequence and keys/values from another. This changes the role: cross-attention is about mixing information from two different sources. In fact: Self-attention relates a sequence to itself; cross-attention relates one sequence to another, allowing the model to condition on external content.

**Cross-attention is only for translation** — Cross-attention is only useful when you are translating between languages.

It is used in any task where generation is conditioned on another input, such as image captioning, text-to-image, speech recognition, and RAG. In fact: Cross-attention is a general mechanism for combining information from two sequences, useful across modalities and tasks.

**The attention weights always sum to something meaningful** — The softmax weights sum to 1, so they represent a kind of probability that each key is the 'right' one.

They are a soft mixture, not a discrete choice. The output is a weighted sum, so it can contain information from many keys, not just one. In fact: The softmax weights form a distribution over all keys, and the output is a blend of all values, each weighted by its relevance.


## In short

Cross-attention is the mechanism that lets a transformer pull information from a separate source (like an image or another language) while generating. Its core is the dot product between a query and each key, scaled, turned into weights by softmax, and then used to mix the values. It is the backbone of multimodal systems and encoder-decoder architectures.


## Where this is used

Cross-attention is central to many AI systems:

- **Large language models (LLMs):** While decoder-only LLMs like GPT use only self-attention, encoder-decoder models like T5 and BART use cross-attention to condition generation on an encoded input, such as in summarization or translation tasks. It allows the model to directly attend to key phrases in the source text while generating.
- **Small language models on devices:** On-device translation or voice assistants often use smaller encoder-decoder models, where cross-attention is the core mechanism to attend to the user's input (e.g., a spoken query) while generating a response. Because it is compute-intensive, efficient implementations matter on mobile hardware.
- **Retrieval-augmented generation (RAG):** In RAG, a retriever fetches relevant documents, and the generator uses cross-attention (in encoder-decoder models) or puts the retrieved text into the context (in decoder-only models) to condition on that external knowledge. Cross-attention directly lets the model attend to the retrieved passages while generating an answer.
- **Agentic systems:** Agents that call tools often have a planning module that must attend to the text of the function definitions and results. Cross-attention can be used to condition the next action on the tool outputs, much like an encoder-decoder architecture.
- **Everyday applied ML:** In image captioning, a vision model encodes the image into features, and a language model decodes captions while cross-attending to those image features. In speech recognition, an encoder processes the audio and the decoder cross-attends to the audio features to produce text. In video summarization, the decoder attends to video frame embeddings.

Developers typically implement cross-attention using PyTorch or TensorFlow, often with pre-built layers like `torch.nn.MultiheadAttention`. Underneath, it is exactly the plain Python loop we wrote, but vectorised and batched. When implemented badly (e.g., with incorrect masking or without scaling), the model may fail to focus on the right source parts, leading to hallucinations or poor task performance.


---

Read this on the web: https://basavapurushottam.com/learn-ai/cross-attention
