# Feed-Forward Blocks

The position-wise multilayer perceptron applied after attention in each transformer layer, typically expanding then contracting the dimension.

**Rests on:** Forward Propagation, Multi-Layer Perceptrons

**Leads to:** Foundation Models, GPT Architecture, Vision Encoders

*Deep Learning · Transformers*


## Why learn this

Understanding the feed-forward block completes your picture of a transformer layer. It is the part that holds most of the model's parameters and where a lot of the model's factual knowledge is stored. You will need it to comprehend how GPT models generate text, why scaling up models works, and how techniques like retrieval-augmented generation hook into the model. Without this, you have only half of the transformer story.


## The intuition

Imagine a transformer layer as a factory assembly line. Self-attention is the section where workers share information with each other—each worker looks at what everyone else is doing and adjusts their part accordingly. After that, each worker takes their updated part and runs it through a private 'processing room'—a little machine that transforms it in a specific way. This processing room is the feed-forward block. It does not communicate with the other workers; it works on its own piece, applying a learned transformation. The transformation usually expands the piece into a larger intermediate form and then contracts it back, allowing the model to consider more complex features before the next attention round.


## An analogy

Think of a chef preparing a multi-step dish. The 'attention' step is like tasting all the ingredients together to decide how they might combine. The 'feed-forward' step is like marinating each ingredient separately in a secret spice mix—each one goes through its own transformation without looking at the others. The spice mix has a lot of ingredients (the expanded dimension), and the final marinade is a reduced, concentrated version (the contracted dimension). This analogy works well because it shows that attention mixes information across positions, while feed-forward processes each position independently and identically—you use the same spice mix on every ingredient. Where it breaks down: in a real transformer, the feed-forward block is not a 'secret' spice mix; it is a learned function that is applied to every position equally, and it is not pre-defined—it is adjusted during training to help the model make better predictions.


## Definition

A feed-forward block is a small neural network applied independently to each position in a sequence, consisting of two linear transformations with a nonlinear activation in between, typically expanding the dimension and then contracting it back, which allows the model to learn complex features at each position.


## How it connects

This concept builds directly on the transformer architecture. After self-attention, each position passes through a feed-forward block. You have notes on Multi-Head Attention, Causal Attention Masking, and Positional Encoding—these are the other components of a transformer layer. Together, they form the repeating block that is stacked to build models like GPT. This block is also crucial for the GPT Architecture that you have notes on, as the decoder blocks are exactly composed of masked self-attention and feed-forward layers.


## The mathematics

A feed-forward block in a Transformer is a small machine that changes one list of numbers into another list of numbers, while keeping the length of the list exactly the same. It does this by passing the list through two stages of calculation.

The first stage expands the information. Imagine your input list has 4 numbers. The block multiplies these 4 numbers by a set of weights to create a new, longer list—say, with 16 numbers. This is like taking a small piece of data and stretching it out so there is more room for detail.

The second stage applies a filter. In this expanded list, any number that is negative is turned into zero. Numbers that are already positive stay exactly as they are. This step removes the 'negative' parts of the signal, which helps the model focus on what is present rather than what is absent.

The third stage shrinks the information back down. The long list of 16 numbers is multiplied by another set of weights to produce a final list of 4 numbers. This brings the data back to its original size so it can be passed to the next part of the Transformer.

Mathematically, if we call the input list x, the process looks like this:

1. Multiply x by a weight matrix W1 and add a bias vector b1. This creates an intermediate list.
2. Apply the ReLU filter: change every negative number in that intermediate list to zero.
3. Multiply the result by a second weight matrix W2 and add a second bias vector b2.

The formula is written as:
FFN(x) = W2 * ReLU(W1 * x + b1) + b2

Here is what each symbol means:
- FFN(x): The final output list, read 'Feed Forward Network of x'.
- x: The input list of numbers.
- W1: The first weight matrix. It determines how the input numbers are combined to create the expanded list.
- b1: The first bias vector. These are fixed numbers added to the expanded list before filtering.
- ReLU: The filter function. It stands for 'Rectified Linear Unit'. It keeps positive numbers as they are and turns negative numbers into zero.
- W2: The second weight matrix. It combines the filtered, expanded numbers back down to the original size.
- b2: The second bias vector. These are fixed numbers added at the very end.


## A worked example

Let us walk through a tiny example with real numbers. We will use very small lists so you can see every step.

Suppose our input list x has 2 numbers:
x = [1, 2]

The feed-forward block will expand this to 4 numbers, filter them, and then shrink them back to 2 numbers.

Step 1: Expand the list using W1 and b1.
We need a weight matrix W1 that turns 2 inputs into 4 outputs. Let us define W1 as a table with 2 rows and 4 columns:
W1 = [[0.1, 0.2, 0.3, 0.4],
      [0.5, 0.6, 0.7, 0.8]]

And a bias vector b1 with 4 numbers:
b1 = [0.1, -0.2, 0.3, -0.4]

To get the first number of the expanded list, we take the first row of W1 and multiply it by x, then add the first number of b1.
Calculation: (1 * 0.1) + (2 * 0.2) + 0.1 = 0.1 + 0.4 + 0.1 = 0.6

To get the second number, we take the second row of W1 and multiply it by x, then add the second number of b1.
Calculation: (1 * 0.5) + (2 * 0.6) - 0.2 = 0.5 + 1.2 - 0.2 = 1.5

We do this for all 4 positions. The intermediate expanded list is:
[0.6, 1.5, ...]
(For the sake of this example, let us assume the other two numbers in the expanded list are negative, say [-0.5, -0.2]. So the full intermediate list before filtering is [0.6, 1.5, -0.5, -0.2].)

Step 2: Apply the ReLU filter.
We look at each number in the intermediate list [0.6, 1.5, -0.5, -0.2].
- 0.6 is positive, so it stays 0.6.
- 1.5 is positive, so it stays 1.5.
- -0.5 is negative, so it becomes 0.
- -0.2 is negative, so it becomes 0.

The filtered list is now:
[0.6, 1.5, 0, 0]

Step 3: Shrink the list back to 2 numbers using W2 and b2.
We need a weight matrix W2 that turns 4 inputs into 2 outputs. Let us define W2 as a table with 4 rows and 2 columns:
W2 = [[1, 0],
      [0, 1],
      [1, 1],
      [1, 1]]

And a bias vector b2 with 2 numbers:
b2 = [0, 0]

To get the first number of the final output, we multiply the filtered list by the first column of W2 and add the first number of b2.
Calculation: (0.6 * 1) + (1.5 * 0) + (0 * 1) + (0 * 1) + 0 = 0.6

To get the second number, we multiply the filtered list by the second column of W2 and add the second number of b2.
Calculation: (0.6 * 0) + (1.5 * 1) + (0 * 1) + (0 * 1) + 0 = 1.5

Result: The final output list is [0.6, 1.5].

Notice that the first number came mostly from the first part of the input, and the second number came mostly from the second part. The ReLU filter removed the negative contributions, which simplified the calculation.


## From scratch, in Python

```python
import random

def relu(x):
    return max(0, x)

def feed_forward(x, W1, b1, W2, b2):
    # x: list of floats (input vector)
    # W1: list of lists (first weight matrix, size len(x) x length of b1)
    # b1: list of floats (bias vector)
    # W2: list of lists (second weight matrix, size len(b1) x length of b2)
    # b2: list of floats (bias vector)
    
    # First linear layer: W1 * x + b1
    intermediate = []
    for i in range(len(b1)):
        sum = 0
        for j in range(len(x)):
            sum += W1[i][j] * x[j]
        sum += b1[i]
        intermediate.append(sum)
    
    # Apply ReLU
    intermediate = [relu(val) for val in intermediate]
    
    # Second linear layer: W2 * intermediate + b2
    output = []
    for i in range(len(b2)):
        sum = 0
        for j in range(len(intermediate)):
            sum += W2[i][j] * intermediate[j]
        sum += b2[i]
        output.append(sum)
    
    return output

# Example usage
x = [1.0, 2.0]
W1 = [[0.1, 0.2], [0.3, 0.4], [0.5, 0.6], [0.7, 0.8]]  # 4x2
b1 = [0.1, -0.2, 0.3, -0.4]
W2 = [[0.1, 0.2, 0.3, 0.4], [0.5, 0.6, 0.7, 0.8]]  # 2x4
b2 = [0.0, 0.1]

result = feed_forward(x, W1, b1, W2, b2)
print(result)
```


## The same thing, with a framework

```python
import torch
import torch.nn as nn

# Define the feed-forward block
ffn = nn.Sequential(
    nn.Linear(2, 4),  # expand to 4 dimensions
    nn.ReLU(),
    nn.Linear(4, 2)   # contract back to 2 dimensions
)

# Example input (batch size 1, sequence length 1, feature size 2)
x = torch.tensor([[1.0, 2.0]])
output = ffn(x)
print(output)
```


## What the code does

The code defines a feed-forward block with two linear layers and a ReLU activation in between. It starts by computing the dot product for each row of W1 with x, adding the bias, then applying ReLU element-wise. Then it does the same with W2 to get the final output. The nested loops implement matrix multiplication manually.


## Common misconceptions

**Feed-forward blocks are just a linear layer** — The feed-forward block is just a linear transformation, so it doesn't add much complexity.

The nonlinear activation (like ReLU) in between makes it a universal approximator. Without it, stacking linear layers would just be a linear function. In fact: The nonlinearity is what allows the block to learn complex patterns beyond simple linear relationships.

**All parameters are in attention** — Since attention is the 'intelligent' part, most of the model's parameters must be there.

In fact, the feed-forward blocks often contain two-thirds of the parameters. In fact: The bulk of the model's capacity is in the feed-forward layers, which is where a lot of knowledge is stored.

**Feed-forward blocks mix information across positions** — The feed-forward block should also let tokens talk to each other.

It is applied identically to each position independently; there is no mixing across positions. In fact: Attention is the only part that mixes information across positions. Feed-forward works on each position separately.

**Bigger feed-forward always helps** — If I just make the feed-forward block larger, the model will be better.

A larger intermediate size increases the number of parameters and can lead to overfitting without enough data or regularization. In fact: The size must match the task's complexity and the available data; bigger is not always better.


## In short

The feed-forward block is a simple two-layer neural network with a nonlinearity in between, applied independently to each position. It is the part of a transformer that contains the majority of the parameters and where much of the model's factual knowledge lives. Along with self-attention, it forms the core of every transformer layer.


## Where this is used

Feed-forward blocks are everywhere in modern AI. In large language models (LLMs) like GPT, each transformer layer has a feed-forward block after the attention. They hold the majority of the model's parameters—often two-thirds of them—and are crucial for storing factual knowledge. For example, the model's ability to recall that 'the capital of France is Paris' is largely attributed to the feed-forward layers. When a language model generates text, every token passes through these blocks at every layer, and the transformations they apply are what allow the model to build rich internal representations. In small models that run on phones or laptops, the feed-forward blocks are also there, but they are smaller. However, the same structure is used, and they are a major part of the model size, so techniques like knowledge distillation or pruning often target these blocks to make models run faster on edge devices. In retrieval-augmented generation (RAG), the feed-forward blocks do not directly interact with the retrieved documents; instead, the retrieved text is passed as input to the model, and the feed-forward blocks process it as part of the sequence. The model's ability to integrate the retrieved information with the prompt relies on these blocks. In agentic systems that plan and call tools, the language model—which contains these feed-forward blocks—is the brain that decides which tool to call and how to interpret the results. The model's capacity to reason about actions is heavily dependent on the feed-forward computations. In everyday applied machine learning, such as in vision models (e.g., CNNs) and forecasting models, similar feed-forward layers are used in the final classification layers or in transformer-based architectures that process tabular data. A developer working with transformers will rarely manipulate the feed-forward block directly—they will use a high-level library like PyTorch and define a transformer encoder layer, which automatically includes the feed-forward block. What they will do is tune the 'intermediate size' (the expansion dimension) and the number of layers. If done badly, for instance setting the intermediate size too large, the model may overfit or become too slow; if too small, it may underfit and not capture necessary complexity.


---

Read this on the web: https://basavapurushottam.com/learn-ai/feed-forward-blocks
