# Multi-Layer Perceptrons

Fully connected feedforward networks — the baseline architecture where every unit connects to every unit in the next layer.

**Rests on:** Perceptron and Decision Boundaries

**Leads to:** Feed-Forward Blocks, Forward Propagation, Foundation Models, LSTM Networks

*Deep Learning · Deep Learning*


## Why learn this

This is the basic building block of deep learning. Once you understand a multi-layer perceptron, you understand the structure behind every modern AI system — including the kind that writes text like this. It also explains why this approach works well for standard data tables, like spreadsheets or financial records, where items don't naturally form images or sentences. You will see how adding more layers allows the system to handle increasingly complex tasks.


## The intuition

A multi-layer perceptron is a stack of processing units called neurons. You may already know that a single neuron takes several inputs, multiplies each by a number representing its importance (a weight), adds a personal adjustment (a bias), and then applies a rule to squash the result into a final value (an activation function). A multi-layer perceptron arranges these neurons in layers. The first layer simply holds your input data; for example, if you are predicting loan risk, the inputs might be 'income: 50,000', 'debt: 5,000', and 'tenure: 10'. These values pass to the next layer, where every neuron connects to every input. Each neuron in this first hidden layer calculates its own weighted sum — for instance, multiplying income by 0.8, debt by -0.3, and tenure by 0.5, then adding a bias of -2.0 — resulting in a specific number. It then applies an activation function to this result. This new value becomes the input for the next layer. The process repeats: each neuron in the second hidden layer looks at all outputs from the first layer, performs its own weighted sum and activation, and passes the result forward. This continues until the final output layer produces a single number or category. The key is that the system starts with random weights, producing nonsense results. Through training, it adjusts these weights so that inputs like 'high income' and 'low debt' gradually come to contribute positively to a 'good risk' prediction, while 'recent defaults' contribute negatively. The network learns to combine simple features into complex patterns automatically.


## An analogy

Think of a manufacturing assembly line. Raw materials enter at one end, and a finished product exits at the other. Each workstation takes the output of the previous station, applies a specific operation, and passes the result forward. In a multi-layer perceptron, each neuron is like a worker on this line. The worker receives inputs from all workers in the previous stage, weighs their contributions based on importance, adds a personal bias, and then decides how strongly to pass their output to the next stage. Initially, the workers are untrained and produce garbage, but as the system learns, they adjust their weights so the final product is correct. The main difference from a real factory is that in an MLP, every worker performs the exact same simple calculation; the complexity comes from having many layers of these identical workers chained together, rather than each station being custom-built for a unique task.


## Definition

A multi-layer perceptron is a system that processes data by passing it through a series of layers, where each layer transforms its inputs into new values using weighted combinations and non-linear rules, moving in one direction from input to output.


## How it connects

This builds directly on the artificial neuron: each unit here is exactly that single processing element. You also need to understand how we combine numbers: each neuron’s computation involves multiplying corresponding pairs of numbers from two lists (inputs and weights) and adding those products together, then adding a bias term. The activation functions you have studied are what make stacking layers powerful; without them, the entire system would behave like a single simple calculation, losing the ability to learn complex patterns. This concept leads into training methods: the process of adjusting these weights to improve accuracy is what allows the network to learn from examples.


## The mathematics

A Multi-Layer Perceptron (MLP) is a chain of simple calculations. Each step in the chain takes numbers from the previous step, mixes them together using weights and biases, and then applies a rule to decide which numbers pass through.

Think of the MLP as a series of rooms. You start with an input vector, which is just a list of numbers. Let us call this list x. For example, if you are predicting house prices based on size and age, your input list might be [2000, 15], representing 2000 square feet and 15 years old.

The first room in the chain is called the first hidden layer. Inside this room, there are several neurons. A neuron is a single calculator. Every neuron in this room looks at every number in your input list x. It multiplies each input number by a specific weight (a number that tells it how important that input is) and then adds a bias (a constant offset). 

Let us say the first hidden layer has two neurons. We will call them Neuron 1 and Neuron 2.

For Neuron 1, let us assign weights w_11 and w_12 to the two inputs in x, and a bias b_1. The calculation inside Neuron 1 is:

z_1 = (w_11 * x_1) + (w_12 * x_2) + b_1

Here, z_1 is the raw result of the mixing. It is not yet the final output of the neuron because we need to apply an activation function. An activation function is a rule that squashes or changes the number z_1. A common rule is ReLU (Rectified Linear Unit). The ReLU rule says: if the number is negative, make it zero. If the number is positive, leave it alone.

a_1 = max(0, z_1)

Here, a_1 is the activation of Neuron 1. This is the number that leaves Neuron 1 and goes to the next room.

Neuron 2 does the exact same type of calculation but with its own unique weights w_21, w_22 and bias b_2:

z_2 = (w_21 * x_1) + (w_22 * x_2) + b_2
a_2 = max(0, z_2)

Now the first hidden layer is done. The outputs a_1 and a_2 form a new list of numbers. This new list becomes the input for the next room, which might be another hidden layer or the final output layer.

If there is a second hidden layer, it takes [a_1, a_2] as its input. It has its own weights and biases. Let us say it has one neuron. The calculation is:

z_3 = (w_31 * a_1) + (w_32 * a_2) + b_3
a_3 = max(0, z_3)

This process repeats for every layer in the MLP. In deep learning terminology, the term "linear layer" is often used to describe the mixing step (multiplying by weights and adding bias). Technically, this is an affine transformation because it includes the addition of a constant bias, not just pure multiplication. However, in practice, people call it linear.

The final room is the output layer. If you are doing regression (predicting a continuous number like price), you might skip the activation function and just use the raw sum z_L as the output. If you are doing classification (picking a category), you might apply a different activation, like sigmoid, to squash the result between 0 and 1.

To summarize the notation used in textbooks:
- x is the input vector.
- W^[l] is the matrix of weights for layer l. A matrix is just a grid of numbers. Each row corresponds to one neuron in that layer, and each column corresponds to one input from the previous layer.
- b^[l] is the vector of biases for layer l. It has one number for each neuron.
- z^[l] is the vector of raw sums (weighted inputs plus bias) for layer l.
- a^[l] is the vector of activations (the outputs after applying the activation function) for layer l.

The formula z^[l] = W^[l] * a^[l-1] + b^[l] is just a compact way of writing "do the multiplication and addition for every neuron in parallel." The matrix multiplication handles all the neurons at once. For example, if layer l has 3 neurons and the previous layer had 2 outputs, W^[l] is a 3x2 grid. Multiplying this grid by the 2-element vector from the previous layer gives you a 3-element vector of raw sums z^[l].


## A worked example

Let us walk through a specific MLP with real numbers.

Setup:
- Input: Two numbers, x1 = 0.5 and x2 = 0.8.
- Hidden Layer: One layer with two neurons (Neuron 1 and Neuron 2).
- Output Layer: One neuron that produces the final result.
- Activation Function for Hidden Layer: ReLU (if the number is negative, output 0; otherwise, output the number itself).
- Activation Function for Output Layer: None (we just take the raw sum for this regression example).

Step 1: Calculate the raw sum for Neuron 1 in the hidden layer.
Neuron 1 has weights w_11 = 0.2 and w_12 = 0.4, and bias b_1 = 0.1.
It multiplies each input by its weight:
0.2 * 0.5 = 0.1
0.4 * 0.8 = 0.32
Then it adds the products together:
0.1 + 0.32 = 0.42
Then it adds the bias:
0.42 + 0.1 = 0.52
So, z_1 = 0.52.

Step 2: Apply the activation function to Neuron 1.
The ReLU rule says: if z_1 is negative, make it 0. If positive, keep it.
Since 0.52 is positive, a_1 = 0.52.
This is the output of Neuron 1.

Step 3: Calculate the raw sum for Neuron 2 in the hidden layer.
Neuron 2 has weights w_21 = -0.3 and w_22 = 0.6, and bias b_2 = -0.2.
It multiplies each input by its weight:
-0.3 * 0.5 = -0.15
0.6 * 0.8 = 0.48
Then it adds the products together:
-0.15 + 0.48 = 0.33
Then it adds the bias:
0.33 + (-0.2) = 0.13
So, z_2 = 0.13.

Step 4: Apply the activation function to Neuron 2.
The ReLU rule says: if z_2 is negative, make it 0. If positive, keep it.
Since 0.13 is positive, a_2 = 0.13.
This is the output of Neuron 2.

Step 5: Pass the hidden layer outputs to the Output Layer.
The output layer has one neuron. It takes a_1 and a_2 as its inputs. Let us call them input_1 and input_2 for clarity, so input_1 = 0.52 and input_2 = 0.13.
The output neuron has weights v_1 = 0.7 and v_2 = -0.5, and bias b_out = 0.0.
It multiplies each input by its weight:
0.7 * 0.52 = 0.364
-0.5 * 0.13 = -0.065
Then it adds the products together:
0.364 + (-0.065) = 0.299
Then it adds the bias:
0.299 + 0.0 = 0.299
So, z_out = 0.299.

Step 6: Final Result.
Since this is a regression task, we do not apply an activation function to the output layer in this example. The final output of the MLP is 0.299.

If this were a classification task (e.g., predicting if a digit is a 7 or not), we would take z_out = 0.299 and pass it through a sigmoid function, which squashes any number into a range between 0 and 1, giving us a probability.


## From scratch, in Python

```python
import math

# A single neuron: takes a list of inputs and a list of weights (same length), and a bias
# This is exactly what you've seen before: dot product + bias, then activation.
def neuron(inputs, weights, bias):
    weighted_sum = 0
    for i in range(len(inputs)):
        weighted_sum += inputs[i] * weights[i]
    weighted_sum += bias
    # Apply ReLU activation (max with 0)
    if weighted_sum > 0:
        return weighted_sum
    else:
        return 0

# A layer: a list of neurons. Each neuron has its own weights and bias.
# The layer takes a list of inputs and returns a list of outputs.
def layer(inputs, layer_weights, layer_biases):
    outputs = []
    for neuron_idx in range(len(layer_biases)):
        # Get the weights for this neuron
        weights = layer_weights[neuron_idx]
        bias = layer_biases[neuron_idx]
        # Compute this neuron's output
        out = neuron(inputs, weights, bias)
        outputs.append(out)
    return outputs

# An MLP: a list of layers. Each element is a tuple: (weights, biases)
# We define an MLP with 2 inputs, a hidden layer of 3 neurons, and an output layer of 1 neuron.
# Hidden layer weights: 3 neurons, each with 2 weights
hidden_weights = [
    [0.2, -0.4],
    [0.7, 0.1],
    [-0.3, 0.6]
]
hidden_biases = [0.1, -0.2, 0.4]

# Output layer: 1 neuron, with 3 inputs (from hidden layer)
output_weights = [[0.5, -0.2, 0.9]]
output_biases = [0.0]

# The full MLP is just these layers combined
mlp = [
    (hidden_weights, hidden_biases),
    (output_weights, output_biases)
]

# Forward pass: process data through all layers
inputs = [0.5, 0.8]
current = inputs
for weights, biases in mlp:
    current = layer(current, weights, biases)

# The final output is a list with one element
output = current[0]
print(output)  # Let's compute this in the walkthrough
```


## What the code does

The from-scratch code has three functions. First, `neuron(inputs, weights, bias)` takes a list of numbers (the inputs) and a list of numbers of the same length (the weights). It loops over each pair, multiplies them, and adds to a running total; after the loop it adds the bias. Then it applies the ReLU activation: if the weighted sum is positive, keep it; otherwise set to zero. This is exactly the single neuron from the previous page. Second, `layer(inputs, layer_weights, layer_biases)` takes a list of inputs and two lists-of-lists: `layer_weights` contains, for each neuron, a list of its weights; `layer_biases` is a simple list, one bias per neuron. The function creates an empty list for outputs, then loops over each neuron index, pulls out that neuron's weights and bias, calls `neuron()` with them, and appends the result. This builds up the layer's output list. Third, the main script defines a small MLP: 2 inputs, a hidden layer of 3 neurons, then an output layer of 1 neuron. The `mlp` variable is a list of tuples, each tuple being (weights, biases) for that layer. The forward pass just sets `current = inputs`, then for each layer tuple, calls `layer()` on `current` and replaces `current` with the result. Finally it prints `current[0]` because the output is a list with one number. The framework version does the same thing in three lines: `nn.Linear(2,3)` creates the hidden layer (weight matrix of size 3×2, bias vector of size 3), `nn.ReLU()` applies the same activation, and `nn.Linear(3,1)` is the output layer. Underneath, PyTorch's `Linear` just does the same multiplication and addition you coded by hand.


## Common misconceptions

**More layers always means better** — If I just add more layers, my model will keep getting better.

Beyond a point, deeper networks become harder to train due to vanishing/exploding gradients, and they overfit the training data. Also, for small datasets, a deep model will just memorize noise. In fact: Depth is useful up to a point, but you also need data, regularization, and careful initialization. In practice, a wide but shallow MLP can often beat a deep one on small tabular data.

**MLP is magic and can learn anything** — An MLP can learn any pattern, so I don't need to worry about feature engineering.

It's a universal approximator only in theory with enough neurons and appropriate training. In practice, with finite data and computation, it needs good features, especially for tabular data. And it doesn't automatically use spatial or sequential structure—that's why CNNs and RNNs/transformers exist. In fact: An MLP is a powerful function approximator, but it ignores structural information. For images, use CNNs; for sequences, use transformers or RNNs. For tabular data, it's a solid choice, but you still need to scale features and possibly do feature engineering.

**MLP weights can be negative? That doesn't make sense** — Weights should always be positive because that's how much a feature matters.

Weights can be negative; they represent positive or negative influence. This is exactly like in the loan example: debt has a negative weight, reducing the score. In fact: Weights are real numbers, positive or negative. A negative weight means the feature pushes the output down.

**The output of the output layer is always a probability** — The final output of an MLP is always a probability between 0 and 1.

Only if you apply a sigmoid (for binary classification) or softmax (for multi-class). For regression, the output is a raw value. In fact: The output layer's activation determines the interpretation. Common choices: none (regression), sigmoid (binary classification), softmax (multi-class).

**MLP is the same as a deep learning model** — An MLP is what people mean by deep learning.

Deep learning encompasses many architectures. An MLP is the simplest feedforward network, but 'deep learning' includes CNNs, RNNs, transformers, etc. An MLP with many hidden layers is one type of deep network, but not the whole field. In fact: An MLP is a specific architecture (fully connected feedforward). Deep learning is the broader discipline of training multi-layer networks, including but not limited to MLPs.


## In short

A multi-layer perceptron is a stack of neuron layers: each neuron computes a weighted sum of its inputs plus a bias, then applies a nonlinear activation. Layers are fully connected — every unit connects to every unit in the next. The whole thing is trained end-to-end by gradient descent (which you'll learn in backpropagation). MLPs are universal approximators in theory but parameter-hungry in practice, and they ignore structure — so for images and sequences, use specialized architectures, but for tabular data, MLPs are a great default. Remember: depth buys representational power, but only with nonlinear activations and careful training.


## Where this is used

MLPs are the backbone of basic neural networks, but you rarely encounter them in their pure form in modern deep learning—because they ignore structure. However, they appear as building blocks everywhere. In **large language models**, the transformer architecture uses MLPs as the feedforward layers after each attention block. Each token's representation goes through a two-layer MLP (typically with an expansion to 4× the size, then back down). That MLP does much of the heavy lifting: it's where the model processes and transforms information beyond just mixing tokens via attention. When you hear about 'parameter count', a large fraction of those are in these MLP layers. A developer building an LLM would write `nn.Sequential(nn.Linear(d_model, 4*d_model), nn.GELU(), nn.Linear(4*d_model, d_model))` — that's the MLP. In **small models running on phones or laptops**, the same structure appears but with fewer parameters. The MLP is often the most compute-intensive part, so developers use tricks like quantization or factorization to speed these layers up. In **retrieval-augmented generation**, the embedding model that converts text into vectors is often an MLP-based network (or a transformer, which itself contains MLPs). The MLP's output vectors are what get compared for similarity. In **agentic systems**, they use language models which contain MLPs, so indirectly it's there. But more directly, many tool-use or planning components are simple classifiers or regressors built as MLPs over extracted features. In **everyday applied machine learning**, MLPs shine for tabular data: fraud detection on transaction records, credit scoring on customer data, churn prediction, sensor fault detection. There is no natural spatial or sequential structure, so a fully connected network is the right default. A data scientist would take a CSV of features, normalize the columns, and feed them into a `sklearn.neural_network.MLPClassifier` or a PyTorch `nn.Sequential` with a couple of hidden layers. What breaks when done badly: overfitting if the model is too wide, vanishing gradients if it's too deep (more on that with backpropagation), and poor performance if the features are not scaled (MLPs assume inputs are on a similar scale — typically you standardize to mean 0, variance 1). Also, MLPs ignore any spatial or sequential structure, so using them on images or audio directly is a mistake — that's why CNNs and transformers were invented.


---

Read this on the web: https://basavapurushottam.com/learn-ai/multi-layer-perceptrons
