# Forward Propagation

Computing a network's output by passing inputs layer by layer through weights, biases and activations.

**Rests on:** Matrix Multiplication, Multi-Layer Perceptrons

**Leads to:** Activation Functions, Activation Patching, Autoencoders, CNN Visualization for Interpretation, Convolutional Layers, Diffusion Image Generation, Dropout, Feed-Forward Blocks, Fourier Analysis of Networks, GELU, Kolmogorov-Arnold Networks, Neural Tangent Kernel, Quantization, Sigmoid and Tanh, Softmax and Logits, Universal Approximation Theorems, Vanishing Gradients, Width vs Depth, Xavier Initialization

*Deep Learning · Deep Learning*


## Why learn this

Forward propagation is the part of a neural network that turns raw data into a prediction. You might know how a single artificial neuron works: it takes inputs, multiplies them by numbers called weights, adds a small constant called a bias, and then applies a squashing rule to keep the result within a specific range. Forward propagation is simply repeating this exact same step for every neuron in every layer of the network, moving from the input side to the output side. Once you understand this, you can build a three-layer network yourself, diagnose why your code crashes due to mismatched data shapes, and prepare for the next step: training the network. Forward propagation is also the engine that runs every time you use a trained model—like checking if a loan application should be approved or getting a response from a chatbot—to generate its answer.


## The intuition

Imagine a production line in a factory. Raw materials enter at one end, and finished goods exit at the other. Each workstation takes what it receives, performs a specific transformation, and passes the result to the next station. In a neural network, the raw materials are your input numbers, such as a loan applicant’s income, debt level, and years of employment. Each layer acts as a workstation. It takes the numbers from the previous layer, multiplies them by its own set of weights, adds its bias, and applies an activation function to produce a new set of numbers. These new numbers become the input for the next layer. This chain continues until the final layer produces the network’s prediction. The key insight is that the transformation at each layer is always the same small operation you already know: multiply, add, and squish. Forward propagation is just performing this operation repeatedly, layer after layer, from start to finish. Nothing more, nothing less.


## An analogy

Think of a relay race where each runner does not simply carry the baton but actively transforms it. At the start, you have a number, such as 5. The first runner follows a rule: 'Take the number, multiply it by 0.9, add 0.2, and if the result is positive, keep it; otherwise, change it to zero.' The runner hands this new value to the next runner, who applies a different set of rules. This continues until the final runner produces the race’s result—the prediction. In this analogy, each runner is a neuron in a layer, and their rules are determined by the weights, bias, and activation function. The baton changes value at every hand-off, encoding more meaningful information for the final decision. However, the analogy stops here: in a real relay, runners follow similar paths; in a network, each neuron has unique, learned rules, and the final output is often a probability between 0 and 1 or a set of scores representing different options, not just a race time.


## Definition

Forward propagation is the process of calculating a neural network's prediction by passing input data through each layer in sequence, where each layer applies a weighted sum plus a bias to its inputs, followed by a non-linear activation function, with the output of one layer becoming the input for the next.


## How it connects

This concept extends directly from the Artificial Neuron, which you know computes a weighted sum, adds a bias, and applies an activation. Forward propagation is simply performing that calculation across many neurons organized into layers. It also connects to Activation Functions, as you will apply rules like ReLU (which outputs zero for negative numbers) at each stage. The mathematical core involves computing the weighted sums of inputs, which you can think of as combining data points according to learned importance values.


## The mathematics

Forward propagation is the process of calculating the output of a neural network by passing data through its layers one at a time. To understand this, we must first define what happens inside a single layer.

A layer takes a list of numbers as input. Let us call this input vector **a_prev**. For example, if the previous layer had 3 neurons, **a_prev** is a list of 3 numbers: [a1, a2, a3].

Inside the current layer, there are several neurons. Each neuron performs the same type of calculation but uses different numbers called weights and biases. Let us focus on one specific neuron in this layer.

Step 1: The weighted sum (often called the pre-activation).
This neuron has a list of weights, which we will call **w**. This list must have the same length as the input vector **a_prev**. It also has a single number called the bias, which we will call **b**.

The neuron multiplies each input number by its corresponding weight and adds them all up. Then it adds the bias.
Mathematically, this is:
z = (a1 * w1) + (a2 * w2) + ... + (an * wn) + b

Here, **z** is the raw score for this neuron. It is a single number.

Step 2: The activation.
The raw score **z** is usually too large or too small to be useful directly. We pass it through an activation function to squash or transform it into a more manageable range. Let us call the activation function **f**. Common choices are:
- ReLU: If z is positive, f(z) = z. If z is negative, f(z) = 0.
- Sigmoid: f(z) = 1 / (1 + e^(-z)). This squashes any number into a range between 0 and 1.

The output of this neuron is:
a_new = f(z)

This **a_new** is the activation for this specific neuron. We repeat Steps 1 and 2 for every neuron in the current layer. The collection of all these new activations forms a new vector, which we call **a_curr**. This vector **a_curr** becomes the input **a_prev** for the next layer.

We repeat this process layer by layer until we reach the final layer. The output of the final layer is the network's prediction.

Matrix Notation Shortcut
When there are many neurons and inputs, writing out every multiplication is tedious. We can use matrices. A matrix is a rectangular table of numbers.
- Let **W** be the weight matrix for the current layer. Each row of **W** contains the weights for one neuron.
- Let **b** be the bias vector for the current layer. It contains the bias for each neuron.
- Let **a_prev** be the input vector from the previous layer.

The calculation for all neurons at once is:
z = W * a_prev + b
Here, '*' denotes matrix-vector multiplication. This operation computes the weighted sum for every neuron in parallel.
Then, we apply the activation function element-wise to get the new activations:
a_curr = f(z)

Summary of Symbols:
- **a_prev**: The vector of activations from the previous layer (or the input data for the first layer).
- **W**: The weight matrix for the current layer.
- **b**: The bias vector for the current layer.
- **z**: The pre-activation vector (raw scores) for the current layer.
- **f**: The activation function applied to each element of z.
- **a_curr**: The activated output vector of the current layer, which becomes a_prev for the next layer.


## A worked example

We will trace forward propagation through a small network with 2 inputs, 1 hidden layer containing 2 neurons, and 1 output neuron.

**Data Setup:**
- Input vector **a_prev** (from input layer): [0.5, 0.8]
- Hidden Layer Weights (**W_hidden**): A 2x2 matrix. Row 1 is weights for neuron 1, Row 2 is weights for neuron 2.
  W_hidden = [[0.1, 0.9],
              [0.4, 0.3]]
- Hidden Layer Biases (**b_hidden**): [0.2, -0.1]
- Activation function for hidden layer: ReLU (max(0, z))

- Output Layer Weights (**W_out**): A 1x2 matrix (since there is only 1 output neuron).
  W_out = [[0.6, 0.7]]
- Output Layer Bias (**b_out**): [0.3]
- Activation function for output layer: Sigmoid (1 / (1 + e^(-z)))

**Step 1: Calculate Pre-activation for Hidden Layer (z_hidden)**
We compute z = W * a_prev + b.

For Neuron 1 (first row of W):
z1 = (0.1 * 0.5) + (0.9 * 0.8) + 0.2
z1 = 0.05 + 0.72 + 0.2
z1 = 0.97

For Neuron 2 (second row of W):
z2 = (0.4 * 0.5) + (0.3 * 0.8) + (-0.1)
z2 = 0.20 + 0.24 - 0.1
z2 = 0.34

So, z_hidden = [0.97, 0.34]

**Step 2: Apply Activation to Hidden Layer (a_hidden)**
Apply ReLU to each element of z_hidden.
ReLU(0.97) = 0.97 (since 0.97 > 0)
ReLU(0.34) = 0.34 (since 0.34 > 0)

So, a_hidden = [0.97, 0.34]
This vector is now the input to the output layer.

**Step 3: Calculate Pre-activation for Output Layer (z_out)**
We compute z = W_out * a_hidden + b_out.

For the single output neuron:
z_out = (0.6 * 0.97) + (0.7 * 0.34) + 0.3
z_out = 0.582 + 0.238 + 0.3
z_out = 1.12

**Step 4: Apply Activation to Output Layer (a_output)**
Apply Sigmoid to z_out.
a_output = 1 / (1 + e^(-1.12))
e^(-1.12) is approximately 0.326
a_output = 1 / (1 + 0.326)
a_output = 1 / 1.326
a_output ≈ 0.754

**Result:**
The network's final prediction is approximately 0.754.


## From scratch, in Python

```python
import math

# Define inputs and weights
i1, i2 = 0.5, 0.8
w1_11, w1_12, w1_21, w1_22 = 0.1, 0.9, 0.4, 0.3
b1, b2 = 0.2, -0.1
w_out1, w_out2 = 0.6, 0.7
b_out = 0.3

# Hidden layer
h1_raw = (i1 * w1_11) + (i2 * w1_12) + b1
h2_raw = (i1 * w1_21) + (i2 * w1_22) + b2

# Activation (ReLU)
def relu(x):
    if x > 0:
        return x
    else:
        return 0

h1 = relu(h1_raw)
h2 = relu(h2_raw)

# Output layer
out_raw = (h1 * w_out1) + (h2 * w_out2) + b_out

# Sigmoid activation
def sigmoid(x):
    return 1 / (1 + math.exp(-x))

y_pred = sigmoid(out_raw)

# Print the result
print(y_pred)
```


## The same thing, with a framework

```python
import numpy as np

# Inputs as a vector
x = np.array([0.5, 0.8])

# Hidden layer weights as a matrix (rows = neurons, columns = inputs)
W1 = np.array([[0.1, 0.9], [0.4, 0.3]])
b1 = np.array([0.2, -0.1])

# Output layer weights as a vector
W2 = np.array([0.6, 0.7])
b2 = np.array([0.3])

# Forward pass
h = np.maximum(W1.dot(x) + b1, 0)  # ReLU activation
out = W2.dot(h) + b2
y_pred = 1 / (1 + np.exp(-out))  # Sigmoid

print(y_pred)
```


## What the code does

In the from_scratch code, we hard-code the weights and biases as individual variables. The hidden layer computes h1_raw and h2_raw using the weighted sum we already practiced. The ReLU function takes a number and returns it if positive, otherwise zero. Then we compute the output layer raw score out_raw using the hidden activations, and finally apply the sigmoid to get a prediction between 0 and 1. The code prints the prediction. In the NumPy version, we put the weights into a matrix W1 (each row corresponds to a neuron's weights) and the input into a vector x. The line W1.dot(x) computes the matrix-vector product, which is essentially doing all the dot products for all neurons at once. The + b1 adds the biases. np.maximum(..., 0) is the ReLU. Then the output layer is a simple dot product and sigmoid. NumPy does the same arithmetic we did by hand, but faster and with less explicit looping.


## Common misconceptions

**Forward propagation is only for prediction** — I only need forward propagation when I want to use the model to predict something, not during training.

During training, you first run forward propagation to get the prediction, then compute the loss, then run backpropagation to update weights. Without a forward pass, you have no output to compare against the label. In fact: Forward propagation is used both at inference time (making predictions after training) and at the start of every training step.

**The output of every layer is a single number** — Each layer just squashes everything down to one number, and the final layer is the only one with multiple outputs.

In most networks, hidden layers have many neurons, so their outputs are lists (vectors) of numbers. Only the final layer might have one output (for binary classification) or multiple (for multi-class). In fact: Hidden layers output vectors of numbers, one per neuron. The shape changes depending on the number of neurons in that layer.

**Weights are like percentages that sum to 1** — The weights for a neuron should add up to 1, because they represent how much each input contributes.

Weights are unconstrained real numbers. They can be negative or sum to anything; they represent both the direction and magnitude of influence. In fact: Weights are free parameters, learned from data. They do not need to sum to 1.

**Activation is optional** — I can skip the activation function; it just adds an extra step.

Without activations, stacking layers would just collapse into a single linear transformation, so a deep network would be no more powerful than a shallow one. Activations introduce nonlinearity, which is what allows networks to learn complex patterns. In fact: Every hidden layer should apply a nonlinear activation after the linear transformation to keep the network expressive.


## In short

Forward propagation is the process of computing a network's output by passing inputs through layers: each layer multiplies the input by a weight matrix, adds a bias, and applies an activation function. The output of one layer becomes the input to the next. This same process runs both during training (to produce a prediction for the loss) and during inference (to make predictions on new data). Mastering forward propagation means you understand the core computation of every neural network, from tiny ML models to massive LLMs. Remember: always keep an eye on shapes — dimension mismatches are the most common bug. Also, save your intermediate activations; you'll need them for backpropagation.


## Where this is used

Forward propagation is the fundamental computation in every neural network. In large language models (LLMs) like GPT-4, forward propagation is performed for each token in a sequence: the token embeddings are passed through dozens of transformer layers, each involving matrix multiplications and activations (with attention mechanisms in between). When you use a chatbot, each word it generates requires multiple forward passes — one for each new token. In small models running on phones (TinyML), forward propagation is what runs on the device, often with quantized weights (using fewer bits per number) to save memory and battery. Every time a smartphone's voice assistant spots a 'Hey Siri' trigger word, that's a small neural network doing forward propagation in real time. In retrieval-augmented generation (RAG), forward propagation is used to compute embeddings for documents and queries — the network turns text into a vector of numbers, and then cosine similarity between these vectors is computed to find relevant documents. For agentic systems that plan and call tools, forward propagation is used to decide which tool to call: the language model's output layer gives probabilities for each possible next token; the agent might sample from those to choose a call. In everyday applied ML — credit risk, recommendation systems, computer vision, forecasting, fraud detection — forward propagation is the core inference step. A developer might use a framework like PyTorch and write something like:

```python
import torch
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.fc1 = nn.Linear(2, 2)  # input size, hidden size
        self.relu = nn.ReLU()
        self.fc2 = nn.Linear(2, 1)
        self.sigmoid = nn.Sigmoid()
    def forward(self, x):
        x = self.fc1(x)
        x = self.relu(x)
        x = self.fc2(x)
        x = self.sigmoid(x)
        return x

net = Net()
pred = net(torch.tensor([0.5, 0.8]))
```

When done badly, common bugs include shape mismatches (e.g., feeding a batch of shape (32, 2) to a layer expecting (2, 2)), forgetting to set the model to evaluation mode (which affects dropout and batch norm), or using the wrong activation function causing outputs to be out of range. Shape mismatches are the most common error — you'll see 'mat1 and mat2 shapes cannot be multiplied' or similar. This is why understanding the dimensions is crucial: each layer expects inputs matching its weight matrix's columns. In practice, developers rely on the shape error messages to debug, but the core is that forward propagation must be correct for training to work.


---

Read this on the web: https://basavapurushottam.com/learn-ai/forward-propagation
