# Vanishing Gradients

Gradients shrinking multiplicatively as they propagate backwards, so early layers receive almost no learning signal.

**Rests on:** Forward Propagation, Upstream Gradients

*Deep Learning · Deep Learning*


## Why learn this

This concept explains why early neural networks couldn't go deep, and why modern AI can. Understanding vanishing gradients unlocks the logic behind every big network you've heard of — from LLMs like ChatGPT to vision models. When you finish this page, you'll know why ReLU is the default activation, why residual connections exist, and why training a 100-layer network is even possible. This is the bridge from 'neurons' to 'deep learning' — without it, the rest of the journey (backpropagation, transformers, large language models) rests on a mystery.


## The intuition

Imagine a long line of people passing a whispered message from one end to the other. Each person hears what the previous person said, makes a small change, and passes it on. If each person only passes on 90% of what they heard, the message at the far end is only 0.9 × 0.9 × 0.9... — after 10 people, it's about 35% of the original; after 30 people, it's 4%; after 100, it's nearly zero. The message doesn't disappear in one step; it shrinks a little each time, and the multiplication of many small shrinkages destroys it. That's exactly what happens in a deep neural network when the learning signal (the 'message' that tells each layer how to adjust) travels backwards from the output to the early layers. Each layer multiplies the signal by a number (its gradient with respect to the weights), and if those numbers are consistently less than 1, the product shrinks exponentially with depth. The early layers, which are supposed to learn the basic features (edges in images, characters in text), receive almost no signal, so they never learn. The network's first layers stay random, and the whole model underperforms. This is the vanishing gradient problem — gradients (the learning signal) vanishing (shrinking to nothing) as they propagate (travel backwards) through the network.


## An analogy

Think of a chain of warehouses, each one passing a box of goods to the next, from the first warehouse (near the input) to the last (near the output). The last warehouse sends a signal back: 'You need to change what you send me.' That signal has to travel back through every warehouse, and each warehouse is a bit forgetful — it only passes on a fraction of the message. If each warehouse passes on 0.7 (70%) of what it received, then the message that reaches the first warehouse is 0.7 multiplied by itself as many times as there are warehouses. With 10 warehouses, that's 0.7^10 ≈ 0.028 — about 3% of the original message. With 30, it's 0.00002 — effectively nothing. The first warehouse, which is the most important because it starts the whole chain, gets almost no instruction. It keeps doing whatever it was doing, and the whole chain can't improve. The fix isn't to make each warehouse less forgetful (that's one approach, but hard to guarantee); it's to add a direct shortcut path so the message doesn't have to travel through every warehouse — or to make each warehouse pass on the full message by design (like using an activation function that doesn't shrink). The analogy breaks down because in a real network, each layer's 'forgetfulness' isn't a single fixed number; it depends on the current weights and the input. But the core idea — multiplicative decay over many steps — is exactly what happens.


## Definition

Vanishing gradients is the problem where the learning signal (gradient) becomes exponentially smaller as it propagates backwards through many layers, so the early layers receive almost no update and fail to learn.


## How it connects

You've already met the artificial neuron (inputs, weights, bias, activation) — that's the building block. This page is about what happens when you stack millions of those neurons into a deep network and try to train it. It directly connects to Activation Functions (you'll see why sigmoid, which squashes everything into a small range, causes the problem, and why ReLU, which doesn't, is the default fix). It also sets the stage for Backpropagation (the algorithm that computes these gradients) and Residual Connections (the famous 'shortcut' fix). In the Learner's Library, this sits under Deep Learning — it's a core reason why depth was so hard to achieve, and why modern deep learning works.


## The mathematics

To understand vanishing gradients, we must look at how a neural network learns. Learning happens by adjusting the weights (the internal settings) of the network to reduce the error (the difference between the network's prediction and the correct answer). This adjustment is guided by a number called the **gradient**. The gradient tells us two things: which direction to change the weight, and how big that change should be.

The gradient for any single weight is calculated using a method called **backpropagation**. Backpropagation works backward from the output of the network to the input. It relies on a rule from calculus called the **chain rule**. The chain rule says that if you want to know how an early part of a process affects the final result, you must multiply together all the small effects of each step in between.

Imagine the network has L layers. Let us number them 1 (input) to L (output). We want to find the gradient for a weight in layer l (where l is some layer near the beginning, like layer 1 or 2).

The formula for this gradient is:

g_l = g_L × f'_{l+1} × f'_{l+2} × ... × f'_L

Here is what each symbol means:
- **g_l**: The gradient (sensitivity) for the weight in layer l. This is the number we are trying to find.
- **g_L**: The gradient at the final output layer. This is known because we can directly compare the network's final output to the correct answer.
- **f'_k**: The derivative of the activation function for layer k. In plain words, this is a number that represents how much layer k amplifies or shrinks the signal passing through it. For many common activation functions (like sigmoid), this number is always less than 1. Specifically, it is often between 0 and 0.25.
- **×**: Multiplication.

The key insight is that g_l is the product of many numbers (the f' values) multiplied together. If every one of these numbers is smaller than 1, then multiplying them together makes the result smaller and smaller with each step.

For example, if you multiply 0.5 by itself 20 times, the result is extremely close to zero. This is what happens in a deep network. The signal that tells the early layers how to update their weights becomes so tiny that it is effectively zero. The early layers stop learning because they receive no meaningful instruction on how to improve.

This phenomenon is called the **vanishing gradient** problem. It makes training very deep networks difficult or impossible without special techniques, such as skip connections (which we will discuss later) or careful initialization of weights.


## A worked example

Let us walk through a concrete example with real numbers to see how quickly the gradient shrinks.

Assume we have a neural network with 20 layers. We want to find the gradient for the weight in layer 1 (the first layer). 

We know the following:
1. The gradient at the output layer (layer 20) is **g_20 = 1.0**. This is our starting point.
2. Each intermediate layer has an activation derivative **f'_k = 0.5**. This means each layer passes on only half of the sensitivity it receives. This is a typical value for sigmoid-like activations.

The formula for the gradient at layer 1 is:

g_1 = g_20 × f'_2 × f'_3 × ... × f'_20

Since there are 19 layers between layer 1 and layer 20 (layers 2 through 20), we multiply by 0.5 exactly 19 times.

g_1 = 1.0 × 0.5 × 0.5 × ... × 0.5  (19 times)

g_1 = 1.0 × (0.5)^19

Now, let us calculate (0.5)^19:
- After 1 multiplication: 0.5
- After 2 multiplications: 0.25
- After 3 multiplications: 0.125
- After 4 multiplications: 0.0625
- ... 
- After 10 multiplications: approximately 0.000976
- After 19 multiplications: approximately 0.00000191

So, g_1 ≈ 0.00000191.

This number is about two millionths of the original gradient. In practical terms, if we use this gradient to update the weights in layer 1, the change will be so small that it will take millions of training steps to make any noticeable difference. The early layers are effectively frozen.

If we had used a smaller derivative, say f'_k = 0.25 (which is common for sigmoid activations), the shrinkage would be even faster:

g_1 = 1.0 × (0.25)^19 ≈ 4.5 × 10^-12

This is essentially zero. The gradient has vanished.


## From scratch, in Python

```python
# Simulate vanishing gradients in a deep network

# Imagine a network with 50 layers. Each layer has a single neuron for simplicity.
# During backprop, each layer multiplies the gradient by its local derivative.
# For a sigmoid activation, this derivative is at most 0.25.

# Step 1: Define the network depth
num_layers = 50

# Step 2: Set the initial gradient at the output (e.g., how much the output needs to change)
gradient = 1.0

# Step 3: Each layer's derivative (sigmoid with input 0 gives max derivative 0.25)
# In a real network, these differ, but for the worst case, assume 0.25
layer_derivative = 0.25

# Step 4: Loop through layers, multiplying the gradient
gradient_at_each_layer = []
for layer in range(num_layers):
    gradient = gradient * layer_derivative
    gradient_at_each_layer.append(gradient)

# Step 5: Print the gradient at a few layers
print("Gradient at layer 1:", gradient_at_each_layer[0])
print("Gradient at layer 10:", gradient_at_each_layer[9])
print("Gradient at layer 50:", gradient_at_each_layer[49])

# Output:
# Gradient at layer 1: 0.25
# Gradient at layer 10: 9.5e-07
# Gradient at layer 50: 1.2e-30 (essentially zero)
```


## The same thing, with a framework

```python
# In PyTorch, you can observe vanishing gradients by computing backprop on a deep network
# with sigmoid activations and printing the gradients.

import torch
import torch.nn as nn

class DeepNet(nn.Module):
    def __init__(self):
        super().__init__()
        self.layers = nn.ModuleList([nn.Linear(10, 10) for _ in range(50)])
        self.activation = nn.Sigmoid()  # Saturating activation causes vanishing

    def forward(self, x):
        for layer in self.layers:
            x = self.activation(layer(x))
        return x

model = DeepNet()
optimizer = torch.optim.SGD(model.parameters(), lr=0.01)
x = torch.randn(1, 10)
y = torch.ones(1, 1)  # Dummy target

# Forward pass
output = model(x)
loss = (output.sum() - y).sum()

# Backward pass
loss.backward()

# Check gradients of first and last layer
first_layer_grad = model.layers[0].weight.grad.norm().item()
last_layer_grad = model.layers[-1].weight.grad.norm().item()
print(f"First layer grad norm: {first_layer_grad:.2e}")
print(f"Last layer grad norm: {last_layer_grad:.2e}")
# Expected: first layer gradient is orders of magnitude smaller
```


## What the code does

**From scratch:**
- `num_layers = 50` sets the depth — 50 layers is a deep network.
- `gradient = 1.0` is the starting signal from the loss.
- `layer_derivative = 0.25` is the maximum derivative of a sigmoid activation — it's always ≤ 0.25.
- The loop `for layer in range(num_layers)` runs 50 times. Each time, `gradient = gradient * layer_derivative` simulates multiplying by the local derivative during backprop.
- `gradient_at_each_layer` stores the gradient after each layer so we can inspect it.
- The print statements show how quickly the gradient collapses — by layer 10 it's already below 0.000001.

**Framework (PyTorch):**
- We define a simple network with 50 linear layers and sigmoid activations — no clever tricks like ReLU or residual connections.
- After a forward pass and computing a dummy loss, we call `loss.backward()` which computes gradients for all parameters using automatic differentiation.
- `model.layers[0].weight.grad.norm()` gives the magnitude of the gradient for the first layer's weights; `.norm()` is the length of the gradient vector.
- Typically, the first layer's gradient norm is 10^10 times smaller than the last layer's — showing the vanishing effect.
- A real developer would see this and recognise the problem; they'd switch to ReLU or add residual connections to fix it.


## Common misconceptions

**Vanishing gradients mean the network has no gradients at all** — The gradients are exactly zero, so the network is completely frozen.

In practice, gradients are never exactly zero — they're extremely small, like 10^-20. The network still 'updates' weights, but by an amount so tiny it has no effect on the loss. The network appears to learn nothing, but technically it's making microscopic steps. In fact: The gradient is very small but non-zero. The weights change by a tiny amount that is negligible compared to the network's scale, so learning effectively stops.

**Only depth causes vanishing gradients** — If my network has many layers, it will always have vanishing gradients.

Depth is necessary but not sufficient. The problem also depends on the activation function (sigmoid/tanh saturate to small derivatives) and weight initialisation (if weights start very small, gradients shrink even faster). With ReLU and careful initialisation (like Xavier), even a 100-layer network can train without vanishing. In fact: Depth is necessary, but the choice of activation and initialisation can prevent the problem.

**Sigmoid is fine for hidden layers; the vanishing just means training is slower** — I'll just train longer or use a higher learning rate — the network will eventually learn.

A higher learning rate can cause exploding gradients in early layers (making the loss NaN), and longer training just multiplies tiny updates — still no progress. The early layers are effectively stuck at random values; no amount of time fixes that. In fact: Sigmoid hidden layers prevent early layers from learning altogether. Using ReLU is the standard fix.

**Residual connections just add more parameters** — A residual block (x + F(x)) is just another layer that increases overfitting.

Residual connections add no new parameters — they simply add the input to the output of the layer. They create a direct path for gradients to flow to early layers without being multiplied by many small derivatives. They improve trainability, not just capacity. In fact: Residual connections are a gradient highway: they let the learning signal skip through the network, preserving it for early layers.


## In short

Vanishing gradients is the reason deep networks were untrainable before the late 2010s. Gradients shrink multiplicatively as they propagate backwards, so early layers get almost no signal. The fixes are: use non-saturating activations (ReLU family), add residual connections (shortcuts), and use good weight initialisation. Without this understanding, any deep network you build will silently fail to learn; with it, you can diagnose and fix the problem, and appreciate why every modern architecture is built the way it is.


## Where this is used

**In large language models (LLMs):** LLMs like GPT have dozens of layers (e.g., 96 in GPT-3). Without fixes, training would be impossible. The primary fix is **residual connections** (a shortcut that adds the input to the output of each block), which ensures the gradient has a direct path back to early layers. Additionally, **normalisation layers** (like LayerNorm) keep the scale of activations stable, preventing both vanishing and exploding gradients. Developers use frameworks like PyTorch and simply stack `MultiheadAttention` and `MLP` blocks with residual paths — the architecture is built so gradients don't vanish.

**In small models on devices (TinyML, mobile):** Small models for keyword spotting or image recognition on embedded devices often use depthwise separable convolutions, which are efficient. But they also face vanishing gradients if they're deep. They usually use ReLU (not sigmoid) and batch normalisation, and they often avoid very deep architectures because they have limited compute. A developer might use TensorFlow Lite Micro or PyTorch Mobile to deploy a model with only a few convolutional blocks — deep enough for the task but shallow enough to avoid vanishing.

**In retrieval-augmented generation (RAG):** RAG itself involves two parts: a retriever (often an embedding model) and a generator (an LLM). The embedding model is a neural network that produces vectors for text — these models are typically not as deep as the generator, but they still use the same fixes (residuals, ReLU). The generator uses the same architecture fixes. Vanishing gradients are relevant because if the embedding model has vanishing gradients during training, the embeddings won't be meaningful — docs and queries would map to nonsense in the vector space. A developer working on RAG would fine-tune the embedding model without worrying about vanishing, because it's already designed with these fixes.

**In agentic systems that plan and call tools:** These systems rely on an underlying LLM (often with tool calling). The LLM is trained with the same techniques as above. The agent itself isn't a neural network being trained with gradient descent; it's a program that orchestrates calls. But the performance of the agent depends on the LLM's ability to reason, which depends on the depth of the model, which is only possible because vanishing gradients were solved. So it's an indirect but critical connection.

**In everyday applied ML (recommendation, vision, forecasting, fraud detection):** Deep neural networks are used for all of these. For example, a fraud detection model might have several dense layers. If a developer accidentally uses sigmoid activations in hidden layers and doesn't add residuals, the model will train very slowly or not at all — the first layers won't learn. The fix is to use ReLU (or GELU) and, if the network is very deep, add residual connections. In PyTorch, the developer writes `nn.ReLU()` instead of `nn.Sigmoid()`, and for very deep networks, they'd add a skip connection: `x = layer(x) + x` (with appropriate shapes).

**What a developer actually does:** They pick a well-tested architecture (like ResNet for images, or a Transformer for text) and rely on these fixes being built in. They rarely hand-craft new attention mechanisms; they use `torch.nn.Sequential` with `nn.Linear`, `nn.ReLU`, `nn.LayerNorm`, and residual blocks. If they train a custom network and see no improvement, the first thing they check is gradient magnitude — they might print `model.layers[0].weight.grad.norm()` and see it's tiny. That's the vanishing gradient signal, and they'd then change activation or add residuals.


---

Read this on the web: https://basavapurushottam.com/learn-ai/vanishing-gradients
