# Layer Normalization

Normalising activations across features within a single example, independent of batch composition.

**Rests on:** Activation Functions

*Deep Learning · Neural Network Architectures*


## Why learn this

Layer normalization is one of the quiet workhorses behind modern AI. It is the reason transformers — the architecture behind ChatGPT and other large language models (computer programs that understand and generate human text) — train reliably at all. Without it, the training of a deep network would be slow, unstable, and prone to numbers becoming too small to matter or growing uncontrollably large. You will meet it again when you study attention, transformers, and the details of how these language models are built. Also, it powers small models that run on your phone — the keyword spotter that wakes your assistant, for instance. So this concept is directly connected to the next several topics in your path.


## The intuition

Imagine you are a teacher grading exams. Each student sits in a row of desks, and each row has the same number of columns (say, 10 columns for 10 subjects). One seating chart lists all rows of numbers — that's your 'batch' of examples. But you soon notice that one student's scores are wildly higher than another's: one student scored 95, 90, 97; another scored 30, 20, 35. If you tried to compare them directly, you'd be misled. What you actually care about is the pattern: is the student strong in maths but weak in languages? To see that, you need to 'normalise' each student's scores — subtract their own average, then divide by their own spread (a measure of how much the numbers vary from the average) — so that every student's row now centres around zero and has a similar scale. That way, when you look at the numbers, you aren't distracted by the overall level, only by the shape. This is layer normalization in a nutshell: for a single example (one row), you look at all its features (the columns), then adjust them so that the average across those features is zero and the spread is fixed. The key is that you do this per example, not across the whole classroom at once. This is what makes it 'independent of batch composition' — the result does not depend on who else is in the batch.


## An analogy

Think of a chef preparing a sauce for each customer. Every day, the chef makes a batch of soup, tastes it, and adds salt to make sure the soup is consistently seasoned. That's like batch normalization — it adjusts based on what's in the whole pot, so one bad day (a salty batch) affects every bowl. Now, the chef's assistant has a different job: for each individual plate, they taste the sauce and adjust it per plate, not per pot. If one plate is too salty on its own, they fix just that plate, regardless of the others. This per-plate adjusting is layer normalization. The great advantage is that it doesn't matter whether the chef made one bowl or a hundred bowls; each plate is treated individually. And that is why transformers use it — those networks process examples one at a time (or in small groups) and can't rely on a big batch to stabilise the numbers. There is a place where the analogy breaks, though: the chef physically tastes the sauce, but a neural network normalises in a routine arithmetic way — it doesn't taste, it computes a mean and a spread.


## Definition

Layer normalization is a technique that rescales the inputs to each layer of a neural network, per individual example, so that the values across that example's features have a mean of zero and a standard deviation of one, before a learned scaling and shifting step is applied.


## How it connects

You have not studied any previous topics, so this is your first building block. But your library notes suggest you are heading toward Neural Network Architectures, Transformers (systems for processing language or other sequences using attention mechanisms), and attention. Layer normalization is a key component of the transformer block. Without it, the residual connections and multi-head attention that you will study later would not train reliably. It also relates to the LSTM concepts in your notes (e.g., LSTM Cell State) — recurrent networks also benefited from normalisation, although they used a different variant (layer norm applied to the cell state). You have not covered that yet, so just keep in mind: normalisation is a recurring theme in deep learning.


## The mathematics

Layer normalization is a way to adjust the numbers coming out of a neuron so they are easier for the rest of the network to handle. It does this by forcing the numbers in a single example to have an average of zero and a spread (standard deviation) of one. Then, it allows the network to stretch or move those numbers if it wants to.

To understand this, we need to look at just one example from the batch. Imagine you are processing one sentence, one image, or one data point. Let us say this single example has n features. We will call the input numbers for this example x sub 1, x sub 2, all the way up to x sub n. The symbol x with a subscript i, written as x_i, simply means the number in the i-th position of that list.

Step 1: Find the average (mean)
The first thing layer normalization does is find the average of these n numbers. We call this average mu (the Greek letter mu). To find mu, you add up all the input numbers and divide by how many there are. In words:
mu equals the sum of x_i for i from 1 to n, divided by n.

Step 2: Center the values
Next, we want to shift the data so that its center is at zero. We do this by subtracting the average mu from each individual number. We call these new 'centered' values d sub i (for deviation). In words:
d_i equals x_i minus mu.

Step 3: Measure the spread (variance)
Now we need to know how spread out these centered numbers are. We measure this by squaring each centered value (multiplying it by itself), adding those squares together, and dividing by n. This result is called the variance. We write this as sigma squared (sigma-squared). In words:
sigma-squared equals the sum of d_i squared, for i from 1 to n, divided by n.

Step 4: Find the standard deviation
The variance is in squared units, which is hard to interpret directly. So we take the square root of the variance to get back to the original scale. This is called the standard deviation. We write this as sigma. In words:
sigma equals the square root of sigma-squared.

In practice, if the variance happens to be exactly zero (meaning all input numbers were identical), the standard deviation would be zero, and we cannot divide by zero. To prevent this crash, we add a tiny number called epsilon (epsilon) to the variance before taking the square root. So the denominator we actually use is the square root of (sigma-squared plus epsilon).

Step 5: Standardize the values
Now we scale the centered numbers so that their spread becomes exactly one. We do this by dividing each centered value d_i by the standard deviation term we just calculated. We call these new 'normalized' values x-hat sub i. In words:
x-hat_i equals (x_i minus mu) divided by the square root of (sigma-squared plus epsilon).

At this point, if you were to calculate the average of all x-hat_i values, it would be zero. If you calculated their standard deviation, it would be one.

Step 6: Apply learned scale and shift
Layer normalization does not stop here. It introduces two new sets of numbers that the network learns during training: gamma (gamma) and beta (beta). These have the same length as the input, so there is a gamma_i and a beta_i for each feature i.

Gamma acts as a 'scale' factor. Beta acts as a 'shift' or offset. The network can choose to keep the normalized values as they are, or it can stretch them (by changing gamma) and move them up or down (by changing beta). For each position i, the final output y_i is found by multiplying x-hat_i by gamma_i and then adding beta_i. In words:
y_i equals gamma_i times x-hat_i plus beta_i.


## A worked example

```text
We have one example (one row) with 4 features. Let's call the values x = [2, 4, 6, 8]. We'll do layer normalization with learned parameters gamma (scale) and beta (shift). For simplicity, we'll set gamma = 1.0 and beta = 0.0 first, to see the raw normalisation. Later we'll see what happens with other values.

Step 1: Average (mean)
Add the four values together: 2 + 4 + 6 + 8 = 20.
Divide by the count of values (4): 20 / 4 = 5.
The mean, mu, is 5.

Step 2: Centred values
Subtract the mean (5) from each original value:
x_1: 2 - 5 = -3
x_2: 4 - 5 = -1
x_3: 6 - 5 = 1
x_4: 8 - 5 = 3
The centred values are [-3, -1, 1, 3].

Step 3: Squares of centred values
Square each centred value (multiply by itself):
(-3) * (-3) = 9
(-1) * (-1) = 1
(1) * (1) = 1
(3) * (3) = 9
The squared values are [9, 1, 1, 9].

Step 4: Variance (mean of squares)
Add the squared values together: 9 + 1 + 1 + 9 = 20.
Divide by the count of values (4): 20 / 4 = 5.
The variance, sigma-squared, is 5.

Step 5: Standard deviation
Take the square root of the variance. We add a tiny epsilon (let's use 0.00001) to avoid division by zero, though it makes little difference here.
Denominator = square root of (5 + 0.00001) ≈ square root of 5.00001 ≈ 2.23607.
The standard deviation term is approximately 2.23607.

Step 6: Standardised values (x-hat)
Divide each centred value by the denominator:
x-hat_1: -3 / 2.23607 ≈ -1.3416
x-hat_2: -1 / 2.23607 ≈ -0.4472
x-hat_3: 1 / 2.23607 ≈ 0.4472
x-hat_4: 3 / 2.23607 ≈ 1.3416
The standardised values are approximately [-1.3416, -0.4472, 0.4472, 1.3416].

Step 7: Apply gamma and beta
We set gamma = [1.0, 1.0, 1.0, 1.0] and beta = [0.0, 0.0, 0.0, 0.0].
For each position i, calculate y_i = gamma_i * x-hat_i + beta_i.
y_1: 1.0 * -1.3416 + 0.0 = -1.3416
y_2: 1.0 * -0.4472 + 0.0 = -0.4472
y_3: 1.0 * 0.4472 + 0.0 = 0.4472
y_4: 1.0 * 1.3416 + 0.0 = 1.3416
The output is approximately [-1.3416, -0.4472, 0.4472, 1.3416].

Result: The normalised output is [-1.3416, -0.4472, 0.4472, 1.3416]. Now the values have mean 0 and standard deviation 1, as you can check by repeating the process.

If gamma were [2.0, 2.0, 2.0, 2.0] and beta were [0.5, 0.5, 0.5, 0.5], each value would be multiplied by 2 and then added 0.5:
y_1: (2.0 * -1.3416) + 0.5 = -2.6832 + 0.5 = -2.1832
y_2: (2.0 * -0.4472) + 0.5 = -0.8944 + 0.5 = -0.3944
y_3: (2.0 * 0.4472) + 0.5 = 0.8944 + 0.5 = 1.3944
y_4: (2.0 * 1.3416) + 0.5 = 2.6832 + 0.5 = 3.1832
The final output would be approximately [-2.1832, -0.3944, 1.3944, 3.1832].
```


## From scratch, in Python

```python
# Step 1: Build a list of raw activations for one example
activations = [2.0, 4.0, 6.0, 8.0]

# Step 2: Compute the mean (average)
mean = sum(activations) / len(activations)
print("Mean:", mean)  # 5.0

# Step 3: Compute the variance (average squared deviation from the mean)
squared_deviations = []
for x in activations:
    diff = x - mean
    squared = diff * diff
    squared_deviations.append(squared)
# squared_deviations is now [9.0, 1.0, 1.0, 9.0]
variance = sum(squared_deviations) / len(squared_deviations)
print("Variance:", variance)  # 5.0

# Step 4: Compute the standard deviation (square root of variance)
std = variance ** 0.5
print("Standard deviation:", std)  # 2.236...

# Step 5: Normalise each activation
normalised = []
for x in activations:
    centred = x - mean
    norm = centred / std
    normalised.append(norm)
print("Normalised:", normalised)  # [-1.342, -0.447, 0.447, 1.342]

# Step 6: Apply learned scale (gamma) and shift (beta)
# In a real model, gamma and beta are learned. Here we set them manually.
gamma = [1.0, 1.0, 1.0, 1.0]
beta = [0.0, 0.0, 0.0, 0.0]
output = []
for i in range(len(normalised)):
    y = normalised[i] * gamma[i] + beta[i]
    output.append(y)
print("Output:", output)  # same as normalised because gamma=1, beta=0

# Let's try gamma = 2.0 and beta = 0.5 for all features
gamma2 = [2.0, 2.0, 2.0, 2.0]
beta2 = [0.5, 0.5, 0.5, 0.5]
output2 = []
for i in range(len(normalised)):
    y = normalised[i] * gamma2[i] + beta2[i]
    output2.append(y)
print("Output with gamma=2, beta=0.5:", output2)  # [-2.184, -0.394, 1.394, 3.184]
```


## The same thing, with a framework

```python
import numpy as np

# Simulate a batch of 3 examples, each with 4 features (like a mini-batch)
x = np.array([[2.0, 4.0, 6.0, 8.0],   # example 1
              [1.0, 3.0, 5.0, 7.0],   # example 2
              [0.0, 2.0, 4.0, 6.0]])  # example 3

# Layer norm in PyTorch (if you had it installed) would be:
# import torch.nn as nn
# layer_norm = nn.LayerNorm(normalized_shape=4)
# output = layer_norm(torch.tensor(x, dtype=torch.float32))

# But to see the arithmetic, compute it manually with NumPy:
mean = x.mean(axis=1, keepdims=True)  # mean per row (per example)
variance = x.var(axis=1, keepdims=True)  # variance per row
gamma = np.array([1.0, 1.0, 1.0, 1.0])
beta = np.array([0.0, 0.0, 0.0, 0.0])
normalised = (x - mean) / np.sqrt(variance + 1e-5)  # epsilon to avoid division by zero
output = normalised * gamma + beta
print(output)

# The key difference: `axis=1` means we compute statistics along the feature dimension,
# not across examples. That is what makes it independent of batch size.
```


## What the code does

First, we create a list `activations` with four numbers. That is a Python list — an ordered collection that you can change. We then compute the mean by summing the list and dividing by its length. Next, we compute the variance: we loop over each value, subtract the mean (that's `diff`), then square it (multiply it by itself), and collect those squares in a new list. Then we average those squares. The square root of the variance is the standard deviation. Then we normalise each value by subtracting the mean and dividing by the standard deviation. Finally, we apply the learned scale and shift — in a real model, `gamma` and `beta` would be arrays of numbers the model has learned, but here we set them by hand. Notice how we used a `for i in range(...)` loop to go through each position in the list. That's a common pattern when you need to use the index number.


## Common misconceptions

**Layer norm and batch norm are the same** — I've heard of batch norm; layer norm must be the same thing.

They normalise different dimensions. Batch norm computes statistics across the batch (over examples) for each feature, so the result depends on the other examples in the batch. Layer norm computes statistics across features for each example, so the result is independent of batch size and composition. In fact: Layer norm normalises per example, across its features; batch norm normalises per feature, across the batch.

**Layer norm shrinks the network's expressiveness** — Forcing all values to have zero mean and unit variance must make the network less powerful.

The network can undo the normalisation through the learned gamma and beta parameters. If it learns gamma set to the original scale and beta set to the original mean, it effectively reverses the normalisation. So it does not lose expressive power. In fact: Layer norm includes learned scale and shift parameters, so the network can always represent the original scale if that is best.

**Layer norm is only for transformers** — I only hear about layer norm in transformer models, so it must not be used elsewhere.

Layer norm is a general normalisation technique first proposed for recurrent networks. It is used in many architectures beyond transformers, including for on-device keyword spotters and speech models. In fact: Layer norm is used in many neural network architectures, not just transformers. It was originally introduced for RNNs.

**Normalisation is optional for small models** — My model is small, so I can skip layer norm without any problem.

Even small models can suffer from unstable training if activations are not kept on a healthy scale. Layer norm is not about size, it is about the scale of numbers flowing through the layers. In fact: Layer norm helps stabilise training at any model size. It is a standard component even in small on-device models.

**Due to epsilon, the normalised values are not perfectly zero mean** — The epsilon I add to the variance means the output mean is not exactly zero.

Epsilon is added to the variance before taking the square root, so the division uses a slightly enlarged denominator. But the mean of the output is still exactly zero because subtracting the same mean from all values and dividing by a constant does not change the mean. The epsilon is there only to prevent division by zero. In fact: Epsilon prevents numerical issues but does not affect the zero-mean property.


## In short

Layer normalization takes a single example's activations, computes their mean and standard deviation, then rescales them so they have zero mean and unit standard deviation, and finally applies learned scaling (gamma) and shifting (beta) parameters. It is done per example, not across the batch, which makes it independent of batch size and composition. This is the reason transformers use it, and it is essential for training stability in deep networks. Remember: layer norm = per-example standardisation + learned scale and shift.


## Where this is used

**In large language models (LLMs)**: Every transformer block — the building block of GPT-style models — contains a layer normalization layer, usually placed before the self-attention sublayer (pre-norm) and before the feed-forward sublayer. In practice, a developer building an LLM from scratch in PyTorch would write something like:
    ```python
    x = x + self.attn(self.norm1(x))   # pre-norm: apply layer norm before attention, then residual
    x = x + self.ffn(self.norm2(x))    # pre-norm before feed-forward
    ```
    The `self.norm1` and `self.norm2` are layer normalization layers. Why is this so important? Without layer norm, the activations at each layer would grow or shrink in scale as they travel through the network; with hundreds of layers, numbers can explode to huge values or decay to zero, causing training to fail. Layer norm keeps the values at a consistent scale, so gradient descent (which you will learn later) can update weights effectively. The placement also matters: your library notes say 'pre-norm versus post-norm placement materially changes training stability.' Pre-norm (applying norm before the sublayer) is now the standard because it gives better stability and allows training with very deep models. Developers tuning an LLM might experiment with whether to put layer norm before or after the attention block, and small differences can determine whether the training converges at all.

**In small language models that run on devices (TinyML / on-device AI)**: When a model runs on a phone or a smart home device, it often processes a single short utterance at a time — a batch of size 1. Batch normalization (another technique you might hear of) behaves differently when the batch is tiny or 1, because it needs to estimate the average over many examples to be stable. Layer norm does not care: it normalises each example on its own. So keyword spotters, voice assistants, and on-device translation models all use layer norm precisely because it is 'independent of batch composition' — it gives the same result whether you process one word or a hundred. A developer building an on-device model would still define the same `LayerNorm` layer, but the key advantage is that the model's behaviour does not depend on how many examples are fed in during inference.

**In retrieval-augmented generation (RAG)**: RAG systems retrieve relevant documents (converted to embeddings) and feed them to an LLM to generate an answer. The embeddings are often normalised before being used in similarity calculations (like dot product). While that embedding normalisation is not exactly layer norm (it's usually L2 normalisation, which is a different operation), the LLM itself contains layer norm layers. So when you hear that a RAG system uses a transformer to encode queries and documents, that transformer's stability comes in part from layer norm. A developer implementing RAG would use a pre-trained transformer encoder and not touch the internal layer norm; but if they were training a custom encoder for better retrieval, they would need to ensure layer norm is present to train reliably.

**In agentic systems that plan and call tools**: An agent uses an LLM to plan steps, call tools, and act in loops. The LLM's transformer stack includes layer norm, just as in any LLM. So while layer norm is not directly involved in the agent's logic, it is the silent support that makes the LLM trainable and stable. A developer fine-tuning a model for tool-calling would keep the layer norm layers as is, possibly adjusting the gamma and beta parameters during fine-tuning to adapt the scale and shift to the new domain.

**In everyday applied machine learning (recommendation, vision, forecasting, fraud detection, speech)**: Layer norm is used in many deep networks beyond transformers. For example, in speech recognition, a network might process spectrogram frames; layer norm helps the network generalise across different speakers or recording conditions because it ignores the overall volume of the audio. In time-series forecasting, a model might use a transformer-based forecast network, again relying on layer norm for stable training. In fraud detection, where a model sees a batch of transactions, layer norm makes the model less sensitive to the number of transactions in a batch. A developer building a neural network in PyTorch would add a `nn.LayerNorm` layer after a linear layer, before an activation function, to keep the activations on a healthy scale. What breaks when it is done badly? If you skip layer norm, you might see the loss plateau, gradients explode, or training diverge altogether. If you place it incorrectly (e.g., after the residual connection instead of before), you might get worse performance or instability. So developers often treat layer norm's placement as a careful design choice.


---

Read this on the web: https://basavapurushottam.com/learn-ai/layer-normalization
