# Autoencoders

Networks trained to reconstruct their input through a narrow bottleneck, forcing a compressed representation that captures the essential structure.

**Rests on:** Data Compression, Forward Propagation

**Leads to:** Bi-Encoders and Cross-Encoders

*Deep Learning · Deep Learning*


## Why learn this

Autoencoders give you a way to compress data into a smaller essence and then rebuild it. That turns out to be useful in many real-world tasks: removing noise from photos, spotting fraud or machine faults, and creating the embeddings that RAG systems use to search documents. Understanding autoencoders is a direct step toward studying embeddings, anomaly detection, and representation learning. Later, when you meet LLMs and their internal representations, you'll recognise the same trick: learn a compact, meaningful form of the data rather than memorising the raw data itself.


## The intuition

An autoencoder is a kind of neural network that is trained to copy its input to its output, but the catch is that in the middle it is forced to make the information much smaller. So the network has to find a 'short description' of the input that still contains enough to reproduce it. That short description is the compressed representation, and the whole process is learning useful structure.

Think of a person who takes detailed meeting notes. If the person writes every word down, the notes are long and almost a copy. But if the person writes a short summary of only the key points, the summary loses some details, but still captures the main ideas. An autoencoder is like that: the input is the full meeting, the compressed middle is the summary, and the output is an attempt to rebuild the meeting from the summary. Because the summary is small, the network cannot simply remember every word—it has to learn what matters most.

In practice, the network has two parts: an encoder that compresses the input down to the small middle, and a decoder that expands the middle back to the original size. Training is done by giving the network many examples, calculating how different the output is from the input, and adjusting the weights to reduce that difference. Once trained, the middle part can be used as a compact fingerprint of the input, and the output can be used to regenerate something close to the original.


## An analogy

Imagine a photographer who has to send photos to a friend over a very slow connection—they have to compress each photo into a tiny file. The photographer learns that certain patterns (faces, trees, sky) can be stored with fewer bits. For a photo of a cat on a sofa, the photographer might encode 'cat, sofa, window-light' into a short code, and then the friend can generate a picture from that code. If the code is too small (too short), they lose important details; if it's large enough, the reconstruction looks almost like the original.

That is exactly what an autoencoder does: the encoder is the photographer compressing, the decoder is the friend reconstructing, and the size of the code is the bottleneck. The network learns what patterns in the images are worth keeping.

Where the analogy stops: the photographer manually decides what to encode, but the autoencoder discovers the features itself from data. Also, the reconstruction is a best-effort—the output is not pixel-for-pixel identical unless the bottleneck is big enough to hold all the information, which would defeat the purpose and is exactly what we avoid.


## Definition

An autoencoder is a type of neural network that is trained to output a close reconstruction of its input, but with a middle layer that is forced to be smaller than the input, so the network has to learn a compressed representation of the data.


## How it connects

You have not studied any neural networks yet, so this page does not assume you know about weights, layers, or training. It only needs the idea of a function: something that takes a number (or a list of numbers) and produces another number (or list). The autoencoder is a function that maps a list of numbers (the input) to another list (the output), and we train it to make the output as close as possible to the input. The twist is that in the middle of the function there is a step where the list becomes much smaller—that is the bottleneck. Later, if you study embeddings or RAG, you will see that the same idea of a compact representation is used there.


## The mathematics

An autoencoder is a specific type of artificial neural network designed to copy its input to its output. It does this by first compressing the input into a smaller, simpler representation, and then expanding that representation back out to match the original input.

Think of it like packing a suitcase. You have a large set of clothes (the input). You must fit them into a small carry-on bag (the compressed representation). Then, you unpack the bag at your destination to get the clothes back (the output). The goal is to make sure that when you unpack, the clothes are still there and in good condition.

Mathematically, this process happens in two stages: encoding and decoding. Both stages use the same basic arithmetic operation: a weighted sum followed by an activation function.

The Weighted Sum
Every layer in the network takes a list of numbers (inputs) and produces a new list of numbers (outputs). To get one number in the output, the network does the following:
1. It takes each input number.
2. It multiplies that input by a specific weight (a number that tells the network how important that input is).
3. It adds up all these products.
4. It adds a small constant called a bias.

This result is just a single number. But before it becomes part of the next layer, it usually passes through an "activation function." This is a rule that squashes or changes the number to keep it within a useful range (for example, between 0 and 1). Common activation functions include ReLU (which turns any negative number into zero) or Sigmoid (which squeezes any number into the range 0 to 1).

The Encoder
The encoder is the first half of the autoencoder. It takes an input vector x with N numbers (for example, N pixels in a small image). It multiplies this vector by a weight matrix W_h and adds a bias vector b_h. The result is passed through an activation function f to produce a hidden vector h with M numbers. Here, M is smaller than N. This step is the compression.

The Decoder
The decoder is the second half. It takes the hidden vector h (which has M numbers) and multiplies it by another weight matrix W_o, adding a bias vector b_o. The result is passed through an activation function g to produce the output vector x_hat with N numbers. This step is the reconstruction.

The Loss Function
To train the autoencoder, we need to know how good the reconstruction is. We compare the original input x with the reconstructed output x_hat. The most common way to measure the difference is the Mean Squared Error (MSE). For each position i from 1 to N, we take the difference between the original value x_i and the reconstructed value x_hat_i, square that difference, and then average all those squared differences.

Loss = (1 / N) * sum of (x_i - x_hat_i)^2 for i from 1 to N

If the loss is zero, the output is identical to the input. If the loss is large, the output is very different. The training process adjusts the weights and biases to make this loss as small as possible.

Training
We do not calculate the perfect weights by hand. Instead, we use an algorithm called gradient descent. This algorithm looks at the loss and calculates how changing each weight would affect the loss. It then takes a small step in the direction that reduces the loss. This is repeated many times over many examples until the loss stops decreasing significantly.


## A worked example

Let us walk through a complete example with real numbers. We will build a tiny autoencoder.

The Setup
- Input size: 3 numbers (let's call them x1, x2, x3).
- Hidden size: 2 numbers (let's call them h1, h2). This is the compression step because 2 is less than 3.
- Output size: 3 numbers (let's call them out1, out2, out3).

We will use the Sigmoid activation function for the hidden layer. The Sigmoid function takes any number z and returns 1 divided by (1 plus e to the power of negative z). It squashes the result to be between 0 and 1.
For simplicity, we will use a linear activation for the output layer, which means we do not change the value after the weighted sum.

The Weights and Biases
These are the numbers the network has learned. In this example, they are given to us.

Encoder weights (W_h): A 2x3 matrix. Each row corresponds to one hidden unit.
Row 1: [0.5, -0.2, 0.1]
Row 2: [0.3, 0.8, -0.4]

Encoder biases (b_h): Two numbers, one for each hidden unit.
b_h1 = 0.1
b_h2 = -0.2

Decoder weights (W_o): A 3x2 matrix. Each row corresponds to one output unit.
Row 1: [0.7, 0.2]
Row 2: [-0.1, 0.5]
Row 3: [0.4, 0.3]

Decoder biases (b_d): Three numbers, one for each output unit.
b_d1 = 0.0
b_d2 = 0.1
b_d3 = -0.1

The Input
x = [1.0, 0.0, -1.0]

Step 1: Compute the first hidden value (h1) before activation.
To get h1, we take the input vector x and multiply it by the first row of the encoder weights, then add the first bias.
Calculation:
h1_raw = (0.5 * 1.0) + (-0.2 * 0.0) + (0.1 * -1.0) + 0.1
h1_raw = 0.5 + 0 + (-0.1) + 0.1
h1_raw = 0.5

Step 2: Compute the second hidden value (h2) before activation.
To get h2, we take the input vector x and multiply it by the second row of the encoder weights, then add the second bias.
Calculation:
h2_raw = (0.3 * 1.0) + (0.8 * 0.0) + (-0.4 * -1.0) + (-0.2)
h2_raw = 0.3 + 0 + 0.4 - 0.2
h2_raw = 0.5

Step 3: Apply the Sigmoid activation to get the hidden vector h.
Now we apply the Sigmoid function to h1_raw and h2_raw.
Sigmoid(z) = 1 / (1 + e^-z)
For z = 0.5, e^-0.5 is approximately 0.6065.
Sigmoid(0.5) = 1 / (1 + 0.6065) = 1 / 1.6065 ≈ 0.622

So, h1 = 0.622 and h2 = 0.622.
The hidden vector is h = [0.622, 0.622].

Step 4: Compute the first output value (out1).
To get out1, we take the hidden vector h and multiply it by the first row of the decoder weights, then add the first bias.
Calculation:
out1_raw = (0.7 * 0.622) + (0.2 * 0.622) + 0.0
out1_raw = 0.4354 + 0.1244 + 0.0
out1_raw = 0.5598
Since the output activation is linear, out1 = 0.5598.

Step 5: Compute the second output value (out2).
To get out2, we take the hidden vector h and multiply it by the second row of the decoder weights, then add the second bias.
Calculation:
out2_raw = (-0.1 * 0.622) + (0.5 * 0.622) + 0.1
out2_raw = -0.0622 + 0.311 + 0.1
out2_raw = 0.3488
So, out2 = 0.3488.

Step 6: Compute the third output value (out3).
To get out3, we take the hidden vector h and multiply it by the third row of the decoder weights, then add the third bias.
Calculation:
out3_raw = (0.4 * 0.622) + (0.3 * 0.622) + (-0.1)
out3_raw = 0.2488 + 0.1866 - 0.1
out3_raw = 0.3354
So, out3 = 0.3354.

The Result
The reconstructed output is approximately [0.560, 0.349, 0.335].
The original input was [1.0, 0.0, -1.0].
The values are not identical because the bottleneck size (2) is smaller than the input size (3), so some information was lost during compression. This is expected behavior for a lossy autoencoder.

The Loss Calculation
To see how bad this error is, we calculate the Mean Squared Error (MSE).
Difference 1: 1.0 - 0.560 = 0.440. Squared: 0.1936
Difference 2: 0.0 - 0.349 = -0.349. Squared: 0.1218
Difference 3: -1.0 - 0.335 = -1.335. Squared: 1.7822
Sum of squared differences: 0.1936 + 0.1218 + 1.7822 = 2.0976
Average (MSE): 2.0976 / 3 ≈ 0.699

This loss value of 0.699 tells the training algorithm that the current weights are not good enough and need to be adjusted.


## From scratch, in Python

```python
# We will learn the basics: a list is a sequence of numbers, a loop repeats a block.
# We will implement a simple autoencoder with one hidden layer, using only plain Python.
# We will use a tiny example: input size 3, bottleneck size 2, output size 3.

import math

# Define the sigmoid activation (squashes any number to between 0 and 1)
def sigmoid(x):
    return 1 / (1 + math.exp(-x))

# Define the forward pass of the autoencoder.
# Input is a list of 3 numbers. We have two parameter matrices and two bias vectors.
def forward(x, W_h, b_h, W_o, b_o):
    # Compute hidden (bottleneck) activations: h = sigmoid(W_h * x + b_h)
    hidden = []
    for i in range(2):  # bottleneck size is 2
        # Compute the weighted sum for this hidden unit
        z = 0
        for j in range(3):
            z += W_h[i][j] * x[j]
        z += b_h[i]
        hidden.append(sigmoid(z))
    # Compute output: out = W_o * hidden + b_o (linear activation)
    out = []
    for i in range(3):  # output size is 3
        z = 0
        for j in range(2):
            z += W_o[i][j] * hidden[j]
        z += b_o[i]
        out.append(z)
    return hidden, out

# Define the mean squared error loss
def mse_loss(x, out):
    total = 0
    for i in range(len(x)):
        diff = x[i] - out[i]
        total += diff * diff
    return total / len(x)

# Training example: just one sample and one update step (you would repeat many times)
# We use a very simple gradient descent with a fixed learning rate.
# For simplicity, we derive gradients manually for this tiny example.
# In practice you would use automatic differentiation, but here we show the arithmetic.

# Initialize weights randomly (here we hardcode small numbers for demonstration)
W_h = [[0.5, -0.2, 0.1], [0.3, 0.8, -0.4]]  # 2x3
b_h = [0.1, -0.2]
W_o = [[0.7, 0.2], [-0.1, 0.5], [0.4, 0.3]]  # 3x2
b_o = [0.0, 0.1, -0.1]

x = [1.0, 0.0, -1.0]
learning_rate = 0.1

# Forward pass
hidden, out = forward(x, W_h, b_h, W_o, b_o)
print("Forward output:", out)
print("Loss:", mse_loss(x, out))

# One step of manual backpropagation for the output layer (derivative of MSE)
# dL/dout_i = 2*(out_i - x_i)/N
# Then update W_o and b_o using the chain rule (we omit the derivative of the
# activation for the output because it's linear).
d_out = [2 * (out[i] - x[i]) / len(x) for i in range(3)]

# Update decoder weights and biases
for i in range(3):
    for j in range(2):
        W_o[i][j] -= learning_rate * d_out[i] * hidden[j]
    b_o[i] -= learning_rate * d_out[i]

# For the hidden layer, we would compute gradients through the sigmoid.
# But this is the basic structure; in a real model you'd use a library.
print("After one update, new output:")
_, new_out = forward(x, W_h, b_h, W_o, b_o)
print(new_out)

```


## The same thing, with a framework

```python
In PyTorch, you would define a class with an encoder and decoder, using nn.Linear layers and an activation. The training loop would look like:

import torch
import torch.nn as nn
import torch.optim as optim

class Autoencoder(nn.Module):
    def __init__(self, n_input, n_hidden):
        super().__init__()
        self.encoder = nn.Sequential(nn.Linear(n_input, n_hidden), nn.Sigmoid())
        self.decoder = nn.Sequential(nn.Linear(n_hidden, n_input))

    def forward(self, x):
        h = self.encoder(x)
        return self.decoder(h)

model = Autoencoder(3, 2)
criterion = nn.MSELoss()
optimizer = optim.SGD(model.parameters(), lr=0.1)

x = torch.tensor([1.0, 0.0, -1.0])
for epoch in range(100):
    optimizer.zero_grad()
    out = model(x)
    loss = criterion(out, x)
    loss.backward()
    optimizer.step()

print('final output:', model(x))

nn.Linear is a layer that performs the same weighted sum and bias addition we did by hand. The loss is also the same. The library handles the backpropagation for you.
```


## What the code does

The code defines a function 'forward' that takes an input list x and the weight matrices and biases. It first computes the hidden (bottleneck) values: for each of the two hidden units, it loops over the three inputs, multiplies each by the corresponding weight, sums them, adds the bias, and applies the sigmoid function to get a value between 0 and 1. Then it does a similar weighted sum for each of the three outputs, but without a nonlinear activation. The output is the list of three numbers. The loss function computes the mean squared error between the input and output. For training, we take one step of gradient descent: we compute the derivative of the loss with respect to the output, then update the decoder weights and biases. The hidden layer update is left out for brevity, but the pattern is the same.


## Common misconceptions

**The bottleneck is unnecessary** — If the bottleneck is the same size as the input, the autoencoder would just learn to copy, but that's fine because we get a perfect reconstruction.

If the bottleneck is the same size (or larger), the network can easily learn to copy the input directly to the output, and the compressed representation is just the input itself—no learning of structure happens. The whole point is to force the network to find a shorter description that still captures the essential information. In fact: The bottleneck must be smaller than the input to force compression. Only then does the autoencoder learn meaningful features.

**Autoencoders output exactly the input** — After training, the autoencoder can reproduce the input perfectly, pixel for pixel.

Because of the bottleneck, information is lost. The reconstruction is an approximation, not an exact copy. Perfect reconstruction would require enough capacity, which defeats the purpose. In fact: The output is a reconstruction that is close to the input but usually not identical. The loss measures how close it is.

**Autoencoders are only for image compression** — Autoencoders are only useful for compressing images, like JPEG.

Autoencoders are used for many tasks beyond compression: anomaly detection, denoising, feature extraction, and pretraining. The idea of learning a compact representation applies to any type of data—text, sound, sensor signals. In fact: The core concept is learning a compressed representation. That has applications in anomaly detection, denoising, and embeddings, among others.

**The bottleneck is the output** — The output of the autoencoder is the compressed representation.

The output is the reconstruction. The compressed representation is the middle, hidden layer, which is usually not the final output. In fact: The encoder produces the bottleneck vector (the compressed representation). The decoder uses that to produce the output. The bottleneck is not the output of the entire network.

**An autoencoder can fix any noisy data perfectly** — If I train an autoencoder on noisy images, it can remove all noise and always return the original clean image.

The autoencoder learns the typical patterns from the training data. If the noise is different from what it saw, or the clean image was not in the training distribution, it cannot perfectly reconstruct it. It will produce a best-effort approximation. In fact: Denoising autoencoders learn to map corrupted inputs to clean versions, but the reconstruction quality depends on how well the clean images match what it learned.

**Training an autoencoder is just prediction, no labels needed** — Autoencoders are supervised because they need labels for training.

The target is the input itself, so no external labels are needed. This is unsupervised learning: the network learns from the structure of the data alone. In fact: Autoencoders are self-supervised: the input serves as its own label. No human-annotated labels are required.


## In short

An autoencoder is a network that learns to reconstruct its input through a smaller middle layer, forcing it to find a compressed representation of the data. The bottleneck is key: without it, the network would simply copy. The two parts are the encoder (input to bottleneck) and decoder (bottleneck to output). The loss measures how far the reconstruction is from the input, and training minimises it. Applications include anomaly detection, denoising, and creating embeddings. The main pitfall is choosing the right bottleneck size: too small loses information, too large loses the purpose.


## Where this is used

Autoencoders are used in several AI systems, though they are often overshadowed by newer models. Here are concrete places:

1. **Anomaly detection**: The most common modern use. You train an autoencoder on normal data (e.g., sensor readings from a healthy machine, or credit card transactions that are not fraudulent). The autoencoder gets good at reconstructing those normal patterns. When a new, unusual signal arrives, the reconstruction error will be high, because the autoencoder has never learned to encode that pattern into the small bottleneck and reproduce it. So the error itself becomes an anomaly score. This is used in industrial fault detection (matching the source book's theme), network intrusion detection, and medical image screening (e.g., detecting unusual growths in X-rays).

2. **Denoising**: A variant called a denoising autoencoder is trained by taking clean images and adding random noise, then feeding the noisy image as input and expecting the output to be the clean image. The autoencoder learns to ignore the noise and capture the underlying signal. This is used to clean up old photographs, remove sensor noise in cameras, and in audio processing to remove background hum.

3. **Dimensionality reduction and embeddings**: The bottleneck representation is a compressed vector of numbers that summarises the input. You can extract that vector (the middle layer) and use it as a feature for other tasks. This was a common way to create embeddings for images and text before modern transformer embeddings. For example, you could train an autoencoder on product descriptions and then use the bottleneck vector to find similar products or cluster them.

4. **Pre-training for deep networks**: Historically, autoencoders were used to pre-train layers of a deep network one at a time, because training deep networks directly was unstable. The idea was: train an autoencoder to reconstruct the input, take the encoder part, and use it as a good initialisation for a supervised network. This is less common now, but the principle of using unsupervised reconstruction to learn representations still appears in modern pretraining.

5. **In large language models**: Autoencoders as such are not the core of LLMs, but the idea of having a bottleneck (like the hidden state of a transformer) and reconstructing the input is similar. BERT, for example, is trained to reconstruct masked words, which is a form of denoising. The representation it learns is then used for many tasks. So the concept of learning a compressed understanding by reconstructing the input is deeply related.

6. **In small models on devices**: An autoencoder is small or can be made small, so it can run on a phone. For example, anomaly detection on a smartphone: you train an autoencoder on normal typing patterns, and if the user's typing on a new day gives a high reconstruction error, it might indicate a different person. Or on a smartwatch, an autoencoder can detect falls by learning normal motion patterns.

7. **In retrieval-augmented generation (RAG)**: RAG systems embed documents into a vector space and retrieve the closest ones to a query. Although modern RAG usually uses transformer embeddings, you could use an autoencoder to produce embeddings for the documents. The bottleneck vector is exactly that embedding. If you want a small, fast embedding on a low-power device, an autoencoder might be a choice.

8. **Agentic systems**: An agent tool that needs to determine if the current situation is normal or not (e.g., a server agent checking for anomalies) might use an autoencoder for anomaly detection. Also, planning and action loops often need to compress state information; an autoencoder could compress a large state into a small vector for the agent's memory.

**How a developer actually uses it**: In practice, a developer would use PyTorch or Keras. They would define a model like the one above, load a dataset of normal examples, train it to minimise the mean squared error between input and output, and then set a threshold for the reconstruction error. Every new sample is fed through; if the error exceeds the threshold, it is flagged as anomalous. The developer might also use the bottleneck as a feature vector for other models.

**What breaks when done badly**: If the bottleneck is too small, the autoencoder cannot capture the variety of normal data and will produce high errors even for normal cases. If it is too large, it can simply memorise the training data and produce low errors for everything, failing to detect anomalies. If you use a linear activation everywhere, the autoencoder can only represent linear relationships and may fail on complex patterns. And if you have too few training examples, the autoencoder may not generalise. In fraud detection, a model that flags too many legitimate transactions as anomalies is costly; so thresholds and validation are crucial.


---

Read this on the web: https://basavapurushottam.com/learn-ai/autoencoders
