# Encoder-Decoder

An architecture that compresses input into a representation and then generates output from it, separating understanding from production.

**Rests on:** Computational Graphs, Matrix Calculus and Jacobians

**Leads to:** TrOCR and OCR

*Deep Learning · Neural Network Architectures*


## Why learn this

Encoder-decoder is the architecture behind machine translation, text summarization, and image captioning. It's also the backbone of modern LLMs—the transformer encoder-decoder. Mastering this unlocks Transformers, Attention, and the ability to build systems that map any input sequence to any output sequence, regardless of length. It leads directly to further study of sequence-to-sequence models and attention mechanisms.


## The intuition

Think of the encoder-decoder as a two-stage process: first, read and understand; second, produce. The encoder reads the entire input and compresses it into a single, dense representation—a thought vector. This vector captures the meaning of the input, not its exact words. The decoder then takes this thought vector and generates the output step by step, word by word. The key insight is that the input and output lengths don't have to match. In translation, "How are you?" (3 words) becomes "Wie geht es dir?" (5 words). The encoder compresses the 3-word input into a thought vector, and the decoder expands it into a 5-word output. This separation of understanding from production is what makes the architecture so flexible. For example, in a chatbot, the encoder compresses the user's question, and the decoder generates a response. The encoder must understand the input completely before the decoder can start writing anything sensible.


## An analogy

Imagine a professional translator working at the UN. She listens to a speech in French (the input). She doesn't translate word-by-word as she hears it; instead, she forms a complete understanding of the meaning—the key points, the tone, the nuances. This understanding is her internal 'thought vector.' Only after the speaker has finished does she deliver the English translation, sentence by sentence, drawing on that full understanding. The encoder is the listening and comprehension phase. The decoder is the speaking phase. The thought vector is the complete mental picture. This analogy stops working, however, because a real translator retains far more detail than a fixed-length vector can hold. The encoder-decoder's thought vector is a compressed representation, so long inputs lose information. In practice, this 'bottleneck' is a real limitation—that's why attention mechanisms were invented. The translator doesn't forget the speech; she has access to all of it. The encoder-decoder without attention, by contrast, must remember everything in a single fixed-size vector, which is a hard constraint.


## Definition

An encoder-decoder is an architecture that uses one neural network (the encoder) to compress an input sequence into a fixed-length representation, and a second neural network (the decoder) to generate an output sequence from that representation, allowing input and output lengths to differ arbitrarily.


## How it connects

You have notes on LSTM Networks and the LSTM Cell State. The encoder and decoder are often built from LSTM or GRU cells, which process sequences step-by-step. The encoder's final hidden state is the thought vector that seeds the decoder. This also connects to your notes on Layer Normalization, which stabilizes training in deeper encoder-decoder stacks. The key difference from a simple sequence model is the two-stage split: one network reads, the other writes.


## The mathematics

This concept rests on linear algebra and a bit of calculus. Linear algebra handles the vectors (lists of numbers) that flow through the network, and calculus is used in training (adjusting the weights, but we don't need derivatives here). The encoder processes the input sequence step by step. At each step, it takes a word (represented as a vector) and combines it with the previous hidden state (another vector) to produce a new hidden state. The combining is a weighted sum of the input vector and the previous hidden state, passed through a non-linear function (like tanh or sigmoid). After processing all words, the last hidden state is the thought vector. The decoder starts with this thought vector as its initial hidden state. It then generates one word at a time: it takes the previous word and the current hidden state, produces a new hidden state, and then uses that hidden state to predict the next word's probability distribution over the vocabulary. The prediction is done by taking a weighted sum of the hidden state and a bias, and then applying a softmax to turn scores into probabilities. So the encoder's hidden states are a list of numbers, and each step's combination is a weighted sum.

Encoder: h_t = tanh(W_xh * x_t + W_hh * h_{t-1})
Decoder: s_t = tanh(W_xh * y_{t-1} + W_hh * s_{t-1})
Output probability: p(y_t) = softmax(W_hy * s_t)
Thought vector: c = h_T (the last encoder hidden state)


## A worked example

```
A tiny example with a vocabulary of {hello, world, goodbye} for input, and {hi, there} for output. We'll use 2-dimensional vectors for simplicity. Input sequence: [hello, world]. Encoder weights are fixed for this example.

Step 1: Encoder starts with hidden state h0 = [0, 0].   # The initial hidden state is a zero vector.
Step 2: Step 1: h1 = tanh(Wxh * x_hello + Whh * h0), with Wxh=[1,0;0,1], Whh=[0.9,0;0,0.8], x_hello=[1,0] → h1 = tanh([1,0]) = [0.76, 0].   # Combine word vector with previous hidden state.
Step 3: Step 2: h2 = tanh([0,1] + [0.9*0.76, 0.8*0]) = tanh([0.68, 1]) = [0.59, 0.76].   # This h2 is the thought vector representing the whole input.
Step 4: Decoder starts with s0 = thought vector = [0.59, 0.76].
Step 5: Decoder step 1: predict output word. We compute scores = W_hy * s0 = [0.5, 0.5; -0.2, 0.3] * [0.59, 0.76] = [0.675, 0.11]. Apply softmax: exp(0.675)/(exp(0.675)+exp(0.11)) = 0.64, so pick 'hi' (index 0).   # The decoder produces a probability distribution over output vocabulary.
Step 6: Decoder step 2: use 'hi' vector [1,0] and s0 to get s1 = tanh(Wxh * x_hi + Whh * s0) ≈ tanh([1,0] + [0.53, 0.61]) = tanh([1.53, 0.61]) = [0.91, 0.54]. Then predict next word: score = W_hy * s1 = [0.5*0.91+0.5*0.54, -0.2*0.91+0.3*0.54] = [0.725, -0.02]. Softmax → 'there' (index 1).   # Continue generating until an end token is emitted.

Result: The decoder generates the sequence ['hi', 'there']. The input was two words, the output is two words, but the architecture can handle any lengths. The thought vector was [0.59, 0.76].
```


## From scratch, in Python

```python
# A minimal encoder-decoder for a tiny vocab: input {hello, world, goodbye}, output {hi, there, <end>}
# We use 2-dimensional vectors for everything.

import math

# Vocabulary
input_vocab = {'hello': 0, 'world': 1, 'goodbye': 2}
output_vocab = {'hi': 0, 'there': 1, '<end>': 2}

# Word vectors (one-hot for simplicity)
input_vectors = {'hello': [1, 0], 'world': [0, 1], 'goodbye': [0, 0]}  # note: 'goodbye' is [0,0] for simplicity
output_vectors = {'hi': [1, 0], 'there': [0, 1], '<end>': [0, 0]}

# Weights (identity-ish for encoder, simple for decoder)
W_xh = [[1, 0], [0, 1]]  # maps input vector to hidden space
W_hh = [[0.9, 0], [0, 0.8]]  # maps previous hidden to next hidden

W_hy = [[0.5, 0.5], [-0.2, 0.3]]  # maps decoder hidden to output scores

# Activation functions
def tanh(x):
    return [math.tanh(v) for v in x]

def softmax(scores):
    exps = [math.exp(s) for s in scores]
    total = sum(exps)
    return [e / total for e in exps]

# Matrix-vector multiplication (just for tutorial)
def mat_vec_mul(M, v):
    return [sum(M[i][j] * v[j] for j in range(len(v))) for i in range(len(M))]

def vector_add(v1, v2):
    return [v1[i] + v2[i] for i in range(len(v1))]

# Encoder
def encode(input_words):
    h = [0, 0]
    for word in input_words:
        x = input_vectors[word]
        # h = tanh(W_xh * x + W_hh * h)
        h = tanh(vector_add(mat_vec_mul(W_xh, x), mat_vec_mul(W_hh, h)))
    return h  # thought vector

# Decoder
def decode(thought_vector, max_length=10):
    s = thought_vector
    output = []
    for _ in range(max_length):
        # predict next word from current state
        scores = mat_vec_mul(W_hy, s)
        probs = softmax(scores)
        pred_index = probs.index(max(probs))
        if pred_index == output_vocab['<end>']:
            break
        # look up the word
        word = [k for k, v in output_vocab.items() if v == pred_index][0]
        output.append(word)
        # update state using the generated word (as vector)
        y = output_vectors[word]
        s = tanh(vector_add(mat_vec_mul(W_xh, y), mat_vec_mul(W_hh, s)))
    return output

# Example
thought = encode(['hello', 'world'])
print("Thought vector:", thought)
print("Decoded:", decode(thought))
```


## The same thing, with a framework

In PyTorch, you would use nn.LSTM for the encoder and decoder, or a Transformer. Underneath, it's the same idea: the encoder processes the input sequence and returns the final hidden state, which is passed as the initial state to the decoder. The framework handles the matrix multiplications and backpropagation for you. For example:

```python
import torch.nn as nn
encoder = nn.LSTM(input_size=10, hidden_size=20, batch_first=True)
decoder = nn.LSTM(input_size=10, hidden_size=20, batch_first=True)
output, (h_n, c_n) = encoder(input_tensor)
decoder_output, _ = decoder(decoder_input, (h_n, c_n))
```


## What the code does

The code defines a simple encoder-decoder with fixed weights. We start with the vocabulary and vector representations. The `encode` function iterates through the input words, updating a hidden state `h` using the weighted sum and tanh. After the loop, `h` is the thought vector. The `decode` function starts with that thought vector as the state `s`. It then predicts the next word by applying the output weight matrix and softmax. It selects the word with the highest probability, appends it to the output, and updates the state using that predicted word. This continues until an end token is predicted or a max length is reached. The matrix-vector multiplication and vector addition are written out explicitly for clarity. In practice, you'd use library functions like `torch.matmul`. The key line is `h = tanh(vector_add(mat_vec_mul(W_xh, x), mat_vec_mul(W_hh, h)))` — this is the same formula from the mathematics section.


## Common misconceptions

**The encoder and decoder must have the same input and output length.** — The encoder and decoder must have the same input and output length because they're connected, so how can they be different?

The thought vector is a fixed-size summary, so the encoder can take any number of inputs and the decoder can emit any number of outputs. The lengths are decoupled by the thought vector. In fact: The thought vector breaks the length constraint. The encoder compresses any length into a fixed vector, and the decoder expands that vector into any length.

**The thought vector stores the entire input verbatim.** — The thought vector stores the entire input verbatim, so I can reconstruct the input exactly from it.

The thought vector is a compressed, lossy representation. It captures the meaning, not the exact words. You cannot reconstruct the exact input from it. In fact: The thought vector is a learned summary, not a copy. It stores the essential information needed for the task, discarding details that don't matter.

**Encoder-decoder is only for text translation.** — Since I've only seen it used for translation, encoder-decoder must only work on text.

The encoder and decoder are neural networks that can process any input type as long as it can be represented as vectors. Images, audio, video, and text can all be encoded and decoded. In fact: Encoder-decoder is modality-agnostic. The encoder can be a CNN for images, the decoder can be an RNN for text—this is how image captioning works.


## In short

The encoder-decoder architecture consists of two neural networks: the encoder compresses a variable-length input into a fixed-length thought vector, and the decoder expands that vector into a variable-length output. This separates understanding from production, allowing input and output lengths to differ arbitrarily. It is the backbone of translation, summarization, and many modern AI systems. If you remember one thing: the thought vector is a compressed meaning, not a copy.


## Where this is used

Encoder-decoder is the foundation of sequence-to-sequence models. In large language models, the transformer uses an encoder-decoder structure: the encoder reads the prompt, and the decoder generates the response. For translation, Google Translate uses a transformer encoder-decoder: the encoder processes the source sentence, and the decoder generates the target. The mechanism is exactly what we described: the encoder compresses the meaning, the decoder expands it. In retrieval-augmented generation (RAG), an encoder-decoder can be used to encode the retrieved documents into a representation that the decoder uses to generate an answer grounded in that context. For small models on devices, like keyword spotting on a smartphone, a small encoder-decoder can be used to compress audio features into a compact representation, and the decoder reconstructs or classifies them. In agentic systems, an encoder-decoder can be used to encode a plan or a set of tool outputs into a representation that guides the next action. In everyday applied ML, encoder-decoder is used in image captioning (encode the image with a CNN, decode with an LSTM) and in speech recognition (encode audio, decode to text). A developer typically uses a high-level library like Hugging Face's Transformers. They instantiate a model like `T5ForConditionalGeneration`, tokenize the input, pass it to `model.generate()`, and the framework handles the encoding and decoding. What breaks when done badly: if you use a fixed encoder, it can't handle inputs longer than seen in training; if the thought vector is too small, it becomes a bottleneck and information is lost. Attention mechanisms were introduced to fix exactly this.


---

Read this on the web: https://basavapurushottam.com/learn-ai/encoder-decoder
