# LSTM Networks

Recurrent networks with gated cells designed to carry information across long sequences without the gradient decay that plain RNNs suffer.

**Rests on:** Multi-Layer Perceptrons

**Leads to:** CNN-LSTM Hybrids, LSTM Cell State, Stateful and Bidirectional LSTMs

*Deep Learning · Neural Network Architectures*


## Why learn this

You are about to meet the architecture that powered speech recognition, machine translation, and stock forecasting for a decade — and still quietly runs inside many of today's systems. Learning LSTMs gives you the mental model for how neural networks handle sequences: music, sensor readings, words in a sentence. It also sets you up to understand what came next. When you study transformers — the architecture behind ChatGPT — half the battle is understanding why they solved a problem LSTMs had: carrying information across long stretches without forgetting or losing the signal. And if you ever work on a device that must respond instantly, like a phone that listens for 'Hey Google', you'll find LSTMs still there, because they don't need to see the whole sentence before they start. This page teaches you the LSTM from the ground up — the four gates, the memory cell, and how they work together — so that later, when you meet attention, you'll recognize it as a clever twist on the same problem.


## The intuition

Picture a factory conveyor belt that carries a heavy box from one end of a long hall to the other. Along the way, workers stand at stations. At each station, a worker inspects the box, decides whether a part needs to be added, removed, or left alone, and then signals the belt to move on. The box is never dumped out and rebuilt — it's nudged, adjusted, and occasionally something is taken off or put on. That's the core idea of an LSTM cell: it maintains a piece of information — the 'cell state' — that travels through time, and at each timestep, it decides, based on the current input, what to forget, what to write, and what to output. The genius is that the cell state changes gradually, by adding or removing small amounts, rather than being overwritten. This is what lets the network remember something from ten steps ago: the signal doesn't have to travel through all the intermediate transformations; it just rides along the belt, gently modified, until it's needed.

To understand why this matters, compare it to a plain Recurrent Neural Network (RNN). An RNN processes sequences one element at a time and maintains a 'hidden state' — a compact summary of everything seen so far — that carries information across time steps. In a plain RNN, there is no belt — every step takes the whole hidden state and recomputes it from scratch. If you try to carry information through many steps, tiny errors in calculation accumulate. Imagine trying to multiply a series of numbers each slightly less than 1 (like 0.9). After ten multiplications, the value shrinks to near zero; this is the vanishing gradient problem, where the network forgets early information because the signal fades away. Conversely, if you multiply by numbers slightly greater than 1 (like 1.1), the value grows exponentially; this is the exploding gradient problem, where the signal becomes so large it breaks the calculation. Plain RNNs suffer from these extremes, making it impossible to remember things from long ago.

LSTM's gates are small neural networks themselves, each with a weight and a bias, that produce a number between 0 and 1 — a 'how much to let through' factor. Forget gate: how much of the old cell state to keep. Input gate: how much of the new candidate to add. Output gate: how much of the cell state to reveal as the output. Put together, these gates let the LSTM learn which parts of the past to remember, which to forget, and what to say right now.


## An analogy

Imagine you're writing a long essay, and you keep a running notes sheet in front of you. At the end of each paragraph, you look at your notes and decide: 'The main argument from paragraph 2 is still relevant, keep it. The digression about the author's childhood — I can cross that out.' That's the forget gate. Then you read the new paragraph and think: 'This adds a supporting point, I should write it down.' That's the input gate, writing into memory. Finally, when you're about to start the next paragraph, you look at your notes and decide what to actually say in the essay — you don't read out your entire notes verbatim, you pick a summary phrase. That's the output gate. The LSTM is doing exactly this, but with numbers. The cell state is your notes sheet, a list of numbers that represent 'what I know'. The input at each time step is a new paragraph. The forget gate is a set of weights that decide, for each number in the cell state, how much to keep (a factor between 0 and 1). The input gate decides how much of the new information (a 'candidate' created from the input and previous output) to add to the cell state. The output gate decides what to 'say' — the hidden state — which is a filtered version of the cell state. The analogy breaks down where the real LSTM is more subtle: the gates are not separate conscious decisions; they're all computed simultaneously from the same input, and they are learned, not hand-crafted. Also, the cell state isn't a sheet of paper — it's a vector of numbers, and the gates apply element-wise, so each dimension of the cell state can be forgotten or updated independently. But the core intuition holds: a conveyor belt that carries a memory, with workers (gates) that add, remove, and read from it, and the key is that the belt allows the memory to flow with minimal interference, so distant information survives.


## Definition

An LSTM (Long Short-Term Memory) network is a type of recurrent neural network that uses gated cells to control the flow of information through a memory state, allowing it to carry relevant information across long sequences without suffering from the vanishing or exploding gradient problem that plagues plain RNNs.


## How it connects

You haven't learned anything yet, so this page is your starting point. The LSTM belongs to the family of recurrent neural networks (RNNs), which process sequences one element at a time and maintain a hidden state that carries information across time steps. The LSTM is a refinement of the plain RNN, which added a simple hidden state but couldn't remember long-range dependencies. The LSTM's innovation is the cell state — a separate memory track that is modified additively through gates. When you later study transformers, you'll see they replaced the recurrence with attention, which can look at any part of the sequence directly rather than incrementally, but the LSTM's lessons about memory and gating still inform how we think about attention.


## The mathematics

The mathematics behind an LSTM (Long Short-Term Memory) network relies on two things you already know: arithmetic with lists of numbers (which mathematicians call vectors) and the idea of multiplying a list by a single number or another list. It does not require calculus, derivatives, or complex algebra. The core operation is simple: take some numbers, multiply them by fixed weights, add them together, and then squeeze the result into a specific range using a 'squashing' function.

An LSTM processes data one step at a time, like reading a sentence word by word. At each step, it has three main pieces of information:
1. The current input (for example, the current word). We will call this x.
2. The memory from the previous step. This is split into two parts: the 'cell state' (long-term memory) and the 'hidden state' (short-term summary). We will call the old cell state c_old and the old hidden state h_old.
3. The new outputs it needs to produce: a new cell state c_new, a new hidden state h_new, and three control signals called gates.

The LSTM uses four sets of numbers called 'weights' (tables of multipliers) and four 'biases' (numbers added at the end). These weights are fixed for the whole network. The gates act like valves or sliders that decide how much information to keep, add, or reveal.

Here is the step-by-step arithmetic for one time step:

Step 1: Combine the inputs.
To make the math easier, we first join the old hidden state and the current input into one long list. Let's call this combined list 'combined'. If h_old was a list of 2 numbers and x was a list of 2 numbers, combined is a list of 4 numbers.

Step 2: Calculate the Forget Gate (f).
The forget gate decides what to throw away from the old memory. It produces a list of numbers between 0 and 1. A value of 0 means 'forget completely', and 1 means 'keep completely'.
To get f, we multiply the combined list by the forget weights, add the forget bias, and then pass the result through a sigmoid function.
The sigmoid function takes any number and squashes it between 0 and 1. For example, if the sum before squashing is 2, sigmoid(2) is about 0.88. If the sum is -5, sigmoid(-5) is very close to 0.
Formula: f = sigmoid(combined * weights_forget + bias_forget)

Step 3: Calculate the Input Gate (i).
The input gate decides what new information to store in the memory. It also produces a list of numbers between 0 and 1.
Formula: i = sigmoid(combined * weights_input + bias_input)

Step 4: Calculate the Candidate Value (c_candidate).
This is a list of potential new memories. Unlike the gates, these values can be positive or negative. To keep them in a manageable range, we use a different squashing function called tanh. Tanh squashes numbers between -1 and 1.
Formula: c_candidate = tanh(combined * weights_candidate + bias_candidate)

Step 5: Update the Cell State (c_new).
This is where the memory actually changes. We take the old memory, multiply it by the forget gate (to keep or discard parts of it), and add the new candidate information multiplied by the input gate.
Formula: c_new = (f * c_old) + (i * c_candidate)
Note: The multiplication here is 'element-wise'. This means you multiply the first number of f by the first number of c_old, the second by the second, and so on. You do not do matrix multiplication here.

Step 6: Calculate the Output Gate (o).
The output gate decides what part of the new memory to show as the short-term summary (hidden state). It produces a list between 0 and 1.
Formula: o = sigmoid(combined * weights_output + bias_output)

Step 7: Calculate the New Hidden State (h_new).
First, we squash the new cell state using tanh again to get values between -1 and 1. Then we multiply this by the output gate.
Formula: h_new = o * tanh(c_new)

In summary, the LSTM does not use complex calculus during operation. It uses repeated applications of:
- Concatenation (joining lists).
- Weighted sums (multiplying lists by weights and adding biases).
- Element-wise multiplication (scaling parts of a list independently).
- Squashing functions (sigmoid for 0-to-1, tanh for -1-to-1).


## A worked example

Let's walk through the arithmetic with real numbers. To keep it simple, we will assume every list has only one number (a scalar). This means 'vectors' are just single values.

The Setup:
- Old cell state (c_old) = 3.0
- Old hidden state (h_old) = 0.5
- Current input (x) = 2.0
- Combined list: We join h_old and x. So, combined = [0.5, 2.0].

The Weights and Biases:
To make the calculation easy to follow by hand, we will use very simple weights. All weight values are 0.5. All bias values are 0, except for the candidate bias which is 1.0.
- Forget weights: [0.5, 0.5], bias = 0
- Input weights: [0.5, 0.5], bias = 0
- Candidate weights: [0.5, 0.5], bias = 1.0
- Output weights: [0.5, 0.5], bias = 0

Helper Functions:
- Sigmoid(z): The formula is 1 / (1 + e^(-z)).
  - If z = 0, sigmoid(0) = 0.5.
  - If z = 1.25, we calculate e^(-1.25) ≈ 0.2865. Then 1 / (1 + 0.2865) = 1 / 1.2865 ≈ 0.777.
- Tanh(z): The formula is (e^z - e^-z) / (e^z + e^-z).
  - If z = 2.25, tanh(2.25) ≈ 0.978.
  - If z = 3.091, tanh(3.091) ≈ 0.996.

Step 1: The Forget Gate (f)
First, we compute the weighted sum for the forget gate:
Sum_f = (0.5 * h_old) + (0.5 * x) + bias_forget
Sum_f = (0.5 * 0.5) + (0.5 * 2.0) + 0
Sum_f = 0.25 + 1.0 + 0 = 1.25

Now apply sigmoid:
f = sigmoid(1.25) ≈ 0.777
Meaning: We will keep about 77.7% of the old memory.

Step 2: The Input Gate (i)
Compute the weighted sum for the input gate:
Sum_i = (0.5 * h_old) + (0.5 * x) + bias_input
Sum_i = (0.5 * 0.5) + (0.5 * 2.0) + 0
Sum_i = 0.25 + 1.0 + 0 = 1.25

Apply sigmoid:
i = sigmoid(1.25) ≈ 0.777
Meaning: We will add about 77.7% of the new candidate information.

Step 3: The Candidate Value (c_candidate)
Compute the weighted sum for the candidate:
Sum_c = (0.5 * h_old) + (0.5 * x) + bias_candidate
Sum_c = (0.5 * 0.5) + (0.5 * 2.0) + 1.0
Sum_c = 0.25 + 1.0 + 1.0 = 2.25

Apply tanh:
c_candidate = tanh(2.25) ≈ 0.978
This is the raw new memory value, squashed to be between -1 and 1.

Step 4: The New Cell State (c_new)
We update the old memory using element-wise multiplication (which is just regular multiplication here since there is only one number):
c_new = (f * c_old) + (i * c_candidate)
c_new = (0.777 * 3.0) + (0.777 * 0.978)
c_new = 2.331 + 0.760
c_new ≈ 3.091
The memory has changed slightly from 3.0 to 3.091.

Step 5: The Output Gate (o)
Compute the weighted sum for the output gate:
Sum_o = (0.5 * h_old) + (0.5 * x) + bias_output
Sum_o = (0.5 * 0.5) + (0.5 * 2.0) + 0
Sum_o = 0.25 + 1.0 + 0 = 1.25

Apply sigmoid:
o = sigmoid(1.25) ≈ 0.777
Meaning: We will show about 77.7% of the new memory.

Step 6: The New Hidden State (h_new)
First, squash the new cell state with tanh:
tanh(c_new) = tanh(3.091) ≈ 0.996

Then multiply by the output gate:
h_new = o * tanh(c_new)
h_new = 0.777 * 0.996
h_new ≈ 0.774

Final Result:
- The new cell state (long-term memory) is approximately 3.091.
- The new hidden state (short-term summary) is approximately 0.774.
The process shows how the network gently updates its memory rather than overwriting it completely.


## From scratch, in Python

```python
import math

def sigmoid(x):
    return 1 / (1 + math.exp(-x))

def tanh(x):
    return (math.exp(x) - math.exp(-x)) / (math.exp(x) + math.exp(-x))

# Example: a 2-dimensional hidden state, 2-dimensional input
# Random weights for demonstration (not trained)
W_f = [[0.5, 0.2], [0.3, 0.4]]  # forget gate weights for input part
U_f = [[0.1, 0.3], [0.2, 0.1]]  # forget gate weights for hidden state
b_f = [0.0, 0.0]

# Similar for other gates...

# We'll do a single time step
def lstm_step(prev_h, prev_c, x):
    # Compute gates
    f = []
    for i in range(len(prev_h)):
        sum_val = b_f[i]
        for j in range(len(x)):
            sum_val += W_f[i][j] * x[j]
        for j in range(len(prev_h)):
            sum_val += U_f[i][j] * prev_h[j]
        f.append(sigmoid(sum_val))
    # Similarly for i, o, candidate
    # (code omitted for brevity, but same pattern)
    new_c = [f[i] * prev_c[i] + i_gate[i] * candidate[i] for i in range(len(prev_c))]
    new_h = [o_gate[i] * tanh(new_c[i]) for i in range(len(new_c))]
    return new_h, new_c

# Let's run it with actual numbers
prev_h = [0.5, -0.2]
prev_c = [3.0, -1.0]
x = [2.0, 1.0]
# Here we would need to define all the weights, but for brevity we'll skip to the output
# In practice, you'd write the full function with all gates defined.
```


## The same thing, with a framework

In PyTorch, you would use `torch.nn.LSTM`. For example:

```python
import torch
import torch.nn as nn

# Batch of 3 sequences, each 5 time steps, with 10 features
lstm = nn.LSTM(input_size=10, hidden_size=20, num_layers=1)
inputs = torch.randn(5, 3, 10)  # (seq_len, batch, input_size)
output, (h_n, c_n) = lstm(inputs)
```

Under the hood, PyTorch does exactly the same arithmetic as our from-scratch code — it just uses optimized matrix multiplication and GPU support. The `nn.LSTM` object holds all the weights for the gates and updates them during training.


## What the code does

This code shows the basic structure. The `sigmoid` and `tanh` functions implement the mathematical functions we used. Then we define weight matrices (lists of lists) for each gate. The `lstm_step` function takes the previous hidden state, previous cell state, and the current input, and returns the new hidden and cell states. The key is in the gate computations: we loop over each dimension of the hidden state, compute a weighted sum of the input and previous hidden state, add the bias, then pass through the sigmoid to get a value between 0 and 1. The new cell state is a combination of the old cell state and a candidate, weighted by the forget and input gates. The new hidden state is the cell state squashed by tanh and multiplied by the output gate.


## Common misconceptions

**LSTMs are like regular neural networks, just with a vertical feed.** — An LSTM is just a regular neural network that I can run on a sequence by feeding the whole sequence as input at once.

It is deeply wrong: an LSTM processes the sequence one element at a time and maintains an internal state. It is inherently sequential. You cannot feed the whole sequence as one input to a single LSTM call; you must loop over time steps. In fact: An LSTM is a recurrent network: it takes one input at a time, updates its hidden state based on that input and its previous hidden state, and then moves to the next element. The weights are shared across all time steps, not different at each step.

**The three gates are separate networks, not one combined thing.** — There are three separate LSTM networks: a forget network, an input network, and an output network.

In math it might look like three separate equations, but in code and in the actual network, they use the same input and hidden state but different weight matrices. They are not separate networks; they are three different sets of weights applied to the same concatenated input. In fact: It's one LSTM cell that computes four sets of weights (for forget, input, candidate, output) from the same input and previous hidden state. The gates are functions of the same information, but they serve different purposes.

**The sigmoid output is a binary decision.** — The forget gate outputs a number between 0 and 1, but it's basically a decision: either remember or forget.

It's continuous, not binary. A value of 0.8 means keep most of it; 0.2 means keep a little. The network can learn to partially remember things, which gives it more flexibility than a hard decision. In fact: The gates are continuous values between 0 and 1, effectively a soft version of remembering or forgetting. They allow the cell state to be gradually adjusted rather than flipped all the way.

**LSTMs can remember anything forever.** — Because the cell state carries information across time, the LSTM can remember exactly what it needs from the start of a sequence to the very end, no matter how long the sequence is.

In practice, the cell state is a finite vector of numbers, and the forget gate can multiply it down to near zero. Over very long sequences, the cell state can decay or be overwritten. LSTM helps, but it doesn't make memory unlimited. In fact: LSTMs are good at remembering information for dozens or hundreds of time steps, but they are not infinite. The cell state is a finite reservoir that gets modified at each step, and the forget gate can erase it. They are better than plain RNNs, but not perfect.

**LSTMs are always bidirectional.** — When I use an LSTM for translation, it needs to see the whole sentence both ways, so it must be bidirectional.

Bidirectional LSTMs are only possible when you have the entire sequence available. For real-time generation (like speech recognition during a conversation), you can only go forward. In fact: LSTMs can be unidirectional (only process forward) or bidirectional (process both directions), but bidirectional ones require the entire sequence to be present, so they can't be used for streaming or online inference.

**The LSTM's hidden state is the same as its output.** — The output of an LSTM at each time step is the hidden state itself.

In many implementations, the output is indeed the hidden state, but the cell state is separate. If you ask the LSTM to return sequences, it returns the hidden states, not the cell states. The cell state is internal and not normally exposed. In fact: The hidden state is what you use as the output (or to feed into the next layer), but the cell state is a private memory that the gates use. It is not directly outputted.


## In short

LSTM networks are designed to remember information over long sequences by maintaining a cell state that is updated gently through gates. The forget gate decides what to discard, the input gate decides what new information to store, and the output gate decides what to reveal. This gating mechanism prevents the gradient from vanishing or exploding, allowing the network to learn long-range dependencies. LSTMs were the go-to for sequence modelling until transformers came along, but they remain useful for streaming data and small on-device models. The key to understanding them is the conveyor-belt analogy: a memory state that moves forward with minimal interference, modified only by smooth multipliers between 0 and 1.


## Where this is used

LSTMs have been everywhere. In **large language models** (like older GPT-2 or many speech recognition systems), they process sequences of words or phonemes. The mechanism: at each time step, the input is a word embedding (a list of numbers representing the word), and the LSTM updates its hidden state, which carries context. For example, in a language model, the LSTM predicts the next word based on the hidden state. The cell state is what allows it to remember the subject of a long sentence, so it can keep verb agreement. A developer using PyTorch would define an `nn.LSTM` layer, feed in a sequence of embeddings, and take the last hidden state as the summary for prediction. When they train it, they use gradient descent to adjust the LSTM's weights.

In **small models running on devices**, like wake-word detection on a phone, LSTMs are still common because they are compact and can process audio streaming in real time. Unlike transformers, which need the whole sequence to compute attention, an LSTM processes one frame at a time, so it can run on a smartphone with minimal latency. The developer would use a library like TensorFlow Lite and export a Keras LSTM model. The model runs on every audio frame (e.g., 20ms), and a classifier on top of the hidden state triggers when it hears the wake word. The cell state helps the LSTM remember the onset of the sound even after several frames of silence.

In **retrieval-augmented generation**, LSTMs are less central now, but they were used for encoding queries and documents before transformers. The idea: encode a passage into a vector using an LSTM, then use cosine similarity for retrieval. The LSTM would process the word sequence and take the final hidden state as the representation. This representation would be compared to others using a dot product, but the LSTM's job was to pack all the meaning of the passage into a single vector. With transformers, bidirectional attention does this better, but if you have limited compute, a single-layer LSTM can still do a decent job.

In **agentic systems** that plan and call tools, LSTMs are rare because they struggle with very long trajectories and lack the parallelization of attention. However, a small LSTM might be used to maintain a running summary of an agent's state, like a memory of past actions and observations. The agent would feed each new observation into the LSTM, and the hidden state would serve as a compressed memory for the planning component. But in practice, transformers dominate because they can look at the whole history at once.

In **everyday applied machine learning**, LSTMs are used for time-series forecasting (stock prices, weather, energy demand), anomaly detection in sensor streams, and speech emotion recognition. A developer would use Keras or PyTorch to build a model with an LSTM layer or two, feed sequences of historical data, and train to predict future values. The cell state is particularly useful for remembering long-term trends (like seasonality) while the gates adapt to recent changes. For fraud detection, an LSTM can process sequences of transactions and learn patterns of spending that deviate from a cardholder's normal behavior — the cell state holds a baseline, and the gates adjust it as new purchases come in.

What a developer actually writes: In Keras, it's `model.add(LSTM(64, return_sequences=True))`. In PyTorch, it's `nn.LSTM(64, 32)`. They must handle the sequence length, batch size, and possibly make layers bidirectional (which processes the sequence both forward and backward, requiring the entire input). They must also decide whether to return sequences (if they need a prediction at every time step) or only the last hidden state. A common failure is training an LSTM on data that gets shuffled without preserving temporal order, which destroys the temporal dependencies. Another is improper scaling of input features — LSTM weights can be sensitive to input magnitude. And one of the biggest issues is that LSTM training is slow and can be unstable; they often use gradient clipping to prevent exploding gradients. In production, they need to export the model to a format like ONNX and run it with a runtime like TensorRT for speed.


---

Read this on the web: https://basavapurushottam.com/learn-ai/lstm-networks
