# LSTM Cell State

The carried vector running through an LSTM across time steps, modified only by gentle gated operations.

**Rests on:** LSTM Networks

**Leads to:** Stateful and Bidirectional LSTMs

*Deep Learning · Neural Network Architectures*


## Why learn this

You have likely faced the problem of remembering details from the beginning of a long document, conversation, or event by the time you reach the end. In standard computer systems, data passed through many steps often fades away or gets distorted beyond recognition. The LSTM cell state solves this by providing a dedicated, stable path for information to travel unchanged across hundreds or thousands of steps. This is crucial for tasks like predicting the next word in a long sentence, analyzing a long video clip, or forecasting stock prices where today's value depends on data from weeks ago. Without this mechanism, computers effectively 'forget' early inputs before making decisions based on late ones.


## The intuition

Imagine a long, straight hallway with no doors, only a few specific stations along the wall. This hallway represents the **cell state**. It is a continuous path where information travels intact from one end to the other.

At each station (which happens at every time step, or moment in time), there is a guard who can interact with the information in the hallway, but cannot delete the hallway itself. The guard uses three specific actions, controlled by internal checks called 'gates':

1.  **The Forget Check:** The guard looks at what is currently in the hallway and decides what to ignore or remove. If a piece of information is no longer relevant, the guard erases it from this spot in the hall.
2.  **The Input Check:** The guard looks at new information arriving at the station. They decide which parts of this new data are important enough to add to the hallway, appending them to the existing path.
3.  **The Output Check:** Finally, the guard decides what part of the hallway's content should be shown to the rest of the world (the next stage of processing). This shown portion is called the **hidden state**.

Crucially, the hallway itself does not change just because time passes. Information stays in it unchanged until a guard explicitly modifies it. Because the path is open and direct, information can travel from step 1 to step 100 without degrading. In other architectures without this dedicated hallway, information must pass through many complex filters at each step, and tiny errors accumulate until the original signal is lost—a problem known as 'vanishing gradients' (where the teaching signal becomes too weak to adjust the system). The cell state bypasses this by keeping a clear, unobstructed line of sight for critical data.


## An analogy

Think of a persistent group chat history on your phone that never deletes old messages unless you explicitly archive them. As new messages arrive, you might choose to ignore some (forget), add replies or reactions to others (input), and occasionally highlight specific past messages when someone asks what was decided last week (output). The chat log itself remains intact and accessible no matter how many new messages are added above it. You can scroll back days or weeks later and find the original context exactly as it was, because the storage mechanism (the phone’s database) didn’t alter the old entries; it just kept them separate while new ones accumulated on top.

The caveat is that in real life, you can accidentally delete a message history, but in the LSTM cell state, the structure itself is mathematically preserved unless the gates are deliberately programmed to change it.


## Definition

The LSTM cell state is a dedicated pathway within a neural network node that carries information across time steps with minimal alteration, allowing the network to retain long-term dependencies by selectively adding or removing data through controlled gate mechanisms.


## How it connects

This concept sits next to **RNN** (Recurrent Neural Network), which is a general type of network designed for sequence data but lacks the cell state’s stable hallway, causing it to lose information over long stretches. It also relates to **Residual connections**, a technique in deep learning where information skips layers and goes directly forward, similar to how the cell state allows information to bypass complex transformations, though residual connections are used in very different types of networks like those for image recognition.


## The mathematics

The cell state is a vector of numbers, one for each dimension of the state. At each time step, it is updated using three gates. Each gate is computed as a weighted sum of the current input and the previous hidden state, passed through a sigmoid (forget and input gates) or tanh (for a candidate update). The forget gate decides how much of the old cell state to keep (multiplying the old state element-wise by a number between 0 and 1). The input gate decides how much of a new candidate vector (computed from input and hidden state) to add to the cell state. Finally, the output gate decides how much of the cell state to expose as the hidden state, by applying tanh to the cell state and multiplying by the output gate.<br><br>The branch of mathematics used here is **linear algebra** (vectors, element-wise multiplication, and addition) plus **calculus** for training (since gradients flow through these operations). But for this concept, we only need arithmetic: multiplying numbers, adding them, and applying simple functions like the sigmoid.

C_t = f_t ⊙ C_{t-1} + i_t ⊙ \tilde{C}_t


## A worked example

```
Suppose the cell state has just one number for simplicity (in reality it has hundreds). Let the initial cell state C_0 = 0.0. At the first time step, we have input x_1 = 2, and previous hidden state h_0 = 0. We'll have some weights (just numbers) that compute the gates.

Step 1: forget gate: f_1 = sigmoid(W_f * h_0 + U_f * x_1 + b_f) = sigmoid(0.1 * 0 + 0.5 * 2 + 0.2) = sigmoid(1.2) ≈ 0.768   # The forget gate value is between 0 and 1, indicating how much of the old cell state to keep.
Step 2: input gate: i_1 = sigmoid(W_i * h_0 + U_i * x_1 + b_i) = sigmoid(0.0 * 0 + 0.8 * 2 - 0.1) = sigmoid(1.5) ≈ 0.818   # The input gate decides how much of the new candidate to add.
Step 3: candidate: tilde_C_1 = tanh(W_c * h_0 + U_c * x_1 + b_c) = tanh(0.0 * 0 + 0.3 * 2 + 0.0) = tanh(0.6) ≈ 0.537   # The candidate is a new vector of values from the input and hidden state, passed through tanh to keep it between -1 and 1.
Step 4: new cell state: C_1 = f_1 * C_0 + i_1 * tilde_C_1 = 0.768 * 0.0 + 0.818 * 0.537 ≈ 0.439   # The old cell state is scaled by the forget gate (here, 0 times 0.768 is 0, so nothing is kept) and the new candidate is added, scaled by the input gate.

Result: The new cell state C_1 is approximately 0.439. This number carries forward to the next time step, where it will be further modified by the gates. In a real LSTM, this vector has many dimensions (e.g., 256 or 512), and each dimension is updated similarly.
```


## From scratch, in Python

```python
# We'll implement the LSTM cell state update for a single time step,
# using just numbers and lists, no frameworks.

# Inputs: a single input value (for simplicity), previous hidden state (a list),
# and previous cell state (a list). We'll use size 1 for clarity.

# Let's define the sigmoid function and tanh function (approximation for tanh)
def sigmoid(x):
    return 1 / (1 + 2.71828 ** (-x))

def tanh_approx(x):
    # tanh(x) = (e^x - e^-x) / (e^x + e^-x)
    e_pos = 2.71828 ** x
    e_neg = 2.71828 ** (-x)
    return (e_pos - e_neg) / (e_pos + e_neg)

# We'll use weight values (just numbers) as examples
# Weights for gates: each gate has a weight for hidden state and a weight for input
# For simplicity, we ignore biases and use size-1 lists.

prev_h = [0.0]  # previous hidden state
prev_c = [0.0]  # previous cell state
x = 2.0  # current input

# Gate computations (using the numbers from the worked example)
# Forget gate: sigmoid(0.1*prev_h + 0.5*x)
f_gate = sigmoid(0.1 * prev_h[0] + 0.5 * x)
# Input gate: sigmoid(0.0*prev_h + 0.8*x - 0.1)
i_gate = sigmoid(0.0 * prev_h[0] + 0.8 * x - 0.1)
# Candidate: tanh(0.0*prev_h + 0.3*x)
candidate = tanh_approx(0.0 * prev_h[0] + 0.3 * x)

# Update cell state: C_new = f * old_c + i * candidate
new_c = [f_gate * prev_c[0] + i_gate * candidate]

print("Forget gate:", f_gate)
print("Input gate:", i_gate)
print("Candidate:", candidate)
print("New cell state:", new_c)
```


## The same thing, with a framework

In PyTorch, you would use nn.LSTMCell or nn.LSTM which encapsulates this entire update. For a single cell, you'd write:

```python
import torch
import torch.nn as nn

lstm = nn.LSTMCell(input_size=1, hidden_size=1)
# Input: (batch_size, input_size)
# Hidden state: (batch_size, hidden_size)
x = torch.tensor([[2.0]])  # one example, one input feature
h = torch.zeros(1, 1)  # initial hidden state
c = torch.zeros(1, 1)  # initial cell state
h, c = lstm(x, (h, c))
# Now c contains the updated cell state, which is what we called C_1
```

Under the hood, PyTorch computes the gates in a vectorized way using matrix multiplications, but it does exactly the same arithmetic as our plain Python code: element-wise multiplication and addition.


## What the code does

In the from-scratch code, we first define sigmoid and a tanh approximation using basic arithmetic. Then we set the previous hidden state and cell state as lists (since Python lists are the natural way to hold a vector). The gates are computed as weighted sums of the hidden state and input, passed through sigmoid (for gates) and tanh (for candidate). The cell state update is a simple multiplication and addition of two numbers. This is the core of the LSTM: it is just a few arithmetic operations on a list of numbers.<br><br>In the framework version, the LSTM cell handles everything internally. You provide the input and previous state, and it returns the new hidden and cell states. The hidden state is a separate value that is typically used as the output of the LSTM at that time step; the cell state is the 'memory conveyor belt' that is carried forward but not output directly.


## Common misconceptions

**Hidden state and cell state are the same thing** — The hidden state and the cell state are both the LSTM's memory, so they are interchangeable.

They serve different roles: the cell state is the long-term memory conveyor belt, modified only by gates, while the hidden state is a filtered version of the cell state that is output and used for predictions. In fact: The cell state carries information across time steps, while the hidden state is what the LSTM actually outputs (after passing the cell state through the output gate and tanh).

**The cell state is overwritten each time step** — The LSTM replaces its old memory with a completely new value at each step, losing the old information.

The cell state is updated additively: old values are scaled by the forget gate (which can be between 0 and 1) and new values are added, but the old values are not completely replaced unless the forget gate is 0. In fact: The cell state is modified gently via gates, which allows information to persist over many time steps.

**The forget gate always forgets something** — The forget gate must have a value less than 1, so some information is always lost.

The forget gate outputs a number between 0 and 1; it can be exactly 1 (keeping everything) or 0 (throwing everything away). It is not forced to forget. In fact: The forget gate can be 1, causing the old cell state to be kept entirely, or 0, causing it to be discarded.

**The cell state is a single number** — In an LSTM, the cell state is just one value that gets updated.

The cell state is a vector (a list) of many numbers, typically hundreds, each representing a different piece of information. In fact: The cell state is a vector of dimensions like 256 or 512, and each dimension is updated independently by the gates.

**The cell state is the output of the LSTM** — The LSTM's output at a time step is the cell state, because that's what the network remembers.

The output is the hidden state, which is a gated (filtered) version of the cell state. The cell state is internal and not directly output. In fact: The hidden state is the output; it is computed from the cell state by applying tanh and then multiplying by the output gate.


## In short

The LSTM cell state is a vector carried through time, modified only by three gates (forget, input, output) using additive and multiplicative operations. It allows the network to retain information over long sequences, avoiding the gradient vanishing problem. The cell state and hidden state are distinct: the hidden state is the filtered output. This concept is foundational to understanding LSTM networks and their applications in sequence modeling.


## Where this is used

The LSTM cell state is used in virtually every sequence-processing application that uses LSTMs, which include:<br><br>**Large language models (LLMs):** Although modern LLMs like GPT use transformers, LSTMs were the predecessor. The cell state was the mechanism that allowed them to remember information across long passages. In an LLM, the cell state (if an LSTM were used) would carry a summary of the conversation so far, allowing the model to generate coherent responses. In practice, transformers use a different mechanism (attention) which directly looks back at all previous tokens, but the LSTM cell state provided a similar but more limited memory.<br><br>**Small models on devices:** LSTMs are much more parameter-efficient than transformers, making them suitable for running on smartphones or embedded devices. For example, a keyword spotting model on your phone that listens for 'Hey Siri' or 'OK Google' might use an LSTM. The cell state carries the context of the audio signal over time, so when it hears the second word, it still remembers the first word. A developer using TensorFlow Lite might write a small LSTM layer and deploy it to a phone, knowing that the cell state keeps memory without needing huge memory or compute.<br><br>**Retrieval-augmented generation (RAG):** RAG combines a retrieval system with a language model. If the language model is LSTM-based, the cell state would carry the context of the retrieved documents as they are processed. However, RAG typically uses transformers. Still, the concept of memory is similar: the cell state would help the model remember key facts from a document while generating an answer.<br><br>**Agentic systems:** Agents that plan and call tools need to remember a sequence of actions and their results. An LSTM's cell state could store a compressed history of what actions have been taken and what outcomes occurred, allowing the agent to decide the next step. In practice, agentic systems often use transformers with a context window, but if a lightweight LSTM were used, the cell state would be that memory.<br><br>**Everyday applied ML:** Applications like speech recognition, video analysis, and financial time-series forecasting heavily use LSTMs. For instance, in a stock price predictor, the cell state would remember the trend of the past few days, so that a sudden dip is interpreted in context. A developer using Keras might write `model.add(LSTM(64))` and the cell state is automatically maintained. When done badly (e.g., not resetting the state between sequences), the model will see interference between unrelated sequences, leading to poor predictions. When done well (e.g., using stateful LSTMs for continuous streams), the cell state can carry information over very long sequences.<br><br>In all these cases, what the developer actually writes is a call to an LSTM layer or cell, and the framework handles the cell state internally. But understanding the cell state is crucial for debugging: if your model is not learning long-term dependencies, it may be because your cell state is being reset too often (e.g., you are treating each sequence independently when they are actually continuous).


---

Read this on the web: https://basavapurushottam.com/learn-ai/lstm-cell-state
