# Stateful and Bidirectional LSTMs

Stateful variants carry state across batches for continuous sequences; bidirectional ones run two passes so each position sees both past and future.

**Rests on:** LSTM Cell State, LSTM Networks

*Deep Learning · Neural Network Architectures*


## Why learn this

This concept is the bridge between LSTMs you already know and the architectures that power modern AI. By mastering statefulness and bidirectionality, you unlock: how LLMs handle long documents, how speech recognition and translation work in real time, and how mobile assistants like Siri or Google Assistant process your voice commands on-the-fly. Concretely, you will be able to: (1) explain why a model that reads a sentence left-to-right can't translate it well, (2) know when to use a stateful model for streaming data like stock prices or sensor feeds, and (3) understand why bidirectional models are 'offline' but stateful ones are 'online'. This is the first step toward Transformer architectures, which build on the same idea of context from both directions.


## The intuition

Imagine you are reading a sentence one word at a time, left to right, like a book. You understand each word based on the words before it. That is an LSTM in its basic form — it processes a sequence in order, carrying a 'memory' (the cell state) forward. But sometimes you need to know what comes after to understand what you just read. For example, the word 'bank' could mean a river bank or a money bank; you can't tell until you see the next few words. A bidirectional LSTM reads the sentence twice — once forward and once backward — and then combines the two passes. So each word gets to see both its past and its future. That is powerful, but it means you must have the whole sentence before you start. You can't do it real-time, because you don't have the future yet. Now, think about a long sequence, like a live audio stream that is longer than your computer's memory can hold in one go. You have to break it into chunks (batches) and process each chunk one after another. But if you reset the memory after each chunk, you lose context from earlier chunks. A stateful LSTM keeps the memory from one batch and carries it into the next — like a runner passing a baton without stopping. That is statefulness: the memory persists across batches, so the model can handle sequences longer than one batch.


## An analogy

Think of a detective solving a case by interviewing witnesses one at a time in a line. The basic LSTM detective accumulates clues in his notebook (the cell state) as he goes from witness A to B to C, but he never gets to re-interview anyone. A bidirectional detective, however, first interviews everyone in order from first to last, then interviews them again from last to first, taking notes both ways. After that, he has the complete picture — each clue is understood in light of both earlier and later clues. That is bidirectional: two passes, one forward, one backward. Now, the stateful part: the detective's notebook is huge, too big to carry all at once. So he writes his notes in a set of volumes (batches). After he finishes one volume, he doesn't throw away his notes — he carries a summary of the key points (the state) into the next volume. That way, the story stays continuous across the books. Where the analogy breaks down: In real life, a detective can re-interview witnesses at any time, so he is always 'bidirectional' in a sense. But an LSTM is a machine that reads once in each direction, and it must finish the forward pass before it starts the backward pass — it can't do both at the same time. Also, the detective can choose to remember everything; an LSTM's state is finite, and it has to decide what to keep and what to forget. Statefulness is not about remembering everything forever — it is about passing a compressed summary across batch boundaries.


## Definition

A stateful LSTM carries the hidden state and cell state from one batch of a sequence to the next, so the model can process a sequence longer than a single batch without resetting its memory; a bidirectional LSTM runs the sequence through the same recurrent layer twice — once forward, once backward — and concatenates the hidden states from both directions at each time step, so each output is conditioned on both past and future context.


## How it connects

You have already studied the LSTM cell state and LSTM networks in your library, including the idea of the cell state as a conveyor belt modified by gates. Statefulness and bidirectionality are enhancements to that basic LSTM. Statefulness changes *when* the cell state is reset — instead of resetting after each batch, it persists across batches. Bidirectionality changes *how* the sequence is read — instead of one forward pass, it does two. These connect to Encoder-Decoder architectures, which often use a bidirectional encoder to understand the input fully before generating output. They also connect to CNN-LSTM Hybrids, where the LSTM part can be made bidirectional for video or spectrogram classification. Statefulness is especially relevant for streaming data, as in Rare Event Prediction, where sensor streams are unbounded and need to be processed chunk by chunk without losing context.


## The mathematics

This concept uses linear algebra and a touch of calculus, but let's start with plain words. An LSTM processes a sequence of numbers (like word embeddings or sensor readings) one step at a time. At each step, it has a hidden state (a list of numbers) that represents what it has seen so far, and a cell state that stores longer-term memory. The LSTM uses three gates — forget, input, output — to decide what to keep and what to add. You've learned that. Now, in a unidirectional LSTM, the hidden state at time t depends only on inputs from time 1 to t. That is what 'forward' means. A bidirectional LSTM runs two separate LSTM layers: one processes the sequence from t=1 to t=T (forward), and the other processes it from t=T to t=1 (backward). At each time step, the final hidden state is the concatenation (joining together) of the forward and backward hidden states. So the output at time t depends on both earlier and later inputs. There is no new mathematics beyond what you already know; the key is the *concatenation* operation and the idea of running the same layer twice. Statefulness is simpler: it just means that when you process the sequence in batches, you do not reset the cell and hidden states to zero between batches; you carry them over. The formulas for the gates stay the same; only the initialization changes.

Let the forward LSTM produce hidden states h_t^f for t = 1 to T, and the backward LSTM produce hidden states h_t^b for t = T down to 1. The bidirectional output at time t is the concatenation:

h_t = [h_t^f ; h_t^b]

where ; denotes concatenation. For stateful LSTM, the initial hidden state for a new batch is the final hidden state from the previous batch:

h_0^(batch k+1) = h_T^(batch k)

and similarly for the cell state c_0^(batch k+1) = c_T^(batch k). In stateless LSTMs, h_0 and c_0 are reset to zero at each batch.


## A worked example

```
Suppose we have a tiny sequence of length 3: inputs x1, x2, x3. We will use a unidirectional LSTM with hidden size 2, so hidden states are pairs of numbers. For simplicity, we only show the hidden state update without all the gate details — imagine the LSTM computes new_h = some function(old_h, x). We'll use numbers: old_h0 = (0, 0) initially. x1 = (1, 0), x2 = (0, 1), x3 = (1, 1). Suppose the LSTM's forward pass gives: after x1, h1 = (0.5, 0.2); after x2, h2 = (0.3, 0.8); after x3, h3 = (0.9, 0.4). Now, a backward pass processes the same sequence reversed: x3 first, then x2, then x1. Suppose backward pass gives: after x3, h1_back = (0.1, 0.7); after x2, h2_back = (0.6, 0.5); after x1, h3_back = (0.2, 0.3). The bidirectional LSTM concatenates forward and backward at each time step: at time 1, the combined hidden state is [0.5, 0.2, 0.1, 0.7] (since forward h1 is (0.5, 0.2) and backward h1_back is (0.1, 0.7) — note the backward pass at time 1 has already seen x3, x2, and x1). At time 2: [0.3, 0.8, 0.6, 0.5]. At time 3: [0.9, 0.4, 0.2, 0.3]. So the combined size is 4 (double the hidden size). For statefulness: suppose we chop the sequence into two batches of length 2 each: batch1 = [x1, x2], batch2 = [x3]. In a stateless LSTM, we start with h0=(0,0) for both batches. So after batch1, h2 = (0.3, 0.8). Then for batch2, we reset to h0=(0,0) and process x3, giving h3 = (0.9, 0.4) but that is based only on x3, not x1 and x2. In a stateful LSTM, we carry the hidden state from the end of batch1 (h2 = (0.3, 0.8)) into batch2, so when we process x3, the LSTM starts with h2 and produces h3 = (0.9, 0.4) — but wait, that was computed incorrectly; let's redo: if we start with (0.3, 0.8) and input x3, we get a different h3, say (0.7, 0.6). So the stateful version's h3 depends on all three inputs, while the stateless one's h3 depends only on x3.

Step 1: Forward pass: h1 = f(0,0, x1) = (0.5, 0.2)   # The LSTM updates the hidden state from initial zero to h1.
Step 2: Forward pass: h2 = f(h1, x2) = (0.3, 0.8)   # The hidden state now depends on x1 and x2.
Step 3: Forward pass: h3 = f(h2, x3) = (0.9, 0.4)   # The final forward hidden state depends on all three inputs.
Step 4: Backward pass: h1_back = f(0,0, x3) = (0.1, 0.7)   # The backward LSTM starts from the end and processes x3 first.
Step 5: Backward pass: h2_back = f(h1_back, x2) = (0.6, 0.5)   # Now it sees x3 and x2.
Step 6: Backward pass: h3_back = f(h2_back, x1) = (0.2, 0.3)   # The backward LSTM has now seen the entire sequence, in reverse.
Step 7: Concatenate at time 2: [0.3, 0.8, 0.6, 0.5]   # The forward hidden state (0.3, 0.8) and the backward hidden state (0.6, 0.5) are joined into a 4-element vector.
Step 8: Statefulness: batch1 ends with h2 = (0.3, 0.8); carry this into batch2   # Instead of resetting to zero, the state is passed on.
Step 9: Stateful: h3 = f((0.3, 0.8), x3) = (0.7, 0.6)   # Because we kept the state, h3 now depends on x1 and x2 as well.
Step 10: Stateless: h3 = f((0,0), x3) = (0.9, 0.4)   # Without statefulness, h3 only knows about x3.

Result: Bidirectional concatenation produces a hidden state vector of double size at each time step, combining past and future context. Stateful carries the state over batch boundaries, so the model sees the full sequence even when it is longer than one batch.
```


## From scratch, in Python

```python
# We'll implement a minimal LSTM cell and then show stateful and bidirectional wrappers.

# First, the building blocks you need:
# - A list keeps things in order and can be changed: [1, 2, 3]
# - A tuple is a list that cannot be changed once made: (1, 2)
# - A loop repeats a block of code for each item in a list
# - A function is a reusable block of code that takes inputs and returns an output

# We'll represent a vector (a list of numbers) and a matrix (a list of lists) as plain Python.
# Instead of learning matrix multiplication from scratch, we'll use a helper function that does
# the multiply-and-add for us.

def dot(a, b):
    """Dot product of two lists of numbers: multiply corresponding entries and sum."""
    return sum(x * y for x, y in zip(a, b))

def mat_vec_mul(M, v):
    """Multiply a matrix (list of lists) by a vector (list) to get a new vector."""
    return [dot(row, v) for row in M]

# Now, a simple LSTM cell.
# We'll use fixed weights for demonstration (real models learn them).

# Hidden size = 2, input size = 2
Wf = [[0.1, 0.2], [0.3, 0.4]]  # weights for forget gate
Wi = [[0.5, 0.6], [0.7, 0.8]]  # weights for input gate
Wo = [[0.9, 1.0], [1.1, 1.2]]  # weights for output gate
Wc = [[1.3, 1.4], [1.5, 1.6]]  # weights for candidate cell state

Uf = [[0.1, 0.0], [0.0, 0.1]]  # recurrent weights for forget gate
Ui = [[0.0, 0.1], [0.1, 0.0]]  # recurrent weights for input gate
Uo = [[0.1, 0.1], [0.1, 0.1]]  # recurrent weights for output gate
Uc = [[0.2, 0.0], [0.0, 0.2]]  # recurrent weights for candidate

# Bias vectors
bf = [0.1, 0.1]
bi = [0.0, 0.0]
bo = [0.0, 0.0]
bc = [0.0, 0.0]

def sigmoid(x):
    return 1 / (1 + 2.71828 ** (-x))

def tanh(x):
    # approximate tanh for simplicity
    e = 2.71828
    return (e**x - e**(-x)) / (e**x + e**(-x))

def lstm_step(x, h_prev, c_prev):
    """One LSTM step: takes input x, previous hidden h_prev, previous cell c_prev,
    returns new hidden and cell state."""
    # Forget gate
    f = [sigmoid(v) for v in mat_vec_mul(Wf, x) + mat_vec_mul(Uf, h_prev) + bf]
    # Input gate
    i = [sigmoid(v) for v in mat_vec_mul(Wi, x) + mat_vec_mul(Ui, h_prev) + bi]
    # Candidate cell
    c_tilde = [tanh(v) for v in mat_vec_mul(Wc, x) + mat_vec_mul(Uc, h_prev) + bc]
    # New cell state
    c_new = [f_j * c_prev_j + i_j * c_tilde_j for f_j, c_prev_j, i_j, c_tilde_j in zip(f, c_prev, i, c_tilde)]
    # Output gate
    o = [sigmoid(v) for v in mat_vec_mul(Wo, x) + mat_vec_mul(Uo, h_prev) + bo]
    # New hidden
    h_new = [o_j * tanh(c_j) for o_j, c_j in zip(o, c_new)]
    return h_new, c_new

# Forward LSTM over a sequence
def forward_lstm(seq, init_h, init_c):
    h = init_h
    c = init_c
    hiddens = []
    for x in seq:
        h, c = lstm_step(x, h, c)
        hiddens.append(h)
    return hiddens

# Stateful version: we pass in the initial state and get back the final state.
def forward_lstm_stateful(seq, init_h, init_c):
    h = init_h
    c = init_c
    for x in seq:
        h, c = lstm_step(x, h, c)
    return h, c  # return final state for next batch

# Now, demonstrate stateless vs stateful with mini-batches.

# Example sequence: 3 time steps, each input is a 2-vector
seq = [[1, 0], [0, 1], [1, 1]]

# Split into batches: batch1 = [seq[0], seq[1]], batch2 = [seq[2]]
batch1 = seq[:2]
batch2 = seq[2:]

# Stateless: reset h and c to zero for each batch
h0 = [0.0, 0.0]
c0 = [0.0, 0.0]
# Batch 1
h_b1_final, c_b1_final = forward_lstm_stateful(batch1, h0, c0)
# Batch 2 starts with zero again
h_b2_final, c_b2_final = forward_lstm_stateful(batch2, h0, c0)
print("Stateless final h for batch2:", h_b2_final)

# Stateful: carry the state from batch1 to batch2
h0_stateful = h_b1_final
c0_stateful = c_b1_final
h_b2_stateful, c_b2_stateful = forward_lstm_stateful(batch2, h0_stateful, c0_stateful)
print("Stateful final h for batch2:", h_b2_stateful)

# Bidirectional: run two LSTMs, one forward, one backward
# Reverse the sequence for the backward pass
seq_rev = list(reversed(seq))
# Forward pass
h_fwd = forward_lstm(seq, h0, c0)
# Backward pass (use the same LSTM but process reversed)
h_bwd = forward_lstm(seq_rev, h0, c0)
# At each time step t, concatenate forward h_fwd[t] and backward h_bwd[T-1-t]
concatenated = []
T = len(seq)
for t in range(T):
    conc = h_fwd[t] + h_bwd[T-1-t]  # list concatenation
    concatenated.append(conc)
print("Bidirectional hidden states (concatenated):")
for t, c in enumerate(concatenated):
    print(f"t={t}: {c}")
```


## The same thing, with a framework

In PyTorch, stateful and bidirectional LSTMs are built into the `nn.LSTM` layer. Here is how a developer would use it:

```python
import torch
import torch.nn as nn

# Parameters
input_size = 2
hidden_size = 2
num_layers = 1

# Bidirectional: set bidirectional=True
lstm = nn.LSTM(input_size, hidden_size, bidirectional=True)

# Input: (sequence_length, batch_size, input_size)
# For a single sequence of length 3, batch_size=1
seq = torch.tensor([[[1, 0]], [[0, 1]], [[1, 1]]], dtype=torch.float32)  # shape (3, 1, 2)

# Forward pass
output, (h_n, c_n) = lstm(seq)
# output shape: (seq_len, batch, num_directions * hidden_size) = (3, 1, 4)
print("Bidirectional output shape:", output.shape)

# For stateful, you need to manage the initial state manually.
# Create an initial state (h_0, c_0) of shape (num_layers * num_directions, batch, hidden_size)
# For stateful, you would carry this from the previous batch.
h_0 = torch.zeros(1 * 2, 1, hidden_size)  # because bidirectional, num_directions=2
c_0 = torch.zeros(1 * 2, 1, hidden_size)

# Process batch1
out1, (h_n1, c_n1) = lstm(seq[:2], (h_0, c_0))
# Now use h_n1, c_n1 as the initial state for batch2
out2, (h_n2, c_n2) = lstm(seq[2:], (h_n1, c_n1))
```

The `bidirectional=True` flag doubles the hidden size in the output, and internally it runs two LSTMs (one forward, one backward) and concatenates their outputs. For statefulness, PyTorch does not have a built-in flag; you simply pass the previous `(h_n, c_n)` as the initial state for the next batch. That is exactly what our plain Python code did manually.


## What the code does

We start with the building blocks. `dot` multiplies two lists element-wise and adds them; `mat_vec_mul` does this for each row of a matrix to produce a new vector. These are the multiply-and-add operations you learned in the mathematics section. Then we define an LSTM step: it takes an input vector `x`, the previous hidden state `h_prev`, and the previous cell state `c_prev`, and returns new hidden and cell states. The gates use sigmoid to decide how much to forget, write, and output, exactly as you learned in the LSTM cell state topic. The code for `sigmoid` and `tanh` is the same functions you saw earlier. Then `forward_lstm` loops over a sequence, updating the state at each time step, collecting the hidden states. `forward_lstm_stateful` is like `forward_lstm` but only returns the final state — you pass that final state as the initial state for the next batch. That is the stateful mechanism. In the demonstration, we split the sequence into two batches. For the stateless case, we start each batch with zero state. For the stateful case, we take the final state from batch1 as the initial state for batch2. You can see that the final hidden state for batch2 differs: the stateful one has information from batch1. Finally, for bidirectional, we run the same LSTM forward on the original sequence and backward on the reversed sequence. At each time step, we concatenate the forward hidden state at `t` with the backward hidden state at `T-1-t` (because the backward pass processes from the end). The result is a list of doubled-length vectors, each containing both past and future context.


## Common misconceptions

**Bidirectional means it is always forward and backward at once** — Bidirectional LSTM can process a sequence in real time, seeing both past and future as they arrive.

A bidirectional LSTM must see the entire sequence first. It runs a forward pass from start to end, then a separate backward pass from end to start. Only after both passes are done can it produce output for any time step. So it cannot output anything until the whole sequence is known. In fact: Bidirectional LSTMs are offline: they require the full input before producing any output. For real-time processing, use a unidirectional LSTM.

**Stateful means the model remembers everything forever** — A stateful LSTM has infinite memory — nothing is ever forgotten.

The cell state has a fixed size (the number of hidden units). The LSTM gates decide what to forget and what to keep, but even with statefulness, old information gets overwritten as new inputs come in. The cell state is finite. In fact: Statefulness only means that the state is carried from one batch to the next, not that memory is limitless. The LSTM still forgets through its forget gate.

**Statefulness and bidirectionality are the same thing** — They both make the model understand more context, so they are basically interchangeable.

Statefulness is about *how the sequence is chunked into batches* — it preserves context across batch boundaries. Bidirectionality is about *the direction of reading* — it uses both past and future within a single sequence. They solve different problems and can be combined (a stateful bidirectional LSTM is possible) but are not the same. In fact: Statefulness handles long sequences that exceed one batch. Bidirectionality handles context from both sides. You could have a stateful unidirectional LSTM or a stateless bidirectional LSTM.

**Bidirectional always improves accuracy** — Using bidirectional LSTM is strictly better, so I should always set bidirectional=True.

Bidirectional models require the entire input before generating output, which makes them unsuitable for real-time tasks. They also double the computational cost and the number of parameters. For some tasks, like autoregressive generation (predicting the next word), bidirectional is impossible, because the future is unknown. In fact: Bidirectional is beneficial when you have the whole input (e.g., classification, translation with encoder) but not for streaming or generative tasks. It is a design choice based on the task's requirements.


## In short

Statefulness carries the hidden and cell states across batches, letting an LSTM process sequences longer than a single batch. Bidirectionality runs the sequence forward and backward, concatenating the hidden states so each output has both past and future context. Use stateful for streaming data; use bidirectional for offline tasks where the whole input is available. Both change what context the model conditions its predictions on.


## Where this is used

Stateful and bidirectional LSTMs are used across many real AI systems. In **large language models (LLMs)**, pure LSTMs are rare (transformers dominate), but bidirectional context is essential. For example, BERT (a language model) uses bidirectional attention to predict a missing word by looking at both left and right context — that is the same idea as a bidirectional LSTM, though implemented differently. In practice, if you are fine-tuning a sentiment analysis model on a full document, you would use a bidirectional LSTM to read the entire text before classifying, so the model considers all words in both directions. In **small models running on devices**, such as on a phone or a laptop, stateful LSTMs are common for streaming applications like voice recognition. The model processes audio in short chunks (say 20ms each) so it can respond in real time. A stateful LSTM keeps its memory across these chunks, so the recognition is smooth and continuous, even though the device cannot hold the whole audio in memory at once. For example, Google's speech recognition on mobile uses stateful recurrent networks. In **retrieval-augmented generation (RAG)**, the embedding step that turns documents into vectors often uses bidirectional LSTMs to embed each word with full context. A document is processed with a bidirectional LSTM, and the last hidden state (or a combination) becomes the embedding passed to a similarity search. That way, a question like 'What is the capital of France?' embeds with the meaning that 'France' is a country, not a person. In **agentic systems that plan and call tools**, language models that act as agents often need to remember the conversation history. If the history is longer than one batch, a stateful LSTM can carry the state across turns, so the agent remembers what the user said earlier. This is less common now that transformers are used, but for limited-resource settings, stateful LSTMs are a lightweight alternative. In **everyday applied machine learning**, bidirectional LSTMs are used for time-series forecasting (e.g., predicting electricity load) where each time point is influenced by both earlier and later values. Stateful LSTMs are used for anomaly detection in sensor streams, like rare-event prediction in industrial processes, where the stream is infinite and must be processed in rolling windows without resetting memory. A developer using these would typically write in PyTorch: they would set `bidirectional=True` in `nn.LSTM` and manage stateful by carrying `(h_n, c_n)` between batches. The code they write is exactly the framework example above. What breaks when done badly: if you forget to reset the state at the right time (e.g., at the start of a new document), the model will carry irrelevant context and predictions become garbage. If you use a bidirectional LSTM for streaming data, you are forced to wait for the entire stream, which is impossible — so you must choose the right directionality for the task.


---

Read this on the web: https://basavapurushottam.com/learn-ai/stateful-and-bidirectional-lstms
