Deep Learning
Topic 11 · Deep Learning · ⏱ about 6 h

Sequence Models: RNNs, LSTMs & GRUs

Networks with memory: recurrent networks, backpropagation through time, vanishing gradients, LSTM and GRU gates, sequence tasks, padding and masking, and when RNNs still make sense.

Show:
In this topic
Beginner

A network with memory

Text, speech, sensor streams and patient vital signs are sequences: order matters, and lengths vary. A recurrent neural network (RNN) reads one element at a time and keeps a hidden state h that summarises everything seen so far:

ht=tanh⁡(xtWx+ht−1Wh+b),y^t=htWy+c.h_t=\tanh(x_tW_x+h_{t-1}W_h+b),\qquad \hat y_t=h_tW_y+c.

The same weights are used at every time step (weight sharing over time, as a CNN shares over space), so an RNN handles any length. To train it, the network is unrolled over the sequence into a deep feed-forward graph with one layer per time step, and ordinary backpropagation runs through it: backpropagation through time (BPTT). For long sequences BPTT is usually truncated to the last k steps.

Beginner

Watch it: Inside an LSTM cell: forget, write, reveal

A short narrated animation. Press ▶ Play, or step through it with ⏮ ⏭. Tick 🔊 narrate to hear each step.

Interactive: this part runs in your browser.

Beginner

The vanishing-gradient problem through time

The gradient that connects the loss at step T to an input at step t passes through T−tT-t copies of the recurrent step, each multiplying it by roughly Wh⊤W_h^\top times the tanh derivative (at most 1). Over 50 or 100 steps that product shrinks exponentially when the factors are below 1 (the network cannot learn long-range dependencies), or explodes when they are above 1 (loss spikes; clipping helps). In practice plain RNNs remember only around 10 to 20 steps.

Try it

Try it: how far back can the gradient reach?

Plot the size of the gradient reaching earlier time steps for a plain RNN with different recurrent weight scales, and for an LSTM whose forget gate stays open.

Interactive: this part runs in your browser.

Beginner

LSTM and GRU: learning what to remember

The LSTM (long short-term memory, 1997) adds a cell state c, a conveyor belt that information can travel along with only small, controlled changes. Three gates, each a sigmoid between 0 and 1, decide what happens to it:

ft=σ(⋅), it=σ(⋅), ot=σ(⋅), c~t=tanh⁡(⋅)ct=ft⊙ct−1+it⊙c~t,ht=ot⊙tanh⁡(ct)f_t=\sigma(\cdot),\ i_t=\sigma(\cdot),\ o_t=\sigma(\cdot),\ \tilde c_t=\tanh(\cdot)\qquad c_t=f_t\odot c_{t-1}+i_t\odot\tilde c_t,\qquad h_t=o_t\odot\tanh(c_t)

where each (·) is a linear function of [xt,ht−1][x_t, h_{t-1}]. The forget gate f decides what to keep, the input gate i what to write, the output gate o what to reveal. Because ctc_t depends on ct−1c_{t-1} through a multiplication by f (close to 1 when remembering) and an addition, gradients can flow across hundreds of steps: the same idea as a residual connection. A common trick is to initialise the forget-gate bias to 1 so the network starts by remembering.

The GRU merges the gates into two (update and reset) and has no separate cell state: fewer parameters and often similar accuracy.

Beginner

Sequence tasks, padding and masking

  • Many-to-one: read a whole sequence, predict one label from the last hidden state (or a pooled one): sentiment, "will this patient deteriorate in the next 6 hours?".
  • Many-to-many, aligned: one output per step: tagging each word, detecting events in a signal.
  • Sequence-to-sequence: an encoder reads the input and a decoder generates a different-length output: translation, summarisation. Decoders are trained with teacher forcing (feeding the true previous token).
  • Bidirectional RNNs read the sequence both ways: useful when the whole sequence is available (not for real-time forecasting).

Batches need equal lengths, so shorter sequences are padded with zeros. The padding must then be masked: excluded from the loss and from pooling, and skipped by the RNN (PyTorch's pack_padded_sequence). Forgetting the mask makes the model learn from padding and is a classic bug.

Beginner

Where RNNs stand today

For text, transformers (next topic) have replaced RNNs almost everywhere: they train in parallel over the whole sequence and handle long-range dependencies directly. RNNs and their relatives remain useful for streaming data that must be processed one step at a time with constant memory (sensors, on-device audio), small datasets, and as strong baselines for time series, along with 1-D CNNs and temporal convolutional networks. Recent state-space models (S4, Mamba) bring back the recurrent idea with transformer-level quality on long sequences.

Try it

Code example: an RNN and an LSTM cell

A worked example: read it step by step (every part is explained), then run it and experiment.

Interactive: this part runs in your browser.

Code

Code: an LSTM classifier in PyTorch

Many-to-one classification of variable-length sequences, with packing so padding is ignored.

Python · PyTorch
import torch, torch.nn as nn
from torch.nn.utils.rnn import pack_padded_sequence

class SeqClassifier(nn.Module):
    def __init__(self, d_in=4, d_h=64, n_classes=2):
        super().__init__()
        self.lstm = nn.LSTM(d_in, d_h, num_layers=2, batch_first=True, dropout=0.2)
        self.head = nn.Linear(d_h, n_classes)
    def forward(self, x, lengths):               # x: (B, T_max, d_in), lengths: (B,)
        packed = pack_padded_sequence(x, lengths.cpu(), batch_first=True, enforce_sorted=False)
        _, (h, c) = self.lstm(packed)            # h: (num_layers, B, d_h), last REAL step
        return self.head(h[-1])                  # (B, n_classes)

m = SeqClassifier()
x = torch.randn(3, 48, 4); lengths = torch.tensor([48, 30, 12])
print(m(x, lengths).shape)
This code needs PyTorch, which cannot run inside a web page. To run it, press Copy, then paste it into a new notebook in Google Colab ↗ (free; Runtime → Change runtime type → GPU for deep learning).
Check yourself

Quiz

1. What does the hidden state of an RNN represent?

2. Why do plain RNNs struggle with long-range dependencies?

3. In an LSTM, what does the forget gate control?

4. Why initialise the LSTM forget-gate bias to about 1?

5. You pad sequences to length 100 but forget to mask the padding in a mean-pooled loss. What happens?

6. A bedside monitor must update a deterioration score every second with constant memory. A good fit is…

Go deeper

📚 Further reading

Trusted, mostly free sources: official documentation, well-known textbooks and courses. When you want to confirm or go deeper on anything here, start with these.

Ready