Lesson 3.8 · 25 min

Recurrent Neural Networks: Processing Sequences in Order

How can a network read "the movie was not good" one word at a time and still remember the "not" by the time it reaches "good"?

In short: A recurrent neural network (RNN) processes a sequence one step at a time and carries a hidden state, a small vector of memory, from each step to the next. The same weights are reused at every step: hₜ = tanh(Wₓₕxₜ + Wₕₕhₜ₋₁ + b). RNNs are trained with backpropagation through time, struggle with long-range memory because gradients vanish or explode, and led to LSTMs, GRUs and eventually Transformers.

Why ordinary networks struggle with sequences

The networks we have seen so far are feed-forward: a fixed-size input goes in, flows through the layers once, and an output comes out. That works for a house described by three numbers. But much of the world arrives as a sequence, where order matters and length varies: the words of a sentence, the notes of a melody, a stock price every minute, a user's clicks on a website.

Our running example is a sentiment classifier for product reviews. "The movie was not good" and "The movie was good, not bad" use almost the same words, but mean opposite things because of their order. A feed-forward network that sees a bag of words, or a fixed-width window, loses that order, and it cannot easily handle reviews of 5 words and 500 words with the same weights.

We want a model that: (1) reads inputs in order, (2) handles any length, (3) remembers what it has seen so far, and (4) reuses the same knowledge at every position, since "not" means the same thing at word 3 and word 30. A recurrent neural network does exactly that.

Think of it like reading with a notepad Imagine reading a long review aloud, one word at a time, while keeping a tiny notepad with room for only a few notes. After each word you rewrite the notepad based on what it said before and the word you just read. At the end, you judge the review using only the notepad. The notepad is the RNN's hidden state; your rule for updating it is the RNN's weights.

The core idea: a loop with memory

An RNN has a hidden state hₜ: a vector of numbers that summarises everything the network has read up to step t. At each time step (each position in the sequence) it takes two things, the current input xₜ and the previous hidden state hₜ₋₁, and combines them into a new hidden state. Optionally it also produces an output yₜ.

The key word is recurrent: the output of the hidden layer feeds back into itself at the next step. And crucially, Wₓₕ, Wₕₕ and Wₕᵧ are shared across all time steps. A 5-word review and a 500-word review use exactly the same parameters, just applied 5 or 500 times. This weight sharing is what lets an RNN handle any length and generalise patterns across positions.

Unrolling means drawing the loop as a chain, one copy of the cell per time step. It is only a picture: there is one set of weights, used repeatedly. But it is the right picture for understanding both the forward computation and training.

How it works, step by step with numbers

One RNN time step

1. Read the input: Take xₜ, for text usually an embedding vector representing the token.
2. Recall the memory: Take hₜ₋₁ from the previous step (zeros at the start).
3. Mix: Compute Wₓₕ·xₜ + Wₕₕ·hₜ₋₁ + b: new information plus transformed old memory.
4. Squash: Apply tanh, giving the new hidden state hₜ with values in (−1, 1).
5. Optionally output: yₜ = Wₕᵧ·hₜ. Some tasks need an output every step (tagging each word), others only at the end (sentiment).
6. Pass it on: hₜ becomes the memory for step t + 1. Repeat until the sequence ends.

Let us run a tiny RNN by hand with hidden size 2 and a one-number input. Weights: Wₓₕ = [0.5, −0.3]ᵀ, Wₕₕ = [[0.8, 0], [0.2, 0.5]], b = 0, Wₕᵧ = [1, −1]. The input sequence is [1, 0, 0, 2].

• t = 1, x = 1: Wₓₕ·1 = [0.5, −0.3], Wₕₕ·h₀ = [0, 0]. h₁ = tanh([0.5, −0.3]) = [0.462, −0.291]. y₁ = 0.462 + 0.291 = 0.753.
• t = 2, x = 0: no new input, so only memory: Wₕₕ·h₁ = [0.8×0.462, 0.2×0.462 + 0.5×(−0.291)] = [0.370, −0.053]. h₂ = tanh(...) = [0.354, −0.053]. The memory of the first input is still there, but fading.
• t = 3, x = 0: it fades further: h₃ = [0.276, 0.044].
• t = 4, x = 2: a strong new input dominates: h₄ = [0.840, −0.480].

Notice how the information from step 1 shrinks at every step when no new input arrives (0.462 → 0.354 → 0.276 in the first unit). That fading is a small preview of the RNN's biggest weakness.

Pause and think: Why can the same RNN process a 5-word review and a 500-word review, when a feed-forward network with a fixed input size cannot?

Because the RNN applies the same weights at every time step and carries a fixed-size hidden state. Longer input just means more steps, not more parameters. A feed-forward network's first weight matrix has a fixed number of inputs.

Input and output shapes: what RNNs can do

Two common extensions: a bidirectional RNN runs one RNN left-to-right and another right-to-left and combines their states, so each position sees both past and future context (useful for tagging, not for generating text). A stacked (deep) RNN feeds the hidden states of one RNN layer as the inputs to another.

Training: backpropagation through time

RNNs are trained with the same backpropagation we already know, applied to the unrolled network. Because the weights are shared, the gradient for Wₕₕ is the sum of its gradients at every time step. This is called backpropagation through time (BPTT).

Here is the catch. To learn that the "not" at step 3 should flip the meaning of "good" at step 50, the error at the end must flow back through 47 steps. By the chain rule, that gradient includes a product of 47 factors, each roughly Wₕₕ times a tanh derivative (which is at most 1). Multiply many numbers smaller than 1 and you get almost zero: the vanishing gradient problem. Multiply many numbers larger than 1 and you get a huge number: the exploding gradient problem.

rnn_demo.py

import numpy as np
# Part 1: a tiny RNN reading a sequence one step at a time (hidden size 2)
W_xh = np.array([[0.5], [-0.3]])          # input -> hidden   (2x1)
W_hh = np.array([[0.8, 0.0], [0.2, 0.5]]) # hidden -> hidden  (2x2), shared by ALL steps
b_h = np.zeros((2, 1))
W_hy = np.array([[1.0, -1.0]])            # hidden -> output  (1x2)
h = np.zeros((2, 1))                      # h_0: empty memory
for t, x in enumerate([1.0, 0.0, 0.0, 2.0], start=1):
h = np.tanh(W_xh * x + W_hh @ h + b_h)    # h_t = tanh(W_xh x_t + W_hh h_(t-1) + b)
y = (W_hy @ h).item()
print(f"t={t} x={x}  h={h.ravel().round(3)}  y={y:+.3f}")
# Part 2: how much does h_T still depend on h_0? (gradient through time)
rng = np.random.default_rng(0)
for scale in (0.5, 1.0, 1.5):
W = scale * np.linalg.qr(rng.normal(size=(16, 16)))[0]   # orthogonal x scale
h, J = rng.normal(0, 0.1, (16, 1)), np.eye(16)
norms = []
for t in range(1, 51):
h = np.tanh(W @ h)
J = (1 - h ** 2) * W @ J          # chain rule: dh_t/dh_0 = diag(tanh') W dh_(t-1)/dh_0
if t in (1, 10, 50):
norms.append(f"{np.linalg.norm(J, 2):.2e}")
print(f"recurrent scale {scale}: |dh_t/dh_0| at t=1,10,50 ->", norms)

Output:

t=1 x=1.0  h=[ 0.462 -0.291]  y=+0.753
t=2 x=0.0  h=[ 0.354 -0.053]  y=+0.407
t=3 x=0.0  h=[0.276 0.044]  y=+0.232
t=4 x=2.0  h=[ 0.84 -0.48]  y=+1.320
recurrent scale 0.5: |dh_t/dh_0| at t=1,10,50 -> ['5.00e-01', '9.76e-04', '8.88e-16']
recurrent scale 1.0: |dh_t/dh_0| at t=1,10,50 -> ['1.00e+00', '9.62e-01', '8.13e-01']
recurrent scale 1.5: |dh_t/dh_0| at t=1,10,50 -> ['1.50e+00', '2.86e+01', '2.26e+02']

The output makes the problem concrete. With a recurrent scale of 0.5, by step 50 the hidden state is essentially independent of the start: the network cannot learn long-range patterns because no gradient signal survives. With 1.5, the gradients grow and training becomes unstable. Keeping everything balanced at exactly the right scale is impractical, so better architectures were needed.

Practical fixes Gradient clipping (rescale the gradient whenever its norm exceeds a threshold, e.g. 1.0) handles exploding gradients and is standard for RNN training. Truncated BPTT only backpropagates through the last k steps (say 100) to save memory and time, at the cost of not learning dependencies longer than k. Vanishing gradients need an architectural fix: gates.

LSTM and GRU: RNNs with gates

The Long Short-Term Memory (LSTM) network, introduced by Sepp Hochreiter and Jürgen Schmidhuber in 1997, adds a separate cell state that runs along the sequence with only small, controlled changes, plus gates: small sigmoid layers that output numbers between 0 and 1 and act like valves.

• Forget gate: how much of the old cell state to keep.
• Input gate: how much of the new candidate information to write.
• Output gate: how much of the cell state to expose as the hidden state.

Because the cell state is updated by adding gated information rather than by repeatedly multiplying through a squashing function, gradients can flow across many more steps when the forget gate stays near 1. The Gated Recurrent Unit (GRU), proposed by Cho and colleagues in 2014, is a simpler variant with two gates (update and reset) and no separate cell state; it often performs similarly to an LSTM with fewer parameters.

RNNs vs Transformers

Even LSTMs have two deep limits. First, sequential computation: step t cannot start until step t − 1 finishes, so you cannot spread one sequence across thousands of GPU cores during training. Second, the bottleneck: everything the model knows about the past must squeeze through one fixed-size hidden vector.

Attention removed the bottleneck by letting the model look back at all previous positions directly, and the Transformer (2017) dropped recurrence entirely, processing all tokens of a training sequence in parallel. That is why today's large language models are Transformers. RNNs still have one advantage: generating each new token costs the same small, constant amount of memory and compute, whereas a Transformer's attention must look over a growing context. This has motivated newer recurrent-style designs, such as state-space models and linear-attention RNNs, that try to combine parallel training with cheap recurrent inference.

From simple RNNs to Transformers

1. Elman network: Jeffrey Elman's simple recurrent network popularises the hidden-state loop. Backpropagation through time is described around the same period.
2. LSTM: Hochreiter and Schmidhuber add a cell state and gates to fight vanishing gradients.
3. GRU and seq2seq: The GRU is proposed, and encoder–decoder LSTMs show strong results in machine translation.
4. Attention: Attention lets a decoder look back at every encoder state instead of one compressed vector.
5. Transformer: "Attention Is All You Need" removes recurrence; parallel training wins at scale.
6. Recurrent ideas return: State-space and linear-recurrent models revisit RNN-style constant-memory inference with parallel training.

Real-world use, pitfalls and when not to use an RNN

Where RNNs were and are used Before Transformers, LSTMs powered many production systems in speech recognition, machine translation, handwriting recognition and keyboard next-word prediction. Today they remain a reasonable choice for small on-device models, streaming sensor and time-series data where inputs arrive one at a time, and situations where memory per step must stay constant.

Common mistakes Using a vanilla RNN and expecting it to remember things 100 steps back. Forgetting gradient clipping and getting NaN losses. Mixing up padding: in a batch of different-length sequences, padded positions must be masked or packed so they do not corrupt the final hidden state. Forgetting to reset the hidden state between unrelated sequences. And using a bidirectional RNN for generation, where the future is not available.

• Do not choose an RNN for large-scale language modelling or tasks needing very long context where you can afford a Transformer: it trains far slower and remembers less.
• Do consider one for small, streaming, low-latency problems, or as a baseline for time series.

Pause and think: An LSTM trains well on short sentences but its loss suddenly becomes NaN on long documents. What is the first fix to try?

Add gradient clipping. Long sequences mean long products in backpropagation through time, and occasional exploding gradients produce huge updates that turn the loss into NaN. Truncated BPTT and a lower learning rate are also worth trying.

Worked example, step by step

We named gradient clipping as the standard fix for exploding gradients, but we never did one. Let us clip a gradient by hand. Suppose backpropagation through time on a long review gives this gradient for three recurrent weights: g = [3, 4, 12]. Our clipping threshold is 1.0 and the learning rate is 0.1.

Clipping by norm

1. Measure the size: The norm is the length of the gradient vector: ‖g‖ = √(3² + 4² + 12²) = √169 = 13.
2. Compare with the threshold: 13 is larger than 1.0, so we clip. If it were 1.0 or less we would leave the gradient alone.
3. Compute the scale: scale = threshold / ‖g‖ = 1 / 13 ≈ 0.077.
4. Rescale every entry: g · scale = [0.231, 0.308, 0.923]. The new norm is exactly 1.0.
5. Update: The weights move by 0.1 × [0.231, 0.308, 0.923]. Without clipping they would have moved by [0.3, 0.4, 1.2], thirteen times further.

Notice what stayed the same: the ratios. The third weight still gets four times the update of the first (0.923 vs 0.231, like 12 vs 3). Clipping by norm keeps the direction of the step and only shortens it. There is a second, cruder way, which cuts each entry on its own:

How do we pick the threshold? A practical way is to log the gradient norm during training. Most steps will sit in a normal range, with rare spikes far above it. A threshold a little above the normal range leaves ordinary steps untouched and only catches the spikes. If almost every step is clipped, the threshold is acting as a hidden learning-rate cut, and it is better to lower the learning rate itself.

Practice: try it yourself

We build the smallest possible RNN: one hidden unit, one input number, two weights. First we watch how long it remembers a single 1 for three values of the recurrent weight. Then we use it as a many-to-one detector that answers "did a 1 appear anywhere?" for sequences of different lengths.

practice_rnn_memory.py

import math
W_X = 2.0     # input -> hidden weight (one hidden unit, one input number)
def run(seq, w_h):
"""Read the sequence with h_t = tanh(W_X * x_t + w_h * h_(t-1))."""
h, trace = 0.0, []
for x in seq:
h = math.tanh(W_X * x + w_h * h)     # the same two weights at every step
trace.append(h)
return trace
# 1) A single 1 followed by silence: how long does the memory last?
seq = [1, 0, 0, 0, 0, 0, 0, 0]
for w_h in (0.5, 1.0, 2.5):
trace = run(seq, w_h)
shown = "  ".join(f"{h:.3f}" for h in trace)
# local gradient factor dh_t/dh_(t-1) = w_h * (1 - h_t^2) at the last step
factor = w_h * (1 - trace[-1] ** 2)
print(f"w_h={w_h}:  {shown}   last factor={factor:.3f}")
# 2) Many-to-one: "did a 1 appear anywhere?" for sequences of any length
for seq in ([0, 0, 0], [0, 1, 0, 0, 0], [0] * 49 + [1], [1] + [0] * 49):
h_last = run(seq, w_h=2.5)[-1]
answer = "yes" if h_last > 0.5 else "no"
print(f"length {len(seq):2d}, a 1 at position "
f"{seq.index(1) + 1 if 1 in seq else '-':>2}: h_last={h_last:.3f} -> {answer}")

Output:

w_h=0.5:  0.964  0.448  0.220  0.110  0.055  0.027  0.014  0.007   last factor=0.500
w_h=1.0:  0.964  0.746  0.633  0.560  0.508  0.468  0.437  0.411   last factor=0.831
w_h=2.5:  0.964  0.984  0.986  0.986  0.986  0.986  0.986  0.986   last factor=0.071
length  3, a 1 at position  -: h_last=0.000 -> no
length  5, a 1 at position  2: h_last=0.986 -> yes
length 50, a 1 at position 50: h_last=0.964 -> yes
length 50, a 1 at position  1: h_last=0.986 -> yes

Now change it:

• In part 2, change w_h=2.5 to w_h=0.5. Before running, predict which of the four answers flip from "yes" to "no", and which one survives.
• In part 1, use the sequence [1, 0, 0, -1, 0, 0, 0, 0]. Predict whether a single −1 can erase the memory of the unit with w_h = 2.5. What does the result say about a memory that can only be written, never cleared?
• Add 1.5 to the tuple of w_h values. Predict whether the memory fades to zero or settles at a fixed level, and whether its last factor is above or below 1.

Pause and think: With w_h = 2.5 the unit remembers the 1 perfectly, yet its local gradient factor is only 0.071. What does that mean for training?

The forward memory and the backward signal are two different things. The unit holds its value because tanh is saturated near 1, and a saturated tanh has a derivative near 0. Each step back in time multiplies the gradient by about 0.071, so after 10 steps almost nothing is left. The unit can keep a fact, but gradient descent can hardly reach back to adjust how that fact was stored. This tension is what the additive cell state of an LSTM was designed to remove.

Pause and think: With w_h = 0.5 the last factor is 0.500 and with w_h = 1.0 it is 0.831. Using these, which setting lets an error signal travel further back, and is that enough for a review of 80 words?

The setting 1.0 is better, because each step keeps 83% of the signal instead of 50%. But neither is enough. Even 0.831 multiplied 80 times is below one millionth (and the true factors are a little different at each step, but all below 1). Any factor that stays under 1 gives a product that shrinks towards zero as the sequence grows. That is the vanishing gradient problem in one number.

Key takeaways

• An RNN reads a sequence step by step, updating a hidden state: hₜ = tanh(Wₓₕxₜ + Wₕₕhₜ₋₁ + b).
• The same weights are reused at every step, so one model handles any sequence length.
• Training uses backpropagation through time; long products of derivatives make gradients vanish or explode.
• LSTMs and GRUs add gates so information can survive many steps; gradient clipping handles explosions.
• Transformers replaced RNNs for most language tasks because they train in parallel and attend to all positions directly.

Key terms

• Recurrent neural network (RNN): A network that processes a sequence one step at a time, passing a hidden state from each step to the next.
• Hidden state: The vector an RNN carries between steps that summarises what it has seen so far.
• Unrolling: Drawing an RNN as a chain of identical cells, one per time step, sharing the same weights.
• Backpropagation through time (BPTT): Backpropagation applied to the unrolled RNN, summing gradients for the shared weights over all steps.
• Vanishing / exploding gradients: Gradients that shrink towards zero or grow huge because they are products of many factors.
• LSTM: A gated RNN with a cell state and forget, input and output gates that preserve information over long spans.
• Gradient clipping: Rescaling gradients whose norm exceeds a threshold to prevent unstable updates.

← 3.7 RMSNorm: Simpler Normalization for Transformers · 3.9 PyTorch Internals: Dynamic Graphs and Autograd →