From Sequences to Attention

Consider two sentences: "dog bites man" and "man bites dog". They contain exactly the same words, but one is ordinary and the other is news. Any representation that treats a sentence as a bag of words, simply counting which words occur, cannot tell them apart:

The two bags are identical. And if we average the embedding vectors of the words, as our sentiment model did in Lesson 5.3, the result is also identical, whatever the embeddings are:

A cosine similarity of 1.0 means the two sentences get exactly the same representation. Averaging is order-blind: adding up the same vectors in a different order always gives the same total.

The same problem appears everywhere in sequences. "Not good" and "good, not" mean different things. In a sensor reading, a temperature that rises then falls is different from one that falls then rises. What we need is a way of combining a sequence that depends on the order of its items.

Key terms

• Sequence: an ordered list of items, such as words in a sentence or readings over time.
• Bag of words: a representation that records which words appear (and how often) but not their order.
• Order-blind: giving the same result whatever order the inputs come in.

Think about how you read this sentence. You don't hold every previous word separately in mind. You keep a running sense of "what this is about so far", and update it as each new word arrives.

A recurrent neural network (RNN) does exactly that. It reads a sequence one item at a time, in order, and keeps a vector called the hidden state, its running memory. At each step it combines two things: the new input and its previous hidden state. Its output is a new hidden state, which is carried forward to the next step.

The update rule at each step t is:

where:

• x_t is the input at step t (for example, the embedding of word t);
• h_(t−1) is the hidden state from the previous step (it starts as all zeros);
• W_x and W_h are weight matrices, and b is a bias; the same weights are used at every step;
• tanh (the hyperbolic tangent) squashes each value into the range −1 to +1, so the hidden state cannot grow without limit.

Figure 5.4.1. One recurrent update with shared weights reads dog → bites → man. Each hidden state depends on the current token and previous state; the final state is a finite-dimensional sequence summary.

The second term, W_h h_(t−1), is what makes the network recurrent: the output of the network feeds back into itself. Let's write this loop by hand, with tiny sizes so that every number is visible: 4 numbers per input word and 3 numbers in the hidden state.

After reading "dog", the hidden state is [0.999, −0.681, 0.204]. After "bites" it has changed to [−0.941, 0.98, 0.626], and after "man" to [0.672, 0.388, 0.319]. Every value stays between −1 and +1, because of tanh. The final hidden state is the network's summary of the whole sentence.

PyTorch provides the same thing as nn.RNN. If we copy our weights into it, it should give identical results:

It matches. Note the two outputs of nn.RNN: out holds the hidden state at every step, shape (batch, T, hidden) = (1, 3, 3), and h_last holds only the final hidden state, shape (layers, batch, hidden) = (1, 1, 3). With batch_first=True, PyTorch expects inputs as (batch, T, features).

Common mistake: forgetting batch_first=True. Without it, PyTorch's recurrent layers expect (T, batch, features), and will silently treat your batch dimension as time if the shapes happen to fit.

Now the key test. Does the RNN tell "dog bites man" from "man bites dog"?

Yes. The final states are very different, with a cosine similarity of only 0.2831, compared with 1.0 for averaging. Because each step depends on everything before it, the same words in a different order produce a different memory. The RNN is order-aware.

Key terms

• Recurrent neural network (RNN): a network that processes a sequence one step at a time, carrying a hidden state forward.
• Hidden state: the RNN's running memory; a vector updated at every step.
• Unrolling: drawing (or computing) an RNN as one copy of the cell per step, connected by the hidden state.

Order-awareness is a good start. But consider this sentence:

"The film, which was shot over three years in four countries with a cast of hundreds and a famously difficult director, was a triumph."

To know that "was" (singular) is correct, the network must connect it to "film", many words earlier. A connection like this, between items far apart in a sequence, is a long-term dependency. Can an RNN learn one?

The RNN learns, like every network, through backpropagation. Recall from Module 1 that backpropagation is credit assignment: it works out how much each input and weight contributed to the error. For an RNN, the gradient has to travel backwards through every step, from the final output back to the early inputs. Let's measure how much of it arrives.

We feed a random sequence of 60 inputs into an untrained RNN, take its final output, and backpropagate. The size of the gradient at each input position tells us how strongly the final output depends on that input, and so how much that input can be "learned from". We average over 3 random starting weights.

The gradient shrinks dramatically with distance: about 1.9 for the very last input, 0.055 five steps back, 0.0029 ten steps back, 0.0000023 twenty steps back, and 8.8e-12 forty steps back. (The notation 8.8e-12 means 8.8 × 10⁻¹², a decimal point followed by eleven zeros and then 88.) For practical purposes, an input 40 steps back has no influence on learning at all. This is the vanishing gradient problem, which you met in Module 3. In an RNN it is especially severe, because the "depth" of the network is the length of the sequence.

Every step backwards multiplies the gradient by the same kind of factor: the recurrent weights W_h, times the slope of tanh. If that factor is a little less than 1, repeating it many times drives the result towards zero; if it is a little more than 1, the result explodes.

Multiplying by 0.9 forty times leaves 0.0148; multiplying by 0.5 forty times leaves 9.1e-13, essentially nothing. Multiplying by 1.1 forty times gives 45.3: the opposite problem, an exploding gradient. A factor of exactly 1 at every step is almost impossible to maintain.

The tanh function makes vanishing more likely. Its slope is 1.0 at zero but drops quickly away from zero: 0.42 at ±1, 0.071 at ±2 and 0.01 at ±3. So whenever the hidden state is pushed towards ±1, which tanh does to large values, the gradient flowing back through that step is multiplied by a small number. Here is the measurement from Section 3.1 as a plot. On a logarithmic scale, a constant shrinking factor per step shows up as a straight downward line.

A vanishing gradient is a warning sign, but the real question is whether the network can still learn a long-range task. So let's set it one. Each sequence is a list of random tokens. The first token is a clue, 0 or 1, and the label is simply that clue. Everything after it is noise. To succeed, the network must remember the first token all the way to the end, and it classifies using only its final hidden state. By changing the sequence length, we control exactly how far the information has to travel.

The two example sequences start with 0 and 1, and their labels are 0 and 1. Now we train an RNN at four lengths, with 3 different random seeds each, and measure accuracy on 1,000 validation sequences. Guessing would score about 0.5, and each validation sequence is worth 0.001 of accuracy.

At length 10, every seed solves the task perfectly (1.0). At length 30, only one of the three seeds succeeds; the other two score 0.475 and 0.461, no better than guessing. At lengths 50 and 100, every seed fails, with accuracies between 0.494 and 0.533.

The task itself is trivial: "repeat the first token". The RNN fails not because the task is hard, but because the error signal from the end of the sequence cannot reach the first step to teach it what to remember. This is the long-term-dependency problem, measured directly.

Key terms

• Long-term dependency: a relationship between items far apart in a sequence.
• Vanishing gradient: a gradient that shrinks towards zero as it travels back through many steps, so early inputs cannot be learned from.
• Exploding gradient: a gradient that grows uncontrollably as it travels back.

This section goes a little deeper than the rest of the note. The key idea, that gates let a network choose what to keep, is worth understanding; the details are optional.

The vanilla RNN overwrites its entire memory at every step: the new hidden state is a fresh tanh of a mix of input and old memory. The Long Short-Term Memory (LSTM) network, designed specifically to solve the problem we just measured, changes this. It keeps a separate memory, the cell state, and controls it with gates.

A gate is a set of numbers between 0 and 1, produced by a sigmoid function, that is multiplied element by element with some other vector. A gate value of 1 lets everything through; 0 blocks everything. The LSTM has three:

• the forget gate decides how much of the old cell state to keep;
• the input gate decides how much new information to write;
• the output gate decides how much of the cell state to reveal as the hidden state.

The crucial change is in how memory is carried forward. Instead of being squashed through tanh at every step, the cell state is updated mostly by multiplying by the forget gate and adding new information. If the forget gate stays close to 1, the memory, and the gradient flowing back along it, passes from step to step almost unchanged.

Figure 5.4.2. The forget gate retains old cell state, the input gate admits a candidate, and the output gate exposes tanh of the new state. A near-one forget gate helps preserve the direct memory path; it does not guarantee perfect long-range learning.

Here is a detail that is easy to miss. The forget gate is computed with a sigmoid of a weighted sum plus a bias. PyTorch starts all biases at around zero, so at the start of training the forget gate sits at about sigmoid(0):

In this zero-logit illustration, the forget gate is 0.500, retaining half the old cell state per step along this direct path. Actual gates vary with the inputs, previous state, random weights and biases; PyTorch does not fix them all at exactly 0.5. After 40 steps that leaves 9.1e-13, the same vanishing as a plain RNN. If instead we set the forget-gate bias so that the gate starts at sigmoid(3) = 0.953, the memory kept after 40 steps is 0.143, enormously more. (Each LSTM has two bias vectors that are added together, so setting each to 1.5 gives a total of 3.) Starting the forget gate open is a well-known practical technique.

Let's test both versions of the LSTM on the same memory task.

The results are striking:

• The default LSTM does no better than the vanilla RNN. It solves length 10, but fails at length 30 on all three seeds (0.471 to 0.502), and at every longer length.
• The LSTM with its forget gate open scores 1.0 on every seed, at every length, all the way to 100.

The plot shows all three models; each line is the average over seeds, and the faint dots are individual seeds.

The gradient measurement from Section 3.1 tells the same story from the inside:

Forty steps back, the gradient is 8.8e-12 for the RNN, 6.1e-09 for the default LSTM, and 0.30 for the open-gate LSTM. The open forget gate keeps the gradient path alive.

So the honest summary is: gates make long-term memory possible, but they have to be used that way. An LSTM's architecture allows memory to persist; whether it does depends on the gate values, which are learned, and which need a good starting point.

The Gated Recurrent Unit (GRU) is a simplified relative of the LSTM. It merges the cell state and hidden state and uses two gates instead of three, so it is a little cheaper. With an input size of 16 and a hidden size of 32, a vanilla RNN has 1,600 parameters, a GRU 4,800 and an LSTM 6,400: each gate needs its own weights.

Key terms

• LSTM (Long Short-Term Memory): a recurrent network with a protected cell state controlled by forget, input and output gates.
• GRU (Gated Recurrent Unit): a simpler gated recurrent network with two gates.
• Gate: a vector of values between 0 and 1 (from a sigmoid) that scales another vector, letting information through or blocking it.
• Cell state: the LSTM's long-term memory line.

Visual walkthrough: Inspect one LSTM memory update.

Follow one scalar coordinate through the forget, input and output gates. The values are a worked example.

These supplied numbers illustrate the calculation. The web app adds a paused, step-by-step animation of the same example.

If an LSTM with a well-set forget gate can carry a fact across 100 steps, why did the field move on? Because remembering one fact is not the only requirement. Real language tasks expose two deeper limitations that no choice of gates can remove.

Limitation 1: everything must squeeze through one fixed-size memory. However long the sequence is, the RNN carries forward only its hidden state.

With a hidden size of 32, the RNN carries forward a (1, 1, 32) state after 10 steps, and exactly the same 32 numbers after 1,000 steps. Every fact from a 1,000-word document (names, dates, who did what to whom) must be compressed into those 32 numbers before the end. Adding facts means overwriting others. This is called the information bottleneck.

Limitation 2: the steps must run one after another. To compute the hidden state at step 1,000, the RNN needs the state at step 999, which needs step 998, and so on: 1,000 steps that cannot be done at the same time. Modern hardware such as GPUs is fast precisely because it does thousands of calculations in parallel. A sequential loop wastes most of that power, which makes RNNs slow to train on large datasets.

Compare the last line of the output. Comparing every position with every other position, for all 1,000 positions, is a single matrix multiplication producing a (1000, 1000) grid of scores. All of it can be computed at once. That operation is the core of attention.

Attention takes a completely different approach. Instead of passing information along a chain, step by step, it lets each position look directly at every other position and decide how much to take from each one.

Here is the idea in one picture. To build its representation, the last word in a sentence asks a question, its query. Every word in the sentence offers a key, a description of what it contains, and a value, the information it can contribute. The query is compared with every key to produce a score; the scores are turned into weights that sum to 1 (with softmax); and the result is a weighted average of the values. A word that is relevant to the query gets a large weight, however far away it is.

Figure 5.4.3. Recurrent information follows sequential updates; attention can directly mix allowed context positions. Connection strengths are illustrative. A causal model can consult only the current and earlier positions.

Let's build the smallest possible version of this, a single attention step in which the last position attends to all 60 positions, and run the same gradient measurement as in Section 3.1. We will study queries, keys and values properly in Module 6; for now, watch what happens to the gradient.

Through attention, the gradient reaching an input 59 steps back is 3.9e-02, about the same as for an input 10 steps back (5.2e-02). Through the RNN, the input 59 steps back receives 7.3e-17. The most recent position gets a larger gradient under attention (2.4e-01) only because it also supplies the query. Here are all four measurements together:

The picture summarises this whole note:

• The vanilla RNN (blue) and default LSTM (orange) fall in straight lines: their gradients shrink by a constant factor per step.
• The open-gate LSTM (green) keeps its gradient, and even grows slightly towards the far past, but it still has the fixed-size bottleneck and still has to process one step at a time.
• Attention (red) is flat: every input is one step away from the output, so distance simply does not matter.

Attention solves all three problems at once:

1. No long chain: every position connects directly to every other, so gradients do not have to survive many multiplications.
2. No single bottleneck: each position keeps its own vector, and any position can draw on any other, so information does not all have to fit into one hidden state.
3. Parallel computation: all the comparisons are matrix multiplications that run at once on a GPU.

It also gives us what Lesson 5.3 was missing: contextual representations. When "bank" attends to "river", or "good" attends to "not", its new vector reflects its surroundings.

Attention has a price. Comparing every position with every other means the number of comparisons grows with the square of the sequence length. For 1,000 positions, that was a (1000, 1000) grid, a million scores. You will meet this cost, and how it is managed, in Module 6.

Common mistake: concluding that RNNs "cannot remember". As you measured, a well-initialised LSTM carried a fact across 100 steps perfectly. The reasons attention replaced recurrence for language are the fixed-size bottleneck and sequential computation, together with attention's direct access to every position.

Key terms

• Attention: a mechanism in which each position computes a weighted average of other positions' information, with weights based on relevance.
• Query, key, value: what a position is looking for, what each position offers as a description, and the information each position contributes.
• Information bottleneck: forcing all information through a fixed-size representation.
• Parallel computation: performing many calculations at the same time instead of one after another.

Sequences carry meaning in their order, but a bag of words, or an average of embeddings, is order-blind: "dog bites man" and "man bites dog" got identical representations (cosine 1.0).

An RNN reads one item at a time and carries a hidden state forward, so order changes its memory (final-state cosine 0.2831). But to learn, its gradient must travel back through every step, and it is multiplied by a shrinking factor each time. Forty steps back, it had fallen to 8.8e-12. On a task as simple as "remember the first token", the RNN succeeded at length 10 and failed beyond 30.

The LSTM adds a protected cell state controlled by gates. With its forget gate starting open, it solved the task at every length up to 100, and its gradient no longer vanished (0.30 forty steps back). With the default start, it failed just like the RNN. Gates make long memory possible; they do not guarantee it.

Even a good LSTM keeps two limitations: all information must pass through a fixed-size hidden state, 32 numbers whether the sequence has 10 steps or 1,000, and the steps must run one after another. Attention removes both. Each position looks directly at every other position and takes a relevance-weighted average, all in parallel. Its gradient reach was flat across all 60 positions. That idea, developed fully, is the Transformer, the subject of Module 6.

Figure 5.4.4. Order-aware recurrence, gated memory and direct attention address different limitations. Attention supports parallel training, while autoregressive generation still proceeds one new token at a time.