How Neural Networks Learn from Text

Here is the whole journey a piece of text takes before a network can learn from it. Every later section explains one step of it.

1. Raw text: "This movie wasn't GOOD.<br />It was great!"
2. Tokenisation: split the text into units called tokens, here words: this, movie, wasn't, good, it, was, great.
3. Token IDs: look each token up in a fixed vocabulary (a numbered list of all known tokens) and replace it with its number.
4. Embeddings: replace each ID with a learned vector, for example 32 numbers per word. Similar words should get similar vectors.
5. Model: combine the vectors and make a prediction, for example "positive review".

Figure 5.3.1. Token IDs are arbitrary vocabulary indices. An embedding lookup turns each index into a learned vector. The displayed IDs are an example vocabulary, not the recorded IDs of this experiment.

The first three steps are fixed preprocessing. Step 4 is where the learning happens, and it is the focus of this lesson.

We will work with IMDb movie reviews: 50,000 reviews from the Internet Movie Database, each labelled positive or negative. It is one of the most widely used datasets for learning about text, and we will return to it in Modules 7 and 8. The download happens only the first time you run the cell.

The file has 50,000 rows with two columns, review and sentiment. But 418 rows are exact duplicates: the same review text with the same label, appearing more than once. Duplicates are dangerous: if one copy lands in the training set and another in the validation set, the model is tested on a review it has already seen, and its score looks better than it really is. So we remove them, leaving 49,582 reviews (24,884 positive and 24,698 negative, almost perfectly balanced), and shuffle the order.

Common mistake: splitting a dataset into training and validation sets before removing duplicates. Copies of the same example then end up on both sides of the split, and validation accuracy is inflated. Deduplicate first, then split. You will meet this issue again, in more depth, in Module 8.

To keep this notebook fast, we use 20,000 reviews for training and 5,000 for validation. Here is the start of the first training review:

Look closely at that review and think about it from a network's point of view. Three problems stand out.

Problem 1: text is made of symbols, not numbers. A network can only do arithmetic. Every character does have a numeric code (its Unicode code point), but those codes are arbitrary labels.

The capital O that begins the review is stored as 79 and the r after it as 114. Nothing about 79 or 114 relates to meaning. By that numbering, "good" and "food" differ by one letter and would look almost identical, while "good" and "great" would look quite different. Arithmetic on character codes is arithmetic on spelling, not meaning.

Problem 2: texts have different lengths. The shortest training review has 10 words, the median has 173 and the longest has 2,470. But the networks you have built expect inputs of a fixed shape, such as a batch of shape (64, 784). Text needs a way to be cut and padded to a common shape.

Problem 3: the same meaning can be written in many ways. "GOOD", "Good" and "good." are the same word to a reader but different strings to a computer. The review also contains HTML line-break tags, <br />, left over from the web page it was copied from.

The rest of this lesson solves these problems one at a time: tokenisation deals with Problem 3 and prepares for Problem 1; token IDs and embeddings solve Problem 1; padding and truncation solve Problem 2.

Tokenisation means splitting text into the units the model will treat as its basic building blocks: the tokens. The simplest idea, splitting on spaces, already runs into trouble:

Splitting on spaces produces tokens such as 'GOOD.<br' and '/>It', which mix words with punctuation and HTML. Our small tokenize function does three things instead: it lowercases the text, removes the <br /> tags, and keeps only runs of letters and apostrophes. The result, ['this', 'movie', "wasn't", 'good', 'it', 'was', 'great'], is much cleaner.

Every tokeniser involves choices like these, and they matter. Lowercasing means "Apple" (the company) and "apple" (the fruit) become the same token. Dropping punctuation throws away the difference between "great!" and "great?". There is no perfect tokeniser, only one that suits the task. In this lesson we use whole words. In Module 6 you will meet subword tokenisers, which split rare words into common pieces, and which are widely used by modern language models.

Next we tokenise all 20,000 training reviews and count how often each word appears.

The training reviews contain 4,617,067 tokens but only 79,754 distinct words. The counts are extremely uneven: the 1,000 most common words cover 76.3% of all the tokens, and the 10,000 most common cover 94.3%. The most common words are tiny "function words" such as the, and, a and of.

The plot below shows this unevenness. Each point is one word, ranked from most common to least common, plotted against how often it occurs. Both axes are logarithmic (each step is a factor of 10).

The curve falls steadily: a handful of words occur hundreds of thousands of times, while the 10,000th most common word occurs only 24 times, and 32,674 words, about 40% of all distinct words, occur exactly once. This pattern, where a word's frequency falls roughly in proportion to its rank, is called Zipf's law, and it holds for almost every language and every large body of text.

It also tells us how to build a sensible vocabulary. Words that appear only once or twice are too rare for a model to learn anything reliable about. So we keep the 10,000 most common words and add two special tokens:

• <pad> (ID 0): a filler token used to make short texts as long as long ones (Section 9.1).
• <unk> (ID 1): "unknown", used for every word that is not in the vocabulary.

We then build two lookup tables: itos ("integer to string") maps an ID to its word, and stoi ("string to integer") maps a word to its ID.

Our example sentence has become the list of numbers [11, 17, 291, 49, 9, 13, 80]. Common words get small IDs because we numbered the vocabulary by frequency: the is ID 2 (right after the two special tokens), and great is ID 80.

On the validation reviews, 6.0% of tokens are not in the vocabulary and become <unk>. That is the price of a fixed vocabulary: the model will see <unk> in place of rare names, misspellings and unusual words.

Key terms

• Token: one unit of text the model works with; here, a word.
• Tokenisation: splitting text into tokens.
• Vocabulary: the fixed list of tokens the model knows, each with a number.
• Token ID: the number of a token in the vocabulary.
• <unk> / <pad>: special tokens for unknown words and for padding.

Token IDs solve the "symbols are not numbers" problem, but only on the surface. An ID is just a label, like a seat number. Word 80 is not "bigger" than word 49 in any meaningful way, and words 49 and 50 are not similar just because their numbers are close. Feeding the raw IDs into a network as if they were measurements would make it treat these arbitrary labels as quantities. We need a better way to give the network each word.

The classic way to give a network a categorical value without implying any order is one-hot encoding, which you have probably used for categorical columns in classical machine learning. Each word becomes a vector as long as the vocabulary, filled with zeros except for a single 1 at the word's ID.

Each one-hot vector has 10,002 entries, and exactly 1 of them is non-zero. One-hot vectors solve the ordering problem, since no word is "bigger" than another. But they create two new problems.

They are huge and wasteful. A single 200-token review becomes 200 × 10,002 = 2,000,400 numbers, almost all zeros.

They make every word equally different from every other word. A standard way to measure how similar two vectors are is their dot product: multiply matching entries and add them up. For one-hot vectors, the 1s are always in different places, so the dot product between any two different words is 0. As the output shows, "good" is exactly as similar to "great" (0.0) as it is to "terrible" (0.0). One-hot vectors carry no notion of meaning at all, so a model cannot transfer anything it learns about "great" to "excellent".

What we would like is a short vector for each word, made of ordinary numbers rather than mostly zeros, arranged so that similar words have similar vectors. That is exactly what an embedding is.

An embedding layer is, at heart, a lookup table: a matrix with one row per word in the vocabulary. Row 49 holds the vector for "good", row 80 holds the vector for "great", and so on. To embed a token, you simply fetch its row. In PyTorch this is nn.Embedding(num_embeddings, embedding_dim), where num_embeddings is the vocabulary size and embedding_dim is the length of each vector.

The crucial point is that the table's entries are ordinary learnable parameters. They start random and are adjusted by backpropagation, like any other weights, so that the vectors become useful for the task.

Our table has shape (10002, 8): one row of 8 numbers for each of the 10,002 tokens, which makes 80,016 parameters. Looking up "good" and "great" returns their two rows. The numbers are random, because nothing has been trained yet.

The last line of the output reveals what an embedding layer really is. Multiplying a one-hot vector by the table picks out exactly one row, so an embedding lookup gives the same result as one-hot encoding followed by a linear layer, and PyTorch confirms the two are identical (True). The embedding layer just skips the wasteful multiplication by thousands of zeros and fetches the row directly.

Figure 5.3.2. Selecting row 49 is equivalent to multiplying a one-hot vector by the table, without materialising that dense one-hot vector. Both table entries and the chosen row are illustrative.

Embedding layers work on whole batches of token IDs at once. A batch of shape (B, T) (B texts, each T tokens long) becomes a tensor of shape (B, T, D), adding one dimension of size D for the vector.

Two sentences of 4 tokens each, shape (2, 4), become (2, 4, 8): each of the 8 tokens has been replaced by its 8-number vector.

Common mistake: passing token IDs with a floating-point dtype to an embedding layer. Embedding layers need integer IDs (torch.long), because an ID is a row number, not a quantity.

Key terms

• Embedding: a learned vector of numbers that represents a token.
• Embedding layer: a learnable lookup table with one vector per vocabulary entry (nn.Embedding).
• Embedding dimension: the length of each vector (embedding_dim).

How long should each word's vector be? The output above shows the parameter cost for our 10,002-word vocabulary: 80,016 parameters with 8 dimensions, 640,128 with 64, and 3,000,600 with 300.

It helps to think of each dimension as one "axis of meaning" that the model can use to tell words apart. With only 2 dimensions you could perhaps separate positive words from negative words, and formal words from casual ones, and nothing else. More dimensions allow more distinctions: tone, topic, grammar, whether a word is about people or objects, and so on. But nobody tells the network what each axis should mean. The axes are learned, and usually no single dimension corresponds neatly to a human concept; the meaning is spread across all of them together.

The trade-off is familiar from Module 3:

• Too few dimensions: the vectors cannot hold enough distinctions, and the model underfits.
• Too many dimensions: many more parameters to learn, which needs more data and invites overfitting, especially for rare words.

Word embeddings trained on their own typically use somewhere between tens and a few hundred dimensions; the embeddings inside large language models are much longer. In this lesson we use 32 dimensions for the sentiment model and 50 for the second model in Section 9.

If embeddings are supposed to place similar words close together, we need a way to measure "close". For word vectors, the standard measure is cosine similarity.

Imagine each vector as an arrow pointing out from the origin. Cosine similarity looks only at the angle between two arrows, not at their length:

Here a · b is the dot product from Section 4, and ‖a‖ is the length (or norm) of vector a, the square root of the sum of its squared entries. Dividing by both lengths removes the effect of size, so only direction counts. The result always lies between −1 and +1:

• +1: the arrows point in exactly the same direction.
• 0: they are at a right angle; the vectors are unrelated.
• −1: they point in exactly opposite directions.

Let's check this on small 2D vectors whose angles we can picture:

[1, 1] and [2, 2] point the same way (one is just longer), so their cosine is 1.000, even though they are not equal. [1, 0] and [0, 1] are at a right angle: 0.000. [1, 0] and [1, 1] are 45° apart: 0.707. And [1, 1] and [−1, −1] point in opposite directions: −1.000.

Figure 5.3.3. Cosine similarity compares directions rather than lengths: 1 for the same direction, 0 for perpendicular vectors, and −1 for opposite directions. It is undefined for a zero vector without a numerical convention.

Now apply cosine similarity to the untrained embedding table from Section 5:

These numbers mean nothing. "Good" comes out more similar to "great" (−0.081) than to "terrible" (−0.682) or "the" (−0.714), but that is pure luck: the table is random. With only 8 dimensions, random vectors can easily have large positive or negative cosines by chance. An embedding table becomes meaningful only after it has been trained, and, as Section 9 will show, what it becomes depends entirely on what it was trained to do.

Key terms

• Dot product: multiply matching entries of two vectors and add the results.
• Norm (length): the size of a vector, the square root of the sum of its squared entries.
• Cosine similarity: the dot product divided by both norms; it measures the angle between two vectors, from −1 to +1.

Before we train anything, there is one limitation of embedding layers that is important to understand, because it explains why the field moved on to the Transformers of Module 6.

Consider the word "bank" in these two sentences:

• She sat by the bank of the river.
• He deposited cash at the bank.

To a reader, these are two different meanings. But an embedding layer is a lookup table, and "bank" has only one row. So it must receive exactly the same vector in both sentences:

The cosine similarity is 1.0: the two vectors are identical. This is called a static representation: each word gets one fixed vector, whatever its surroundings. The lookup table alone cannot change the vector of "bank" with context. A model that simply averages these vectors also loses word order and struggles with negation. An order-aware model can instead build contextual representations from the same static starting vectors.

A contextual representation, by contrast, gives each word a vector that depends on the words around it. "Bank" near "river" and "bank" near "cash" would get different vectors. Producing contextual representations requires a model that lets words exchange information with one another. Sequence models (Lesson 5.4) and, above all, Transformers (Module 6) do exactly this. They still begin with an embedding layer like ours, and then transform those static vectors into contextual ones.

Figure 5.3.4. A static table returns the same vector for a token type in every sentence. A contextual model produces different output vectors as surrounding tokens change; a static vector can still encode multiple usages.

Key terms

• Static representation: one fixed vector per word, the same in every context.
• Contextual representation: a vector for a word that depends on the surrounding words.

Now we train embeddings for real, in two different ways, and compare what they learn.

Reviews have different lengths, but a batch must be a rectangular tensor. The standard solution has two parts:

• Truncation: cut every review to at most MAX_LEN tokens; we use 200.
• Padding: fill shorter reviews up to 200 with the <pad> token (ID 0).

The training set is now a tensor of shape (20000, 200): 20,000 reviews, each exactly 200 token IDs. Truncation affects 40.4% of training reviews, which are longer than 200 tokens, and 20.9% of all positions in the tensor are padding.

Truncating that many reviews loses information, because the end of a review often contains its verdict. It is a deliberate trade-off between accuracy and speed. Padding, meanwhile, must not influence the prediction: a review should not become more negative just because it is short. We deal with that next.

Our first model is deliberately simple. It is a bag of embeddings: look up the vector for every token, average the vectors of the real (non-padding) tokens, and feed that one average vector to a linear layer that outputs a single score. A positive score means "positive review". The loss is BCEWithLogitsLoss, the binary cross-entropy loss for two-class problems.

The line that handles padding is the mask: it is 1 for real tokens and 0 for padding, so that padding vectors are multiplied by zero and excluded from the average. We also pass padding_idx=PAD to the embedding layer, which keeps the <pad> vector fixed at zero and never trains it.

The embedding table is trained only through the sentiment loss. The words' vectors are free to become whatever helps tell positive reviews from negative ones. We train for 4 epochs and keep the epoch with the best validation accuracy.

Validation accuracy rises to 0.8646 at epoch 3, then slips to 0.8632 at epoch 4 while the training loss keeps falling. That is the first sign of overfitting, so we keep the epoch 3 weights. Not bad for a model that ignores word order completely.

Now the interesting part: what did the embedding table learn? For a few words, we find the nearest neighbours: the vocabulary words whose vectors have the highest cosine similarity to that word's vector.

The sentiment words have very sensible neighbours. "Excellent" sits next to wonderfully, best and fantastic; "awful" sits next to poorly, dull, waste and worst. But look at the neutral words. "Husband" is next to betray, aimed and crude; "music" is next to testament, ida and evokes; "actor" is next to characterization and diane. These neighbours are less clearly related in meaning, and their similarities (around 0.55 to 0.79) are noticeably lower than those of the sentiment words (around 0.85).

This makes perfect sense once you remember what the table was trained for. The sentiment loss only ever asked one question: does this word push a review towards positive or negative? So the embeddings organised themselves along that single question. Words with no sentiment, like "husband" and "music", received no direct supervision for general semantic similarity.

Can we learn embeddings that capture general meaning instead? A famous idea from linguistics says yes: "You shall know a word by the company it keeps." Words that appear in similar contexts tend to have similar meanings. "Husband" and "wife" both appear near "married", "her", "his" and "divorce"; "music" and "soundtrack" both appear near "score", "songs" and "beautiful".

Skip-gram turns this idea into a training objective. It needs no labels at all, only raw text:

1. Take a word in the text, the centre word.
2. Look at the words up to a few positions on either side, the context words. Each (centre, context) pair is a positive example: "these two words appeared together".
3. Also pick a few random words from the vocabulary, called negative samples or "noise" words. Each (centre, random word) pair is a negative example: "these probably did not appear together".
4. Train two embedding tables so that the dot product of a centre word's vector with a true context word's vector is high, and with a noise word's vector is low.

This is the training method behind word2vec, one of the most influential models in natural-language processing. Because it learns from text alone, it is an example of self-supervised learning: the training labels ("did these words appear together?") come from the data itself.

First we build the training pairs. We also randomly drop many occurrences of very frequent words such as "the", a step called subsampling, because very frequent words dominate the pair counts. We build each window within one review, so the end of one review never becomes the context of another.

From 15,000 training reviews, with a window of up to 5 words on each side, subsampling keeps 1,307,513 tokens, which produce 12,625,140 (centre, context) pairs. Now we train, for one pass over all the pairs.

Inside the loop, pos is the dot-product score for each true pair and neg holds the scores for the 5 noise words of each pair. The loss uses logsigmoid, the logarithm of the sigmoid function, to push pos up and neg down. It is the same idea as binary cross-entropy, with the true pairs labelled 1 and the noise pairs labelled 0.

The loss falls from an average of 3.407 over the first 100 steps to 2.598 over the last 100. Now the same nearest-neighbour test, on the skip-gram embeddings:

The neutral words now have more clearly related neighbours. "Husband" sits next to wife, mother and daughter; "music" next to soundtrack, sound, songs and score; "actor" next to comedian and talented, as well as actors' names. The sentiment words also have plausible neighbours: "excellent" next to superb, flawless and exceptional, and "awful" next to terrible, bad and horrendous. These are observations from this run, rather than guaranteed semantic groupings.

Nearest-neighbour lists are easy to cherry-pick, so let's measure the difference more systematically. We take three sets of word pairs:

• related pairs, which a reader would call similar in meaning, such as husband–wife, music–soundtrack and film–movie;
• opposite pairs with opposite sentiment, such as excellent–awful and love–hate;
• 200 random pairs drawn from the 2,000 most common words, as a baseline.

We compute the average cosine similarity within each set, for three embedding tables: a random untrained one, the sentiment one and the skip-gram one.

Always read these numbers against the random column, which tells you what "no relationship" looks like in each table.

• Untrained: all three averages are close to zero (−0.076, 0.085 and 0.011). As expected, random vectors encode nothing.
• Sentiment: related pairs average only 0.141, barely above random (−0.001). But opposite pairs are strongly negative, at −0.609. This table has learned one thing very well, positive versus negative, and little else.
• Skip-gram: related pairs average 0.830, far above its random baseline of 0.408. Notice the baseline itself is high: skip-gram vectors tend to share a common overall direction, which is why comparing against random pairs matters. The surprise is the opposite pairs: 0.495, above the random baseline. To skip-gram, "excellent" and "awful" are somewhat similar, because they appear in the same kinds of contexts ("the acting was ___", "an ___ film").

Within the 20 words of the picture below, the same story shows up group by group: the family words average −0.06 under the sentiment objective and 0.87 under skip-gram, while the positive words are tightly grouped in both (0.77 and 0.83).

Here is a 2D picture of 20 words from four groups in each table. Each panel is a PCA projection (principal component analysis, a method that finds the two directions along which the points vary most and flattens everything onto them). A 2D picture of 32- or 50-dimensional vectors always loses information: these pictures keep 58% and 67% of the variation in the 20 vectors, so treat them as illustrations of the numbers above, not as proof.

In the sentiment panel, positive and negative words form two tight clusters at opposite sides, while family and music words are scattered without structure. In the skip-gram panel, family and negative words form clear groups, while positive and music words also show structure but partly overlap in this 2D projection. The full-dimensional cosine measurements provide the stronger comparison.

This is the key lesson of this lesson: an embedding learns whatever its training objective rewards. Both demonstrations use learnable embedding tables and the same vocabulary, but they are not an objective-only controlled experiment: sentiment uses 32 dimensions and 20,000 reviews, while skip-gram uses 50 dimensions and 15,000 reviews, with a different prediction head and training procedure. The results illustrate task-shaped representations; they do not isolate the objective as the only cause of every difference. A sentiment objective builds a "sentiment axis". A context objective builds a map of how words are used, which captures topic and meaning but cannot, by itself, tell good from bad. Neither is "the true meaning" of a word; each is a representation shaped for a purpose. You saw the same idea with images in Lesson 5.1: a representation is good for a task.

Finally, let's go back to Section 8 and test our sentiment model on four short sentences.

"The movie was good" gets a probability of 0.982 of being positive, and "the movie was bad" gets 0.000. So far, so good. But "the movie was not good" still gets 0.569, more likely positive than negative, and "the movie was not bad" gets 0.000, as negative as "bad" itself.

The model cannot understand negation. It sees each word's static vector, averaged together, with no idea that "not" changes the meaning of the word after it. Averaging also throws away word order completely. Fixing this needs a model that processes the words in relation to one another, which is exactly where Lesson 5.4 begins.

Common mistake: judging embeddings by a few hand-picked neighbour lists. As you saw, a sentiment table gives perfect neighbours for "awful" and nonsense for "husband". Always test several kinds of words, and compare against a random baseline.

Let's retrace the full path from text to meaning, now that you have built every step.

Raw text is a sequence of symbols, of varying length and inconsistent formatting, so a network cannot use it directly. Tokenisation splits it into tokens; we used lowercase words. A vocabulary of the 10,000 most frequent words covered 94.3% of training tokens, thanks to Zipf's law, with <unk> for the rest. Token IDs turn tokens into numbers, but those numbers are only labels. One-hot vectors remove the false ordering but are enormous, and make every pair of words equally different (dot product 0).

An embedding layer replaces each ID with a short, learnable vector. It is equivalent to one-hot encoding followed by a linear layer, but much cheaper. Cosine similarity measures how close two embeddings point. But the vectors mean something only after training, and they become whatever the training objective rewards:

Embedding layers are static: "bank" gets the same vector in every sentence (cosine 1.0), and a model built only from averaged static vectors cannot handle "not good" (still 0.569 positive). Getting contextual representations requires models that combine words with one another, the subject of Lesson 5.4 and Module 6.

Figure 5.3.5. Illustrative geometry, not the experiment’s measured projection. Objectives, data, dimensions and training can all change an embedding space. Read the actual cosine tables and PCA plots above for evidence from this run.