Synthesis: Putting Representation Learning into Practice

To compare representations fairly, we reuse the three-way split from Lesson 5.1. Each group of images has one job, and no image is used for more than one job:

• 40,000 images are used only to train the two encoders, so that they learn their representations.
• 10,000 images are used to train the downstream classifiers (the linear probes) and as the reference set for nearest neighbours.
• 10,000 images are held out, and every accuracy in this lesson is measured on them.

We train two encoders, each with a 10-class head on top, for 3 epochs each:

• an MLP encoder: flatten, then one hidden layer of 128 units, the same as in Lesson 5.1;
• a CNN encoder: the two convolution blocks from Lesson 5.2, followed by a fully connected layer of 64 units.

After training, we keep only the encoders and discard the heads, exactly as in Lesson 5.1.

Both encoders learn normally: the MLP's average training loss falls from 0.649 to 0.404, and the CNN's from 0.742 to 0.357. Now we pass every image through each encoder and collect the representations.

Each image now has three descriptions: 784 raw pixel values, 128 MLP features and 64 CNN features. The CNN's description is the most compact of the three.

As in Lesson 5.1, we measure each representation by training the same simple classifier on top of it: a standardised logistic regression, trained on the 10,000 probe images and evaluated on the 10,000 validation images. Any difference in accuracy comes from the representation.

The ranking is clear: raw pixels reach 0.803, MLP features 0.867 and CNN features 0.882. The CNN's representation is the most useful and the smallest.

But a difference of 0.015 between the MLP and the CNN is only 150 images out of 10,000. Could it be luck? We use the sign test from Lesson 5.2: look only at the images where two classifiers disagree, and ask whether the split could plausibly be a coin flip.

For these fitted classifiers, the paired tests provide strong evidence of accuracy differences on this validation sample; they do not establish that every retraining will have the same ranking. Against raw pixels, the CNN-feature classifier was right on 1,143 images that the raw-pixel classifier got wrong, against only 351 the other way round (p = 6.9e-98). Against MLP features the margin is smaller but still convincing: 477 against 319 (p = 2.4e-08). With 10,000 validation images, each image is worth 0.0001 of accuracy, so even modest accuracy gaps rest on hundreds of images.

An overall accuracy hides a lot. Let's compare raw pixels and CNN features class by class.

The gains are very uneven. Trousers barely change (+0.012): their long two-legged shape is already obvious in raw pixels, so there is little left to gain. The biggest gains are for coats (+0.122), pullovers (+0.114), dresses and shirts (+0.112 each) and T-shirts (+0.110), the classes that look alike in outline and differ in details such as sleeves, collars and texture. Those are exactly the local patterns that convolution filters are designed to pick up.

Notice also what the learned representation did not fix. Shirts remain by far the hardest class, at 0.640 even with CNN features. We will look at why in Section 1.6.

In Lesson 5.1 we saw that good representations shine when labelled data is scarce. Let's check that the CNN representation holds up here too: we train each linear probe on just 100 labelled images, about 10 per class, repeated with 5 random draws.

With 100 labels, CNN features reach 0.838 on average, against 0.792 for MLP features and 0.669 for raw pixels. The spread across the 5 draws is narrower for the CNN, too (0.826 to 0.850), than for raw pixels (0.613 to 0.698). With only 100 labels, the CNN representation already beats raw pixels trained with all 10,000 labels (0.803).

A representation is a kind of map in which every image has a position. A good map should place images of the same kind close together. We can check this directly with nearest neighbours: for a query image, find the images whose representations are closest to it (here, by straight-line distance).

Let's start with the first four validation images, chosen by position, not because they make any representation look good:

For these four queries, an ankle boot, a pullover, a pair of trousers and a T-shirt, both spaces find almost entirely correct neighbours. The one mistake shown is a CNN neighbour of the ankle boot that is actually a sneaker. Easy images are easy in any reasonable representation.

To see a difference, we need harder cases. Shirts were the hardest class, so here are the first four validation shirts, again chosen by position, not by outcome:

This picture is humbling. For the first shirt, both spaces find shirts. But look at the second "shirt": it looks much more like a dress, and both spaces return dresses. The fourth looks like a T-shirt, and both spaces return T-shirts. Some of these mistakes say more about the data than about the representation. At 28×28 pixels, some shirts genuinely look like other garments, and no representation learned from these pixels can recover a distinction that the pixels do not contain.

The numbers below the picture put the four examples in context. Across 2,000 validation images, 0.782 of the 5 nearest neighbours share the query's class in raw-pixel space, against 0.841 in CNN space. For shirts alone the agreement is far lower in both spaces, 0.542 and 0.576. Four examples could have suggested that the CNN space is worse for shirts. The measurement over 200 shirts says it is slightly better, and that shirts are hard for everyone.

We can turn this neighbour idea into a classifier: k-nearest neighbours (k-NN) predicts a validation image's class by majority vote among its 5 nearest probe images. Its accuracy summarises how well each representation groups classes together.

The ranking is the same as for the linear probes: 0.828 for raw pixels, 0.850 for MLP features and 0.877 for CNN features. Two very different ways of using a representation, a linear classifier and a neighbour vote, agree about which one is best. These full-dimensional k-NN numbers are our reference point for the pictures that follow.

Common mistake: judging a representation from a handful of examples. Four shirts could have suggested the opposite of what 200 shirts show. Always back examples with a measurement over many cases.

People love to look at representations, and pictures can be genuinely illuminating. But a representation with 64 or 784 dimensions cannot be drawn on a flat page without throwing information away. The question is always: how much does the picture distort? Here we answer it with the same k-NN measurement: if a 2D picture preserved the neighbourhoods of the full representation, k-NN in the picture would be about as accurate as k-NN in the full space.

Principal component analysis (PCA) finds the two directions along which the data varies most and projects everything onto them. It is a linear projection: like casting a shadow of a 3D object onto a wall, it keeps the overall shape but flattens everything else.

The 2D PCA view keeps 47% of the variance (spread) of the raw pixels and 69% of the CNN features. More importantly, look at what happens to the neighbourhoods. In 2D, k-NN accuracy drops to 0.504 for raw pixels (from 0.828 in full dimensions) and to 0.617 for CNN features (from 0.877). The flat picture loses a large part of the structure in both cases, so we should read it with care.

The CNN panel looks more organised: trousers, bags and footwear form clear separate groups, while all the upper-body garments are piled together in one corner. The raw-pixel panel looks like a blurred ring. That impression is consistent with the numbers: CNN features are better organised. But neither picture is a faithful map. The heavy overlap in the CNN corner does not mean those classes are inseparable; in the full 64 dimensions, a linear classifier separates them far better than the picture suggests.

t-SNE (t-distributed stochastic neighbour embedding) takes a different approach. Instead of projecting, it moves points around on the page, trying to keep each point's nearest neighbours near it. It does not try to preserve large distances. It is widely used because it produces striking, clustered pictures. t-SNE is slower than PCA and cannot place new points without re-running, so we apply it to 2,000 validation images, and we measure k-NN on those same 2,000 images with 5-fold cross-validation (train on four-fifths, test on the remaining fifth, five times), both in the t-SNE picture and in the full representation.

Note that this full-dimensional k-NN uses only 2,000 images as its reference set instead of 10,000, so its accuracies are lower than in Section 1.6. That is fine: what matters is comparing the picture with the full space on the same images.

This is a very different story from PCA. In the t-SNE pictures, k-NN accuracy is almost as high as in the full space: 0.769 against 0.783 for raw pixels, and 0.841 against 0.846 for CNN features. Here, t-SNE retains class information useful to k-NN. Similar classification accuracy does not prove that the identities of individual neighbours are preserved. This is a transductive visualisation diagnostic: t-SNE was fitted to all 2,000 points before cross-validation, so this is not an estimate of performance on unseen points. To measure neighbour preservation directly, use neighbour overlap or trustworthiness.

The CNN picture shows tight, well-separated clusters for trousers, bags, sandals, sneakers and ankle boots, with the upper-body classes closer together and partly mixed. But notice that the raw-pixel picture also looks quite organised, much more so than its PCA picture. t-SNE is very good at making clusters visible. That is its strength, and also a reason for care: a clustered t-SNE picture on its own is not evidence that a representation is good. Here the numbers tell us which exposes more class information for this full-dimensional k-NN diagnostic (0.846 against 0.783), and the pictures only illustrate it.

If t-SNE keeps neighbourhoods, can we read the rest of the picture too, such as which clusters are far apart? Let's re-run t-SNE on the same CNN features with different random starting points and see what changes.

The class-prediction scores are similar across these runs: k-NN in the picture stays at 0.845 and 0.838, close to the 0.841 of the first run. But the global layout is not. The two classes whose cluster centres end up farthest apart are trousers and sandals in one run, trousers and bags in another, and coats and ankle boots in the picture above. The same data, with a different random start, gives a different arrangement of clusters.

So the rule for reading t-SNE pictures is:

• Use as hypotheses: local groupings that remain stable across settings; verify important neighbours in the full representation.
• Do not trust: distances between clusters, the sizes of clusters, or the empty space between them.

Figure 5.6.1. Conceptual projection analogy. PCA retains directions of greatest variance; t-SNE emphasises local neighbourhoods. Both can distort structure, so validate claims using appropriate measurements in the full representation space.

Common mistake: concluding from a t-SNE plot that "class A is very different from class B because their clusters are far apart". t-SNE does not preserve distances between clusters, and the layout changes between runs. Compare distances in the full representation instead.

Key terms

• k-nearest neighbours (k-NN): classifying an item by majority vote of the k most similar items.
• PCA: a linear projection onto the directions of greatest variance.
• Explained variance: the share of the data's spread that a PCA projection keeps.
• t-SNE: a non-linear method that arranges points in 2D to preserve each point's nearest neighbours.
• Cross-validation: repeatedly splitting data into training and test parts and averaging the results.

Let's step back and answer the question that has run through all of Module 5.

The description changes, and the description decides what is easy. Raw data such as pixels and characters is complete but unhelpful: individual values mean little, and important patterns are spread across many of them. A learned representation re-describes each example in terms of features that matter for a task. In this lesson, the same linear classifier went from 0.803 on raw pixels to 0.882 on 64 CNN features, a difference confirmed by a sign test (p = 6.9e-98).

The architecture shapes what can be learned. The CNN's representation beat the MLP's (0.882 against 0.867, p = 2.4e-08) while using half as many numbers, because convolution builds in the knowledge that nearby pixels matter. That same idea runs through the module: CNNs for images, embeddings for words, recurrence and then attention for sequences. Each design encodes an assumption about the data.

The objective shapes what is learned. In Lesson 5.3, the same embedding layer learned a sentiment axis or a map of word usage depending purely on the training objective. In Lesson 5.5, contrastive learning shaped two encoders so that images and text shared a space. A representation is always good for something.

Good representations save labels. With 100 labels, CNN features (0.838) beat raw pixels with 10,000 labels (0.803). This is the reason pretrained models matter so much: someone else has already spent the data and compute to learn a good representation, and you can reuse it.

Not everything can be learned away. Some shirts look like dresses or T-shirts at this resolution, and no representation could fix that (5-NN agreement for shirts: 0.542 raw, 0.576 CNN). Representation learning extracts what is in the data; it cannot invent what is not.

Evaluate with numbers, illustrate with pictures. Linear probes, k-NN, per-class breakdowns and sign tests told us what was true. PCA pictures lost much of the structure (k-NN 0.617 against 0.877), and t-SNE retained similar class-prediction scores in this transductive diagnostic (0.841 against 0.846) but not the layout between clusters, which changed from run to run.

Figure 5.6.2. Module 5 connects representation learning for images, text and sequences with shared spaces and measurement. A learned representation need not be compact or have individually interpretable coordinates.

Where this leads. Module 6 takes the attention idea from Lesson 5.4 and builds it into the Transformer, the architecture that produces contextual representations for text, and, as Vision Transformers, for images too. Modules 7 and 8 then show how to reuse and adapt representations that have already been learned at enormous scale.