Understanding Representation Learning

Let's start with a question that sounds almost too simple: what does a computer actually receive when you give it a photo?

We will use Fashion-MNIST throughout this module. It is a collection of small greyscale photos of clothing items, each labelled with one of ten categories such as Trouser, Sneaker or Bag. It was created as a slightly harder replacement for the famous handwritten-digit dataset MNIST, and it is small enough to train on a laptop in minutes. In this lesson we only need its 60,000 training images.

The cell below downloads the training images and labels (only the first time you run it) and loads them into NumPy arrays.

The output tells us that images has shape (60000, 28, 28): sixty thousand images, each a grid of 28 rows by 28 columns. The dtype is uint8, which means each entry is a whole number from 0 to 255. There are 60,000 labels, one per image, covering the 10 classes.

Now let's look inside a single image. The first image in the dataset is an ankle boot. Here is a small window of it, rows 8 to 15 and columns 8 to 19, printed exactly as the computer stores it:

Each number is the brightness of one pixel (short for picture element, one tiny square of the image): 0 is black, 255 is white. That is all the computer has. There is no "boot" anywhere in these numbers, no "heel", no "laces". There are only 784 brightness values (28 × 28) arranged in a grid.

To make this concrete, the figure below shows the whole image as we see it on the left, and the window inside the orange box as the computer sees it on the right.

You can recognise the boot instantly. But notice what your brain did: it ignored individual brightness values and picked out edges, shapes and parts, then combined them into "boot". Those are not in the data. They are a description you built from the data.

Text has exactly the same problem. A sentence reaches the computer as a sequence of characters, and each character is stored as a number (its Unicode code point, a standard numbering of every character in every writing system):

The sentence has 32 characters, and the capital T is stored as 84. Nothing in the number 84 says "this word is positive about a film". Meaning is not in the raw symbols; it has to be built from them.

This is the core problem representation learning solves. We can now define the key term properly.

Key terms

• Raw data: data in the form it was recorded: pixels, characters, audio samples. Individually, these values carry little meaning.
• Feature: a single number that describes something useful about an example, such as "how tall is the object in this image?"
• Representation: the full list of features used to describe an example. Think of it as the description the model works from. A representation is usually a vector (a list of numbers), so you will also hear it called a feature vector, and for text and other discrete things, an embedding.
• Representation learning: training a model so that it discovers useful features from raw data by itself, instead of a person designing them.

A helpful way to picture this: imagine describing a stranger to a friend so they can spot them in a crowd. Listing the colour of every square centimetre of their clothing would be accurate but useless. "Tall, red jacket, curly hair, carrying a guitar" is far shorter and far more useful. A good representation is like that second description: compact, and focused on what matters for the task.

Figure 5.1.1. The boot is described first by pixel intensities, then by task-useful features. The named properties are an intuition; learned coordinates need not have individual human-readable meanings or fewer dimensions.

Now that we know what the problem is, let's see the overall shape of the solution before we look at any details.

Every deep-learning model in this module can be read as two parts working one after the other:

1. An encoder takes raw data and turns it into a representation. In a neural network, the encoder is simply the early layers. Each layer transforms the numbers it receives into a new set of numbers, and the output of the last encoder layer is the representation.
2. A head takes the representation and turns it into the final answer: a class, a score, a next word. The head is usually small, often a single linear layer, because if the representation is good, the final step should be easy.

Both parts are trained together with the same loss and the same backpropagation you already know. Nobody tells the encoder which features to build. It builds whatever features make the head's job easiest, because those are the features that reduce the loss.

Figure 5.1.2. The encoder builds a representation and the head uses it. In joint supervised training, one loss sends gradients through both parts. The displayed class bars are illustrative; a linear head produces logits before softmax.

This split is worth remembering because it explains a lot of what comes later in the course:

• CNNs (Lesson 5.2) are encoders designed for images.
• Embedding layers (Lesson 5.3) are encoders for words and other symbols.
• Pretrained models (Module 7) are encoders that someone else has already trained on enormous datasets, which you can reuse with your own head.
• Fine-tuning (Module 8) is the process of adjusting a pretrained encoder for your task.

In the rest of this lesson, we will build all three kinds of description for Fashion-MNIST: features a person designs by hand, the raw pixels themselves, and features a network learns. Then we will compare them fairly.

Key terms

• Encoder: the part of a network that turns raw input into a representation.
• Head: the final part of a network that turns a representation into the output the task needs.
• Downstream task: any task that uses a representation as its input, such as classification, search or clustering.

Before deep learning became practical, the standard way to handle images was feature engineering: an expert studied the problem and wrote code to compute features they believed would be useful. You have done this in classical machine learning: creating ratios, flags and aggregates from raw columns. Let's try it for clothing images, so we have something concrete to compare against.

To compare different representations fairly, we need to use the data carefully. We split the 60,000 training images into three non-overlapping groups, each with a different job:

• 40,000 images (rep_idx) are used only to train a network so it can learn a representation.
• 10,000 images (probe_idx) are used to train the downstream classifier on top of each representation.
• 10,000 images (val_idx) are never trained on. Every accuracy in this lesson is measured on them.

Keeping these groups separate means that no representation gets an unfair peek at the images it is judged on.

If you had to tell clothing items apart using only a few numbers, what would you measure? A sensible person might say: how much of the image is filled in, how tall and how wide the object is, whether it is taller than it is wide, whether the top or bottom half is heavier, and whether there is a dark gap down the middle (as there is between trouser legs). The function below computes exactly those seven features.

Each image is now described by just 7 numbers instead of 784. The printed averages show that some of these features genuinely carry information. Trousers are tall and narrow: on average 27.90 pixels tall but only 11.62 wide, an aspect ratio of 2.46. Sneakers are the reverse: 12.80 tall, 27.39 wide, an aspect ratio of 0.47. A person's intuition has produced real signal.

But look at the "centre gap" feature, which was meant to pick out the dark space between trouser legs. Sneakers score lower (0.26) than trousers (0.37). The idea was reasonable, yet in the data it does not behave the way its designer expected. This is the typical story of feature engineering: some features work, some don't, and you only find out by measuring.

To compare representations, we give each one to the same simple classifier: a logistic regression, with the features standardised first (rescaled to mean 0 and standard deviation 1, so that no feature dominates just because its numbers are larger). This is called a linear probe. Because the classifier is deliberately simple, any difference in accuracy comes from the representation, not from a cleverer classifier.

With seven handcrafted features, the classifier reaches 0.694 validation accuracy. With all 784 raw pixels, it reaches 0.803. Random guessing among 10 balanced classes would give about 0.1, so both are far better than chance, but the raw pixels beat the handcrafted features.

That may surprise you. Shouldn't the expert's carefully chosen features win? Not necessarily. Seven numbers throw away most of the information in the image, and the expert can only encode the patterns they think of. The raw pixels keep everything. Fashion-MNIST images are also centred and similarly sized, so a given pixel position tends to mean roughly the same thing across images, which helps a linear model on raw pixels more than it would on real-world photos.

Now we try the third option. We train a small network, a multilayer perceptron (MLP, the stack of fully connected layers you built in Module 2), on the 40,000 representation-learning images. It has one hidden layer of 128 units. We split it explicitly into an encoder and a head, exactly as in Section 2.

The average training loss falls each epoch, so the network is learning as expected.

The interesting part comes next. After training, we throw away the head, keep the encoder, and use its 128 outputs as a new representation of every image. Then we give that representation to exactly the same linear probe as before.

The ranking is now clear:

The learned representation uses about a sixth as many numbers as the raw pixels, yet the same simple classifier does better with it. The network has found features that make the classes easier to separate with a straight line, which is exactly what a good representation should do.

Be precise about why the learned features won, because this is easy to overstate. The encoder had something the other two did not: it was trained on 40,000 additional labelled images. Representation learning does not create information from nothing. It moves the work of finding good features from a person to the training process, and that process needs data.

It is tempting to say that deep learning makes feature engineering obsolete. It does not. Look back at what we still decided in this section:

• We converted pixel values from 0–255 to 0–1 by dividing by 255.
• We chose a network with one hidden layer of 128 units and a ReLU activation.
• We decided how to split the data so the comparison was fair.

In later notes, these human decisions become even more important: CNNs build in the assumption that nearby pixels belong together, and tokenisers decide how text is chopped into pieces before any learning happens. The accurate statement is that deep learning shifts much of the representation-building into the model, while people still design the conditions under which it is learned.

Common mistake: concluding that learned features are "better" in general. They were better here, on this task, given 40,000 extra labelled examples. With very little data, or when an expert knows exactly what matters (for example, a physics formula), carefully designed features can still win. You saw a similar lesson in the gradient-boosting vs. neural-network comparison in Module 4.

We now know that a learned representation can beat raw pixels. The natural next question is: what did the network actually learn? Opening the box also leads us to one of the most important ideas in deep learning, the idea of a hierarchy of features.

Each of the 128 hidden units in our encoder computes a weighted sum of all 784 pixels and then applies ReLU (which outputs the sum if it is positive and 0 otherwise). So a unit "switches on", meaning it outputs a value above zero, when an image matches the pattern in its weights.

We can investigate each unit in two ways: look at its weights directly, and find the images that switch it on most strongly. The cell below records every unit's output for every validation image and summarises what each unit responds to.

A few things stand out.

First, 14 of the 128 units never switch on for any validation image. These units are inactive on this validation sample: their pre-activation is non-positive for every example we checked, so ReLU outputs 0. This alone does not prove they are dead for every possible input. This is a known side effect of ReLU, and it means our encoder is really using 114 units, not 128.

Second, many units have become detectors for particular kinds of clothing, even though nobody told them to. Unit 125 is the clearest: 98% of the 50 images that switch it on most are bags. Across all 114 live units, the typical (median) unit has 78% of its top-50 images from a single class, and 36 units reach at least 90%.

Third, the eight units we printed are the eight with the highest average output, and they are less specialised than typical (46% to 82%). That makes sense: units that switch on strongly for many images tend to respond to anything large and bright, such as tops, pullovers and coats, rather than to one class.

The figure below shows these eight units. The top row shows each unit's 784 weights arranged back into a 28×28 grid: red pixels push the unit on, blue pixels push it off. The bottom row is the average of the 50 images that switch that unit on most.

The weight maps are noisy, but the averaged images make the pattern clear: each unit has become tuned to a whole-garment outline, such as a long-sleeved shape, a T-shirt shape or a bag with a handle.

Our encoder has only one layer, so each unit has to go straight from individual pixels to something like "bag-shaped". That is a big leap, and it is why the weight maps look like blurry whole-object templates. A template like that is fragile: shift the bag a few pixels to the left and the template no longer lines up.

Deeper networks avoid this by building understanding in stages, each stage reusing the features from the stage before:

• Early layers detect simple, local patterns: edges (places where brightness changes sharply) and small spots.
• Middle layers combine edges into textures and parts: a curve, a corner, a strap, a sole.
• Later layers combine parts into whole objects: "boot" is a sole, plus a tall shaft, plus a heel.

This layered structure is called a hierarchical representation. It is powerful because simple features can be reused: the same edge detector helps recognise a boot, a bag and a dress, so the network does not need to learn a separate template for every object in every position. Language works the same way: characters combine into words, words into phrases, phrases into meaning.

Figure 5.1.3. A conceptual hierarchy from local edges to larger combinations. These are explanatory illustrations, not measured filters from the notebook experiment.

You will see this hierarchy directly in Lesson 5.2, where a convolutional network learns edge detectors in its first layer without being told to.

Key terms

• Hidden unit (neuron): one output of a hidden layer; a weighted sum of its inputs passed through an activation function.
• Dead unit: a ReLU unit whose output is 0 for every input, so it never contributes or learns.
• Hierarchical representation: a representation built in stages, where each layer combines the simpler features of the layer before it.

So far we have compared representations when the downstream classifier had plenty of labelled data: 10,000 examples. In real projects, labelled data is often the scarcest resource. Labels cost money, time and expert attention. So the most practical question about a representation is: how much labelled data does it save you?

To find out, we repeat the linear-probe experiment while giving the classifier fewer and fewer labelled examples: 20, 100, 500, 2,000 and 10,000. With very small samples the result depends heavily on which images happen to be chosen, so for every size below 10,000 we draw 5 different random samples and average the results. We also record the gap between the best and worst of the 5 draws, to see how noisy each estimate is.

Read the table row by row, then column by column.

• With only 20 labelled examples (about two per class), the learned representation reaches 0.562, against 0.414 for raw pixels and 0.404 for handcrafted features. But look at the spread row: the 5 draws differ by up to 0.161 to 0.199. With so few examples, a single lucky or unlucky draw moves the result a lot, which is exactly why we averaged.
• With 100 labelled examples, the learned representation reaches 0.794. Raw pixels need 2,000 labelled examples to reach the same accuracy (0.794). In other words, on this task the learned representation does the job with about one twentieth of the downstream labels, after the encoder used its separate 40,000 labelled examples.
• With 10,000 examples, the handcrafted features level off at 0.694. Adding more labels hardly helps, because seven numbers simply cannot describe the differences between a shirt and a pullover. The representation, not the amount of data, has become the limit.

The plot shows the same results. The grey dashed line is chance level.

This is the practical heart of representation learning. A good representation:

1. reaches higher accuracy with the same simple classifier;
2. needs far fewer labels to get there;
3. can be reused: we trained the encoder once, and could now attach many different heads to it for different tasks.

The third point is the foundation of modern AI engineering. In Module 7 you will use encoders that were trained on billions of examples, and attach your own small head, which is the same experiment as this one at a vastly larger scale.

Common mistake: trusting a single small-sample result. With 20 labelled examples, the accuracy of one draw varied by close to 0.2 depending on which images were picked. Whenever your training set is small, repeat the experiment with different samples and report the average and the spread.

Let's reassemble the whole picture.

Raw data such as pixels and characters is plentiful but individually meaningless. To make predictions from it, a model needs a representation: a compact description that captures what matters for the task. There are three ways to get one, and we measured all three on the same 10,000 validation images using the same linear probe:

A network learns its representation by being split into an encoder and a head that are trained together. The encoder discovers whatever features make the head's job easiest. Inside, its units became detectors: one responded almost only to bags. A deeper network would build these detectors in a hierarchy, from edges to parts to objects, so that simple features can be reused.

The pay-off is data efficiency and reuse: the learned representation matched, with 100 labels, what raw pixels needed 2,000 labels to achieve. The cost is that learning the representation needed data of its own, plus human decisions about architecture, preprocessing and data splits. That is why we say deep learning shifts feature engineering rather than eliminating it.

Figure 5.1.4. The reference run compares representations with the same kind of linear probe and 10,000 probe labels. The learned encoder had prior labelled training, so these results do not isolate representation choice at an equal total data budget.