How Neural Networks Learn from Images

Think about how you would describe the way you recognise a sneaker. You don't examine each point of the photo separately. You notice edges where the shoe meets the background, the curve of the sole, the stripes on the side, the laces. Those local details are found wherever they appear in the picture, and then combined into the overall shape that says "sneaker".

A CNN is built to work in the same spirit. It has three big ideas:

1. Look locally. Instead of connecting every pixel to every unit, a CNN slides a small window, typically 3×3 pixels, across the image and asks the same question at every position: "Is my pattern here?" The small window of weights is called a filter (or kernel).
2. Reuse the same detector everywhere. The filter's weights are the same at every position. A vertical-edge detector that works in the top-left corner also works in the bottom-right. This is called weight sharing.
3. Build up in stages. The output of one layer of filters, a set of feature maps showing where each pattern was found, becomes the input to the next layer. Early layers find simple patterns such as edges; later layers combine them into parts and shapes. Every so often, a pooling layer shrinks the maps so that later layers see a wider area.

At the end, a small fully connected head, exactly the kind of head you met in Lesson 5.1, turns the final feature maps into a prediction.

Figure 5.2.1. The exact lesson architecture: 1×28×28 → 16×28×28 → 16×14×14 → 32×14×14 → 32×7×7 → 1,568 features → a classifier. Batch dimension is omitted; feature-map appearance is schematic.

Keep this picture in mind. Sections 2 and 3 explain the input and the problem CNNs solve; Sections 4 to 7 examine each part; Sections 8 and 9 put the whole thing to work.

Before any network can process an image, the image has to be stored as a tensor: a multi-dimensional array of numbers, which you have used throughout the course. For images, the shape of that tensor follows a convention that you need to know well, because most bugs in vision code are shape bugs.

Let's load Fashion-MNIST again. This time we also download its separate test set: 10,000 images that we will touch only once, at the very end of the note.

The raw arrays have shape (60000, 28, 28) and (10000, 28, 28): number of images, height, width. PyTorch's convolution layers expect one more dimension, the number of channels. A channel is one "layer" of values per pixel. A greyscale image has 1 channel (brightness). A colour image has 3 channels (red, green and blue), so every pixel is described by three numbers.

PyTorch's standard layout for a batch of images is therefore:

The cell below adds the channel dimension with unsqueeze(1), scales the values from 0–255 to 0–1, and splits the 60,000 training images into 50,000 for training and 10,000 for validation.

A single image is now [1, 28, 28]: 1 channel, 28 rows, 28 columns. The training set is [50000, 1, 28, 28]. Values run from 0.0 to 1.0, stored as 32-bit floats, which is what PyTorch layers expect.

During training we feed the network batches (small groups of images processed together), so a batch of 64 images has shape [64, 1, 28, 28]. A colour image of 32×32 pixels would be [3, 32, 32].

Notice the last line. PyTorch puts channels first (C, H, W), but most image libraries, including matplotlib, put channels last (H, W, C). permute(1, 2, 0) reorders the dimensions, turning [3, 32, 32] into [32, 32, 3].

Common mistake: passing a [3, H, W] tensor straight to plt.imshow, or passing an [H, W, 3] image straight into a PyTorch convolution. The first gives an error or a garbled picture; the second makes PyTorch treat the image's height as if it were the number of channels. Whenever an image looks wrong or a layer complains about channels, print .shape first.

Key terms

• Channel: one value per pixel of a particular kind, such as brightness, or the red, green or blue component. Greyscale images have 1 channel; colour images have 3.
• Channels-first / channels-last: the order of dimensions. PyTorch uses (N, C, H, W); matplotlib and many image libraries use (H, W, C).
• Batch: a group of examples processed together; it adds a leading dimension N.

The MLP from Lesson 5.1 begins with nn.Flatten(), which turns each [1, 28, 28] image into a list of 784 numbers. That seems harmless. Let's see what it does to neighbouring pixels.

The pixel at row 10, column 10 lands at position 290. Its right-hand neighbour is at 291, but its equally close neighbour below is at 318.

Flattening removes the explicit 2D layout, not the underlying pixel information. Every value remains in a fixed order. If we know the original dimensions, we can reshape the vector and reconstruct the image exactly.

The disadvantage appears when this vector enters a dense layer. That layer gives each input position its own weights. It does not explicitly treat nearby pixels as neighbours or reuse a local detector across positions. It can learn spatial patterns, but must discover useful relationships from data, often with more parameters. A CNN builds locality and weight sharing into its architecture.

We can investigate this difference by applying one fixed random permutation of the pixel positions to every image. The experiment changes which pixels appear adjacent while preserving all values and their consistent identities: physical pixel positions still map to fixed vector indices.

The shuffled images are unrecognisable to us. But nothing has been lost from the MLP's point of view. Every image still contains exactly the same 784 values, just in a consistently different order, and the dense architecture has no built-in preference for the original image neighbourhoods.

To test this, we need a model and a training loop. The MLP below is the same shape as in Lesson 5.1. We also write two helper functions that we will reuse throughout this lesson:

• evaluate returns the accuracy and the predictions. It switches the model to evaluation mode with model.eval() and turns off gradient tracking, then switches back.
• train is the familiar training loop, with an optional list snapshots that stores a copy of the weights after every epoch, so that we can later pick the best epoch.

Now we train two identical MLPs, with the same starting weights, for 2 epochs each: one on the original images, one on the shuffled images.

The two accuracies are 0.8528 and 0.8527, essentially identical. But are they really the same, or is a small difference hiding behind the rounding? With a validation set of 10,000 images, one image is worth 0.0001 of accuracy, so the two models differ by a single image overall.

A more careful check is the sign test. We ignore the images that both models get right or both get wrong (they tell us nothing about which model is better) and count the disagreements. If the two models were truly equally good, each disagreement would be a coin flip.

The original-pixel MLP was correct on 90 examples where the shuffled one was wrong; the reverse happened on 89. The paired binomial test returns p = 1: these observations provide no evidence of an accuracy difference under its null hypothesis. This does not prove exact equivalence.

A dense layer can accommodate a fixed pixel permutation by learning a correspondingly permuted set of input weights. The experiment illustrates its lack of a built-in spatial preference; it does not show that the model cannot learn spatial relationships. In Section 8 we compare this with a CNN's explicit local connections.

Key terms

• Spatial structure: the arrangement of pixels in two dimensions.
• Permutation: a rearrangement of positions, applied consistently here.
• Paired sign test: compare the cases on which exactly one classifier is correct, using a binomial test with probability 0.5 under equal accuracy. It concerns these fitted models on these evaluation examples, not variation across training seeds.

Visual walkthrough: Follow a grid into a vector.

Watch the same values move in row order. Reshaping with the original dimensions recovers the image grid.

These supplied numbers illustrate the calculation. The web app adds a paused, step-by-step animation of the same example.

We have seen the problem: a dense MLP does not build in local spatial connections. Here is the full CNN we will use in this lesson, before we examine its pieces. Read the comments on the right: they show the shape of the data after each layer, for a batch of N images.

The two networks are almost the same size: 101,770 parameters for the MLP and 105,866 for the CNN. So any difference in how they behave is about how the parameters are arranged, not how many there are.

The CNN is organised exactly as the encoder–head split from Lesson 5.1:

• features (the encoder) contains two rounds of convolution → ReLU → pooling. Throughout this part, the data stays in the 2D shape (channels, height, width), so spatial structure is preserved.
• head flattens the final feature maps and classifies them with two fully connected layers. By this point, the encoder has already turned "where is everything" into "which patterns are present", so the head combines learned local features; flattening itself still preserves every feature-map value.

The best way to understand a network's shapes is to watch data flow through it. The cell below passes a dummy batch of 8 images through each layer and prints the shape after every step.

Follow the numbers down the list:

• Conv2d (first): 1 channel in, 16 out. The image stays 28×28, but we now have 16 feature maps per image, one for each filter.
• MaxPool2d: height and width halve to 14×14; the number of channels does not change.
• Conv2d (second): 16 channels in, 32 out, still 14×14.
• MaxPool2d: down to 7×7.
• Flatten: 32 × 7 × 7 = 1,568 numbers per image.
• Linear layers: 1,568 → 64 → 10 class scores.

Notice the trade the encoder makes as data flows through it: the maps get smaller (28 → 14 → 7) while the number of channels gets larger (1 → 16 → 32). The network gives up fine positional detail in exchange for a richer description of what is present.

Every piece in this list is examined in the next three sections: the filters and feature maps produced by Conv2d (Section 5), the size changes caused by padding and pooling (Section 6), and why stacking layers builds hierarchical features (Section 7).

The central operation of a CNN is convolution. The idea is simple: take a small grid of weights, the filter, place it over a patch of the image, multiply each weight by the pixel underneath it, and add everything up. That sum is one output value. Then slide the filter one step and repeat, until you have covered the whole image. The grid of outputs is a feature map.

Let's do this on a tiny 5×5 image, so that every number is visible. The image is dark (0) in its two left columns and bright (1) in its three right columns, so it contains one vertical edge. Our filter has −1 in its left column, 0 in the middle and +1 on the right. In words, the filter asks: "Is it brighter on my right than on my left?"

Take the top-left output value. The filter covers rows 0–2 and columns 0–2 of the input. The left column of that patch is 0, 0, 0 (multiplied by −1 gives 0); the middle column is 0, 0, 0 (multiplied by 0 gives 0); the right column is 1, 1, 1 (multiplied by +1 gives 3). The total is 3. The filter has found an edge.

Now take the top-right output value. The patch covers columns 2–4, which are all bright. The left column gives −3, the right column gives +3, and they cancel to 0. No edge there, so no response. Our hand calculation and PyTorch's F.conv2d agree exactly.

Two facts about this output are worth noticing:

• The 5×5 input produced a 3×3 output. A 3×3 filter only fits in 3 positions across a 5-wide image. We'll return to this in Section 6.
• The filter's weights add up to zero. So on a perfectly uniform patch, whether all dark or all bright, it outputs 0. It responds only to change. That is what makes it an edge detector.

A note on the name: strictly, what deep-learning libraries compute is called cross-correlation; mathematical convolution also flips the filter first. Because the filter's weights are learned anyway, the flip makes no practical difference, and everyone in deep learning simply calls it convolution.

Now apply two classic hand-designed edge filters to the ankle boot. The first is a stronger version of the vertical-edge filter above; the second is the same filter turned on its side, which detects horizontal edges. Notice that we stack the two filters into one tensor of shape [2, 1, 3, 3] (2 filters, each looking at 1 input channel, each 3×3), and F.conv2d produces one feature map per filter.

Each feature map is a picture of where the filter's pattern occurs. In the vertical-edge map, the left side of the boot is red (dark on the left, bright on the right) and its right side is blue (bright on the left, dark on the right). In the horizontal-edge map, the top edges and the sole light up instead. One filter, slid across the whole image, has produced a full map of one kind of feature.

In a CNN, nobody designs these filters by hand. The first Conv2d layer of our network has 16 filters whose weights start random and are learned by backpropagation, just like the weights of any other layer. The network discovers which patterns are worth detecting. We will look at what it learns in Section 9.

Because each filter is reused at every position, a convolution layer needs remarkably few parameters. Compare our first convolution layer with a fully connected layer that produces the same number of outputs:

Our first convolution layer has 160 parameters: 16 filters × 9 weights, plus 16 biases. It produces 12,544 output values (16 maps of 28×28). A fully connected layer mapping the same 784 inputs to the same 12,544 outputs would need 9,847,040 parameters, 61,544 times as many.

That saving comes from two assumptions built into convolution:

• Locality: each output depends only on a small neighbourhood of the input, because nearby pixels matter most.
• Weight sharing: the same pattern is worth detecting anywhere in the image.

Assumptions like these are called an inductive bias: knowledge about the problem that is built into the architecture rather than learned from data. When the assumptions are true, as they are for images, the network learns faster from less data. When they are false, they can hurt. You will see both outcomes in Section 8.

Key terms

• Filter (kernel): a small grid of learnable weights that is slid across the input.
• Convolution: sliding a filter over the input and, at each position, computing the sum of weights × inputs.
• Feature map: the grid of outputs a filter produces; it shows where the filter's pattern appears.
• Weight sharing: using the same filter weights at every position.
• Inductive bias: an assumption about the problem built into a model's design.

Visual walkthrough: Slide one shared filter.

Step through nine exact dot products. Notice which values change and which filter weights stay fixed.

These supplied numbers illustrate the calculation. The web app adds a paused, step-by-step animation of the same example.

In Section 5.1 a 3×3 filter shrank a 5×5 image to 3×3. If every layer shrank the image like that, a deep network would quickly run out of image, and the pixels near the border would be used less than those in the middle. Three settings control what happens to the size of the data: padding, stride and pooling.

• Padding adds a border of zeros around the input before the filter is applied. With a 3×3 filter, adding one pixel of padding on every side (padding=1) keeps the output the same size as the input. That is why our CNN's convolutions keep 28×28 as 28×28.
• Stride is how far the filter moves at each step. With stride=1 it visits every position; with stride=2 it skips every other position, roughly halving the output size.

These combine into one formula for the output size along each dimension:

The brackets ⌊ ⌋ mean "round down to a whole number". Let's check the formula against PyTorch for several settings on a 28×28 input:

The formula and PyTorch agree in every case. Take the first line: (28 + 0 − 3) / 1 + 1 = 26. Without padding, a 3×3 filter loses one pixel on each side. Add padding=1 and the result is 28 again. With stride=2 as well, the output drops to 14.

Common mistake: getting the input size of the first nn.Linear after Flatten wrong. It must equal channels × height × width of the last feature maps (here 32 × 7 × 7). If you change any padding, stride or pooling setting, recompute it with the formula, or simply pass a dummy batch through the encoder and print the shape, as we did in Section 4.

Pooling shrinks a feature map by summarising small blocks of it. A 2×2 pool with stride 2 splits the map into non-overlapping 2×2 blocks and replaces each block by one number:

• Max pooling keeps the largest value: "was the pattern found anywhere in this block?"
• Average pooling keeps the mean value: "how strongly, on average, was it found here?"

Here is a 4×4 grid pooled both ways:

The top-left block contains 1, 3, 4 and 6. Max pooling keeps 6; average pooling gives (1 + 3 + 4 + 6) / 4 = 3.5. The bottom-right block contains 9, 5, 3 and 7, which gives 9 and 6.

Max pooling is the usual choice inside CNNs because of what it throws away. If an edge shifts by one pixel within a block, the maximum hardly changes. So the network becomes a little less sensitive to exactly where a pattern appears, which is usually what we want for recognising objects. Pooling also has no learnable parameters at all.

Key terms

• Padding: extra values (usually zeros) added around the input's border.
• Stride: the step size of the filter as it slides.
• Pooling: summarising small blocks of a feature map (by max or average) to make it smaller.

A single 3×3 filter can only see 3×3 pixels. That is enough for an edge, but not for a sole, let alone a whole boot. So how does a CNN ever recognise large shapes?

The answer is stacking. The second convolution layer does not look at pixels; it looks at the feature maps from the first layer. Each of its outputs covers a 3×3 block of those maps, and each of those values already summarised a 3×3 block of pixels. Pooling in between makes the reach grow faster still. The region of the original image that can influence one output value is called its receptive field.

We can measure the receptive field directly. We pass an image through the encoder, pick the output value at the centre of the final map, and use backpropagation to ask: which input pixels have any effect on it? Pixels whose gradient is not zero are inside the receptive field. (For this measurement we use average pooling instead of max pooling, because max pooling passes the gradient only to the single winning pixel in each block, which would hide part of the window. The receptive field is the same size either way.) We also compute the same quantity with a simple formula: each layer adds (kernel size − 1) × (the combined stride of all earlier layers).

Measured and calculated agree: the receptive field grows from 3×3 after the first convolution, to 4×4 after the first pool, to 8×8 after the second convolution, and to 10×10 after the second pool. In a 28×28 image, a 10×10 window is large enough to cover a whole heel, a strap or the toe of a shoe.

This growth is the mechanism behind hierarchical feature learning:

• First layer, small receptive field: can only detect simple local patterns, such as edges, spots and changes of brightness.
• Second layer, larger receptive field: combines several edge maps, so it can detect corners, curves and textures: the parts of an object.
• Deeper layers in larger networks: keep combining, so later units can respond to whole objects. The networks that classify ordinary photos have dozens of layers and receptive fields covering most of the image.

Figure 5.2.2. For this model, receptive-field widths grow from 3 after conv1 to 4 after pool1, 8 after conv2, and 10 after pool2. This describes theoretical input coverage, not a measured activation map.

Key terms

• Receptive field: the region of the input image that can affect one particular output value.
• Hierarchical features: features organised in levels, with each level combining the simpler features of the level below.

We have now seen every part of the CNN. Let's train the whole thing and test whether its built-in assumptions pay off.

We train for 5 epochs with the Adam optimizer and a learning rate of 0.001, using the train function from Section 3. After each epoch we record the training loss and the validation accuracy, and keep a snapshot of the weights. At the end, we pick the epoch with the best validation accuracy. We never use the test set to make this choice; it stays untouched until the final evaluation.

The training loss falls every epoch, from 0.678 to 0.283, and validation accuracy rises every epoch, from 0.8392 to 0.8929. The best epoch on validation is therefore the last one, epoch 5, and those are the weights we keep. The plot shows both curves.

Validation accuracy is still rising at epoch 5, so we have not trained to the point of overfitting; a few more epochs would probably help a little. We stop here to keep the notebook quick to run.

In Section 3, the MLP did just as well on shuffled pixels as on real ones. If the CNN truly relies on neighbourhoods, it should lose accuracy when the pixels are shuffled, because its filters will be looking at "neighbours" that are not really neighbours.

To keep the comparison fair, we train a fresh CNN on shuffled images for 2 epochs, with the same starting weights, and compare it with our main CNN's snapshot after the same 2 epochs.

This time the difference is large. After 2 epochs, the CNN reaches 0.8643 on real images but only 0.8341 on shuffled ones. The paired sign test gives strong evidence of a difference for these fitted models: the real-image CNN was right on 557 images that the shuffled one got wrong, against only 255 the other way round, with a p-value of 1.14e-26. Such an imbalance is very unlikely under the equal-accuracy null; repeat training seeds to assess training variability too.

Notice also that the CNN on shuffled pixels (0.8341) did worse than the MLP on shuffled pixels (0.8527). When the CNN's built-in assumption, that neighbouring pixels belong together, is false, it becomes a handicap. This is the inductive bias from Section 5.3 in action. It helps when it matches reality and hurts when it does not.

Now, once and only once, we evaluate the chosen model on the 10,000 test images.

The test accuracy is 0.8903, very close to the final validation accuracy of 0.8929. That closeness is reassuring: it suggests we did not accidentally tune our choices to the validation set. Out of 10,000 test images, 1,097 are misclassified.

The per-class accuracies are far from equal. Bags (0.970), sneakers (0.964), trousers and sandals (0.959 each) are easy. Shirts (0.694) and pullovers (0.767) are hard. To see why, let's find the most common mistakes.

The four most common mistakes are all between upper-body garments: 121 pullovers predicted as coats, 119 shirts predicted as T-shirts, and 105 each of T-shirts predicted as shirts and shirts predicted as coats. At 28×28 greyscale, these items have nearly the same outline, and the details that separate them (buttons, collars, fabric texture) occupy only a few pixels. These are mistakes that a person squinting at the same tiny images would also make.

Common mistake: reporting only the overall accuracy. An accuracy of 0.8903 hides the fact that the model is right on only 0.694 of shirts. Always look at per-class results and the most common confusions; they tell you where the model needs to improve and whether its errors are acceptable for your application.

In Section 5, we used filters designed by hand. Our trained CNN designed its own. Let's open it up and look.

Each of the 16 filters in the first convolution layer is a 3×3 grid of weights. Looking at 3×3 pictures and deciding which ones "look like edges" is very subjective, so we first measure something objective. Recall from Section 5.1 that an edge detector's weights add up to roughly zero, so that it ignores uniform brightness and responds only to change. We define a filter's balance as:

A balance near 0 means the positive and negative weights cancel, so the filter responds to change (edge-like). A balance near 1 means the weights all have the same sign, so the filter responds to overall brightness or darkness. To know whether training changed anything, we compare the learned filters with the same filters at their random starting values.

Training has pushed many filters towards balanced, contrast-sensitive weights. Balance alone does not prove that a filter detects a particular edge orientation. The median balance fell from 0.30 at the random start to 0.06 after training, and 9 of the 16 learned filters have a balance below 0.2. A few filters went the other way: filter 15 has a balance of 1.00 and filter 3 has 0.93, meaning their weights all share one sign. These respond to brightness or darkness itself rather than to edges. Here are all 16 filters, labelled with their balance:

The low-balance filters contain both strong red and strong blue weights, often on opposite sides or corners of the grid, at various angles. That is the signature of an edge or contrast detector. They are rougher and less symmetric than the hand-designed filters of Section 5.2, which is typical of learned filters, but the network has arrived at the same kind of tool purely from the training signal. Filters 3 and 15, by contrast, are almost entirely one colour.

Filters are easier to understand by what they produce. Let's pass the ankle boot through the trained network and look at the feature maps after each convolution layer (after ReLU, which sets negative values to 0).

The first-layer maps (top row, 28×28) are easy to read. Some light up along particular edges of the boot, such as the sole or the front. Others light up across the whole boot or, like map 3, across the background, which matches filter 3's same-sign weights. Each map is a different "view" of the same image.

The second-layer maps (bottom row, 14×14) are coarser and harder to interpret, which is normal: each value now combines 16 first-layer maps across an 8×8 window of the original image. They respond to combinations, such as a particular edge in a particular region.

The numbers back this up. In the first layer, 0.38 of the values are zero; in the second layer, 0.50 are. On this one image, the deeper features are more sparse: each one switches on for fewer locations, because it responds to a more specific combination of patterns.

Common mistake: over-reading feature-map pictures. A map that "looks like" it detects heels may be responding to something else entirely, such as overall brightness in that region. Pictures are a starting point for questions, not proof; back them up with measurements, as we did with the balance score and the zero fractions.

Let's reassemble the picture from Section 1, now that you have seen every part.

An image enters as a tensor of shape (N, C, H, W). An MLP flattens it immediately, preserving values but removing the explicit grid interface. Its lack of built-in locality is illustrated here: an MLP trained on shuffled pixels scored 0.8527, the same as on real images (0.8528, sign-test p = 1).

A CNN keeps the 2D layout. Its convolution layers slide small learned filters across the image, with the same weights at every position, producing feature maps that show where each pattern occurs. This weight sharing is enormously efficient: our first layer needed 160 parameters, against 9,847,040 for a fully connected layer with the same outputs. Padding controls whether the maps shrink, and pooling halves them, so that each layer's receptive field grows: 3, 4, 8 and then 10 pixels wide in our network. That growth lets later layers combine edges into parts. Finally, a small head flattens the last maps and classifies.

Trained on Fashion-MNIST, this CNN reached 0.8929 on validation and 0.8903 on the held-out test set. Its errors are concentrated in visually similar upper-body garments. Many first-layer filters developed balanced, contrast-sensitive weights (median balance 0.06, down from 0.30 at the start). And its built-in assumption that neighbours matter is real: the same CNN on shuffled pixels lost accuracy, falling from 0.8643 to 0.8341 after 2 epochs, with p = 1.14e-26.

Figure 5.2.3. Convolution applies local shared weights; feature maps retain spatial organisation, pooling reduces resolution, and the head combines features into class scores. The feature maps shown are conceptual.