The Learning Mechanism, Traced By Hand

Strip away the vocabulary and a neuron does two things in sequence.

First it computes a weighted sum of its inputs and adds a bias:

You have seen this before. It is a linear model. If you stopped here, a neuron would be exactly the logistic regression that failed in Lesson 1.1.

Second, it passes that sum through an activation function:

That second step is the entire difference, and Section 4 shows why removing it destroys the network.

The vocabulary, precisely:

The last distinction matters more than it looks. Parameters are found by gradient descent. Hyperparameters are found by you, by experiment, and getting them wrong is the most common reason a training run fails. That is Module 3.

Here is the network we will use for the rest of this lesson. It is deliberately the smallest thing that is still genuinely a neural network.

Two inputs, two hidden neurons, one output. Let us count what it has to learn.

Every layer of shape (inputs → outputs) contributes inputs × outputs weights plus outputs biases.

• Input to hidden: \(2 \times 2 = 4\) weights, plus 2 biases = 6
• Hidden to output: \(2 \times 1 = 2\) weights, plus 1 bias = 3
• Total: 9 parameters.

Nine numbers. That is the entire model. Everything in this lesson is about how those nine numbers change.

For contrast, the spiral network in Lesson 1.1 had 4,417, and a modern language model has somewhere between a billion and a trillion. The arithmetic below is identical in all three cases. Only the count changes.

Now we fix actual numbers. Four training examples, two features each, binary labels. The initial weights below are not random — they are chosen so the arithmetic is easy to follow.

Do this by hand before running the next cell. Take the first example, \(x = [0.5, -1.0]\), and compute the pre-activation of hidden neuron 1:

Now do hidden neuron 2 yourself. Its weights are the second column of W1: \(-0.20\) and \(0.80\), with bias \(-0.10\).

You should get \(z_2 = 0.5 \times (-0.20) + (-1.0) \times 0.80 + (-0.15) = -0.10 - 0.80 - 0.15 = -1.05\).

Next, apply ReLU, which is simply \(f(z) = \max(0, z)\) — pass positives through unchanged, clamp negatives to zero. So \(a_1 = 0.05\) and \(a_2 = 0.00\).

Notice what just happened to hidden neuron 2. Its pre-activation was negative, so ReLU clamped it to exactly zero. For this example it contributes nothing at all to the prediction — and, as you will see in Section 8, no gradient will flow back to it through this example either. That is not a malfunction. It is ReLU behaving as designed, and it is why deep networks are sparser than they look.

Check the first row against your hand calculation. Z1[0] should read [0.05, -1.05], and A1[0] should read [0.05, 0.00].

The final column is the model's opinion. Example 1 should be class 1 and the model says about 0.56 — right side of the fence, but barely. Example 4 should be class 0 and the model puts it above 0.5, which is simply wrong. Three of four correct, by something close to luck. This network knows nothing yet, which is precisely what we are about to fix.

Two details worth naming, because both will bite you later:

Why sigmoid at the output but ReLU in the hidden layer? Different jobs. The output must be a probability, so it needs squashing into \((0, 1)\). Hidden units have no such requirement — they just need to be non-linear, and ReLU is faster and better behaved during training.

Why do all four examples go through at once? Because X @ W1 is one matrix multiplication instead of four separate ones. That is the entire reason GPUs matter to deep learning: this shape of computation parallelises almost perfectly. Lesson 1.3 makes this concrete.

It is tempting to view the activation function as a detail — a tweak that helps a bit. It is not. Without it, depth is worthless, and the reason is a two-line piece of algebra.

Suppose we drop ReLU, so each layer is purely linear. Then:

Look at the right-hand side. \(W_1 W_2\) is just some matrix. \(b_1 W_2 + b_2\) is just some vector. So the whole two-layer network collapses into \(XW' + b'\) — a single linear layer. Stack fifty such layers and it still collapses to one.

That is the punchline: a network with no activation function, however deep, is exactly as expressive as logistic regression. It would fail on the spirals for precisely the reason Lesson 1.1 failed. Let us verify it numerically rather than take it on faith.

Identical to floating-point precision. The depth bought nothing.

The third output is different — and it is different because ReLU zeroed some values, which is an operation no single matrix can reproduce. That small act of clipping is what makes the composition irreducible, and therefore what makes depth meaningful.

The blueprint for this course warns against turning it into a taxonomy of activation functions, so we will not tour them. What you need in 2026 is short: ReLU and its close relatives (GELU, SiLU) for hidden layers; sigmoid for a binary output; softmax for multi-class output. Modern Transformers overwhelmingly use GELU or SiLU internally, which are smoothed versions of the same idea. You will meet them again in Module 6.

The model has produced its predictions. Some are correct; some are wrong.

But for the model to improve, we need a way to measure how wrong it is.

More precisely, we need that wrongness expressed as a single number that changes smoothly as the model’s parameters change.

Both parts of that requirement matter.

A single number, because gradient descent needs one quantity to minimise. It needs a clear answer to the question: “How good or bad is the model right now?”

Smoothly, because gradient descent improves the model by making tiny adjustments to its parameters. We need to know how a tiny change in a weight affects that number. If the number changes smoothly, we can calculate a slope and use it to determine which direction to move.

This is why we do not optimise accuracy directly.

Imagine changing a weight by 0.0001. The model's accuracy will probably stay exactly the same. Change the weight a little more, and eventually one prediction may cross a decision boundary. Accuracy suddenly jumps.

So accuracy can look like:

flat → flat → flat → sudden jump

There is no useful slope telling gradient descent which direction to move.

Instead, we use a loss function.

Accuracy tells us how many predictions were correct. Loss tells us how well the model's predictions matched the true answers, while providing a smooth quantity that gradient descent can optimise.

So remember:

Accuracy is something we use to evaluate and report performance. Loss is what we use to train the model.

For binary classification, one of the most common loss functions is binary cross-entropy (BCE).

At first, the formula can look intimidating. But we can understand it by looking at one prediction at a time.

If the true label is 1, the loss asks:

How much probability did the model give to class 1?

Predict 0.99 when the answer is 1, and the loss is very small.

Predict 0.01, and the loss is very large.

When the true label is 0, the same idea works in reverse.

Predict 0.01, and the loss is very small.

Predict 0.99, and the loss is very large.

The important idea is this:

The more confidently the model is wrong, the more heavily the loss punishes it.

An uncertain wrong prediction is bad.

A confident wrong prediction is much worse.

That gives gradient descent a useful, smoothly changing number to work with as it adjusts the model's parameters.

The per-example losses show where the model is hurting most. Example 4 is the worst offender — the model says 0.55 for something that should be 0.

We now have one number that tells us how wrong the model is: the loss.

The next question is the heart of deep learning:

If I change one parameter by a tiny amount, does the loss go up or down, and by how much?

That is what the gradient tells us.

There is nothing mysterious about it. The gradient of the loss with respect to a parameter, such as a weight \(w\), tells us the local slope of the loss as we change that parameter while keeping everything else fixed.

Think of the parameter as a knob.

We turn the knob slightly and ask:

Did the loss get better or worse? And how strongly did it change?

The gradient gives us two important pieces of information:

• The sign tells us the direction. A positive gradient means increasing the parameter makes the loss increase, so gradient descent moves the parameter in the opposite direction. A negative gradient means increasing the parameter decreases the loss, so gradient descent moves it upward.

• The magnitude tells us how steep the slope is. A large gradient means a small change in that parameter produces a relatively large change in the loss. A gradient close to zero means the loss barely changes when we nudge that parameter.

So, in simple terms:

The gradient tells us which way to move a parameter and how strongly that parameter should be adjusted.

Now we can put the idea into an equation.

The basic gradient descent update is:

Don't let the notation make this look more complicated than it is.

• \((w)\) is the parameter we are updating.
• \((L)\) is the loss.
• \((\frac{\partial L}{\partial w})\) is the gradient: how the loss changes when we change \(w\).
• \((\eta)\) is the learning rate: how big a step we take.

The most important part is the minus sign.

We subtract the gradient because the gradient points in the direction where the loss increases. Gradient descent wants to move in the opposite direction, toward a lower loss.

For example, if:

the gradient is positive, so we decrease \(w\).

If:

the gradient is negative, so subtracting it causes \(w\) to increase.

The learning rate controls how far we move.

A large learning rate takes bigger steps. A small learning rate takes smaller steps.

So the whole process can be reduced to one simple idea:

Look at the slope, move in the opposite direction, and control the size of the step with the learning rate.

And we don't actually need calculus to understand the basic idea.

We can estimate the gradient by brute force.

Take a weight \(w\). Add a tiny amount \((\epsilon)\) to it. Recalculate the loss. Compare the new loss with the original loss.

If the loss changed a lot, the slope is steep. If it barely changed, the slope is shallow. Whether the loss went up or down tells us the direction of the slope.

This is called a finite-difference estimate.

In practice, neural networks use calculus-based methods such as backpropagation to calculate gradients efficiently across millions or even billions of parameters. But finite differences remain extremely useful: engineers use them to check that their gradient calculations are correct.

So the big picture is:

Loss tells us how wrong we are. The gradient tells us how changing each parameter affects that wrongness. Gradient descent uses the gradient to decide which direction to move and the learning rate to decide how far.

Our brute-force experiment has now given us a gradient for every parameter.

In this small network, that means nine numbers: one gradient for each parameter. Together, they tell us how the loss responds to changing every part of the model.

You can think of the collection of gradients as a map of the local landscape:

For every parameter, which direction should we move, and how strongly does it affect the loss?

Now look at the two columns of dL/dW1.

The second column, which corresponds to the weights of hidden neuron 2, has noticeably smaller gradients than the first.

We can explain this by looking back at A1 from Section 3.

Hidden neuron 2 was activated on only one of the four examples. On the other three examples, ReLU output zero. That means neuron 2 contributed nothing to those three predictions, so changing its weights cannot affect the loss for those examples.

Hidden neuron 1, on the other hand, was active on two of the four examples, so its weights had more opportunities to influence the predictions and, therefore, the loss.

This gives us an important idea:

A neuron that contributed nothing to a prediction cannot receive blame for that prediction's error.

That is credit assignment beginning to appear directly in the numbers, even before we formally define the term.

So why not just use numerical gradients?

At this point, you might reasonably ask:

If we can calculate gradients this way, why do we need backpropagation?

The problem is the cost.

Our numerical method works by changing one parameter at a time, running the entire forward pass again, and measuring how much the loss changes.

For this tiny network, nine parameters require roughly nine additional forward passes.

That is manageable for a toy example.

But imagine a neural network with one billion parameters.

Using this method would require roughly one billion forward passes for a single training step.

That is computationally impractical.

Backpropagation solves this problem in a fundamentally more efficient way. Instead of separately probing every parameter, it uses the chain rule of calculus to calculate the gradients of all the parameters together, in roughly the computational cost of one forward pass plus one backward pass.

That efficiency is not merely a nice optimisation.

Backpropagation is one of the fundamental reasons we can train modern deep neural networks at all.

Our brute-force numerical method is therefore useful for something different: understanding what a gradient means and verifying that our backpropagation implementation is producing the right answers.

To get all nine gradients cheaply, we need to see the forward pass as what it really is: a chain of simple operations, each one recording what it did.

Every node is one elementary operation. Each knows two things: the inputs it received, and how to convert a gradient at its output into gradients at its inputs. That is all.

The rule linking them is the chain rule, and its plain-language version is the only version you need today:

If \(L\) depends on \(A\), and \(A\) depends on \(w\), then the sensitivity of \(L\) to \(w\) is the sensitivity of \(L\) to \(A\), multiplied by the sensitivity of \(A\) to \(w\).

Sensitivities multiply along the chain. Nothing more.

This is also why frameworks appear to "remember" your computation. When you run a forward pass in PyTorch, every operation quietly appends itself to a graph. Calling .backward() walks that graph in reverse, applying each node's local rule. It is bookkeeping, executed fast — and it is why a forward pass under torch.no_grad() uses less memory, a fact that becomes very practical in Module 4.

Here is the sentence to remember:

Backpropagation is credit assignment through the computational graph.

The loss is one number produced by all nine parameters working together. Backpropagation works backwards from that loss, efficiently determining how much each parameter contributed to the error.

We’ll do this one step at a time. You only need two derivative facts, and both are worth understanding rather than simply looking up.

Fact 1: Sigmoid + binary cross-entropy

For a sigmoid output with binary cross-entropy loss, the gradient at the output pre-activation is remarkably simple:

In plain English:

Prediction minus truth, averaged across the examples.

The sigmoid derivative and the logarithm terms from cross-entropy cancel neatly. This is not an accident. The sigmoid + binary cross-entropy pairing is widely used in part because this combination gives us this elegant gradient.

Fact 2: ReLU

ReLU behaves like an on/off switch during backpropagation.

Its derivative is 1 when the input is positive and 0 when the input is negative.

So when we move backwards through the network, an active neuron lets the gradient pass through; an inactive neuron blocks it.

And that gives us the basic mechanism:

Start with the loss → move backwards through each operation → pass the gradient according to each operation's derivative → eventually reach every parameter.

Now, let’s walk backwards.

They agree. You now have the same nine numbers by two completely independent routes: brute-force probing, and one backward walk through the graph.

Take a moment on step 3, because it is the heart of the idea. dZ2 @ W2.T distributes the output error back to the hidden neurons in proportion to the weights connecting them. A hidden neuron wired to the output with a large weight receives a large share of the blame; one wired with a small weight receives little. That is credit assignment made arithmetic.

And step 4 is where the sparsity you noticed earlier becomes formal. Multiplying by (Z1 > 0) cuts off every path through an inactive neuron. No activity, no responsibility, no update.

We know, for each of the nine parameters, whether increasing it raises or lowers the loss. So take a small step in the direction that lowers it:

The minus sign is the whole algorithm: gradients point uphill, and we want to go down. The symbol \(\eta\) is the learning rate — how big a step to take.

Before running the cell, predict the outcome. The current loss is 0.5845. After one step with \(\eta = 0.5\), will the loss go up or down?

It went down. Small, unglamorous, and entirely real.

That is one training step. Everything else in deep learning is this operation repeated — with more parameters, more data, and cleverer rules for choosing \(\eta\), but the same three moves: forward, backward, update.

One step helped a little. Now do it repeatedly. This cell is a complete training loop written in NumPy — and structurally it is the same loop you will write in PyTorch in Lesson 1.3, and the same loop that trains models with billions of parameters.

That curve is the single most-read diagnostic in deep learning, and you will spend most of Module 3 interpreting its variants. For now, note its characteristic shape: steep at first, where gradients are large and there is plenty to fix, then flattening as the model approaches a solution and the slopes shrink.

One caveat, stated now so it does not become a bad habit. This model is fitting four examples and we are measuring loss on the same four. That is training loss, not validation loss, and a model that memorises four points has demonstrated nothing about generalisation. With real data this would be malpractice. Lesson 1.3 introduces the split properly.

The learning rate is the hyperparameter you will get wrong most often, so it deserves its own picture. Run all three regimes side by side.

Three failure modes, three signatures you should be able to recognise on sight:

• Too small. The loss falls, but so slowly that you cannot tell whether the model is learning or stuck. This wastes far more engineering time than it should, because nothing looks broken.
• About right. Rapid early progress, then a smooth flattening.
• Too large. The step overshoots the valley and lands further up the opposite side. The loss oscillates, plateaus at a bad value, or diverges outright.

In 2026 nobody picks a single fixed learning rate and leaves it there. Optimisers such as Adam adapt the effective step size per parameter, and schedulers vary it over training — typically warming up and then decaying. You will use both in Module 3. But every one of those methods is a refinement of the trade-off in this picture, and understanding the picture is what lets you diagnose them when they misbehave.

Everything above was NumPy that you could audit line by line. Now we hand the identical problem to PyTorch — same data, same initial weights, same loss — and compare its gradients with the ones you derived.

Zero difference, to the last bit.

Sit with that for a moment, because it is the point of this entire note. loss.backward() is not a magic box. It is the five-step walk you performed in Section 8, executed by code that knows the local derivative rule for every operation. When you write it in Lesson 1.3, you will know precisely which arithmetic you delegated and why it is safe to do so.

This is also what makes you able to debug training. When a loss becomes NaN in Module 3, when gradients vanish in a deep network in Module 6, when a fine-tuning run refuses to move in Module 8 — the person who has traced the mechanism once knows where to look. The person who has only ever called .backward() is guessing.