Training Dynamics & Optimization
Why does a network learn well, badly, or not at all?
You will learn
Hands-on: Diagnosing learning curves and fixing deliberately broken experiments.
Lessons
-
Lesson 3.1: Optimizers and the Learning Rate
optimizer.step(), opened up. Plain SGD is literally p -= lr * p.grad; momentum adds a velocity that cancels zig-zags; Adam keeps two running averages per weight and normalises every step, which is why its good learning rate (about 1e-3) looks nothing like...
Not started Start -
Lesson 3.2: Overfitting, and What to Do About It
A network that is too big for its data memorises it. You make that happen on purpose (German Credit: 500 training rows, 48,897 parameters, training AUC 1.0 and a validation loss 3.4 times worse than its best), learn why validation loss and AUC tell...
Not started Start -
Lesson 3.3: When Gradients Misbehave
A falling loss does not prove every layer is learning. You turn Module 2's single gradient norm into a per-layer probe and use it to catch five failures: a broken initialisation (all zeros, or far too large), vanishing gradients through a deep sigmoid...
Not started Start -
Lesson 3.4: The Diagnosis Clinic
Lessons 3.1–3.3 gave you three kinds of evidence; this clinic turns them into a procedure: observe, compare with a reference, hypothesise, test, intervene, verify. You learn six curve shapes, a config-driven harness that records losses, metrics, prediction...
Not started Start
Module challenge
Module 3 challenge: why a network learns well, badly, or not at all
8 scenario questions that mix ideas from all 4 lessons. One question at a time, an explanation after each, and your score at the end. Pass mark 75%, so 6 of 8.
Best after you have practised all 4 lessons.
Two optimizers are compared at one shared learning rate of 1e-3: Adam reaches AUC 0.79, SGD 0.65. A report concludes "Adam is better". What is missing?
An optimizer cannot be judged apart from its hyperparameters (Lesson 3.1 §5). Give every method a fair chance.
Switching from Adam to AdamW with weight_decay=0 changed nothing. A teammate says AdamW is broken. What do you explain?
Lesson 3.1 §4.
A run's loss sits flat at 0.36 (on an 11.7%-positive dataset) and validation AUC is 0.50. The learning rate is 0.5. What happened, and what do you do?
A flat loss near the base-rate loss with AUC at chance is collapse, not convergence (Lessons 3.1 §6, 3.3 §5, 3.4 Case 1).
On 500 training rows a (256, 128) MLP reaches training AUC 1.0 while validation loss triples after epoch 5. What is the cheapest first intervention to try?
Capacity reduction and early stopping are cheap and, in Lesson 3.2, early stopping removed most differences between regularisers.
Config A scores validation AUC 0.824, config B 0.818. The same config varies by SD 0.015 across splits. A teammate wants to ship A because it is higher. Your response?
Measure the noise floor before tuning (Lesson 3.2 §9); pair the comparison (Lesson 4.4).
A 16-layer plain sigmoid MLP barely reduces its loss. The per-layer probe shows first-layer gradients around 1e-10 and last-layer gradients around 1. Which change targets the cause?
Vanishing gradients: sigmoid slopes (≤ 0.25) multiply through depth. Structural fixes address gradient flow (Lesson 3.3 §3, §6, §7).
Your model uses dropout. Validation AUC printed twice for the same checkpoint: 0.751, then 0.709. What is the fix?
Deterministic evaluation must give identical numbers; dropout is still active in training mode (Lesson 3.2 §7, 3.4 case).
A new feature raises validation AUC from 0.78 to 0.93; every training diagnostic looks healthy. Which protocol check should now fire?
Lesson 3.4 Case 6: duration was only known after the call. Good numbers deserve scrutiny too.