The Diagnosis Clinic

Before diagnosing a run, establish a visual vocabulary.

The plot produced by the next code cell contains six representative training/validation loss patterns drawn from the behaviours explored in Lessons 3.1–3.3. They are not merely shapes to memorise: each shape is a symptom, and the rest of the diagnostics tell you which underlying cause produced it.

Keep this output visible while working through the cases.

When reading a training curve, ask three questions in order:

1. Is the loss moving at all?
2. Are training and validation behaving differently?
3. Does the metric and prediction behaviour agree with the story suggested by the loss?

The third question is essential. Loss is an optimisation quantity; it is not automatically a measure of whether the trained model is useful.

Three of these shapes are especially easy to misread. Commit the reasoning behind them to memory rather than memorising the letters.

D looks like success. A flat, stable loss usually means the optimiser has converged. But convergence only means that the optimisation has stopped making meaningful progress. A model can converge to a useful solution or to a trivial one. The distinguishing evidence is therefore the metric and the prediction distribution, not the fact that the loss is flat.

C looks like a hard problem. A slowly descending curve can tempt you to conclude that the data is weak or the model lacks capacity. But if the curve is still descending when the epoch budget ends, the experiment may simply be undertrained. You have measured how far the optimiser got, not the performance the model would reach if it had enough training.

F looks like a triumph. An unexpectedly excellent result is not automatically evidence that the model is excellent. It can be evidence that the experiment is broken — for example through validation leakage or target leakage. A result that is dramatically better than a sensible baseline should trigger an investigation into why before it triggers celebration.

Module 2, Lesson 2.4 ended by extracting the notebook into a project repository with a config-driven entry point:

python -m src.training.train --config configs/baseline.yaml

The important idea is that the experiment's settings live in a configuration rather than being scattered through the training code. The runner can therefore execute the same pipeline repeatedly while changing a controlled set of values.

That is exactly what we need here. Six diagnostic cases are only useful if the underlying pipeline stays the same. Otherwise, when a result changes, you cannot tell which change caused it.

This notebook extends the earlier harness in two ways:

1. it supports both datasets, selected through a configuration key; and
2. it records per-layer gradient norms from Lesson 3.3 alongside the loss and metric history.

A run therefore carries several kinds of evidence:

• loss history — how optimisation progressed;
• validation metrics — whether the model's behaviour is useful on held-out data;
• prediction summaries — whether the model is producing varied, informative outputs;
• gradient profiles — whether different parts of the network are receiving a usable learning signal;
• configuration — exactly what settings produced the run.

This is what makes the later diagnosis reproducible rather than anecdotal.

That is the reference run: the example of what normal behaviour looks like when the same pipeline is configured sensibly.

Every case below changes one key in the configuration unless the section explicitly says otherwise. Keep the reference numbers visible while you work. The reference has a final validation AUC around 0.74 and a best validation AUC around 0.82, on a dataset where majority-class accuracy is 0.70 and published results are reported around 0.78–0.80 AUC.

These values are not universal targets. They are a local reference for this exact dataset, split, preprocessing pipeline, model family, and evaluation setup.

That is what a baseline is for. Diagnosis is fundamentally comparative: you cannot call a run abnormal unless you know what normal looks like under the same experimental conditions.

A useful mental model is:

Baseline = your control condition.

If the baseline changes every time you diagnose a problem, you lose the ability to attribute differences to the suspected fault.

When a training run looks wrong, the tempting response is to change something plausible and run it again. That is search, not diagnosis.

Search can eventually find a working configuration, but it does not necessarily tell you why the original run failed. It is also expensive: every retraining attempt consumes time and introduces another result that must be interpreted.

The protocol below is therefore ordered by cost, not by how interesting a test feels.

These use information already produced by the run.

1. Are the losses finite?
   If the loss is NaN or Inf, downstream metrics and comparisons cannot be trusted.

2. What is the metric doing?
   A stable loss with AUC near chance can indicate that the model has converged to a useless solution rather than a useful one.

3. What do the predictions look like?
   Inspect their minimum, maximum, spread, and number of distinct values. A model producing essentially one value has collapsed. A model producing only a narrow band of probabilities may be heavily regularised or otherwise unable to separate decisions.

4. Is the result implausibly better than a sensible baseline?
   If yes, investigate leakage before interpreting the number as genuine model quality.

These require looking inside the model, but do not require a full retraining run.

5. Total gradient norm.
   Extremely large gradients suggest explosion; extremely small gradients can indicate that the learning signal has disappeared.

6. Per-layer gradient profile.
   Compare gradient magnitudes across layers. If one part of a network receives much less signal than another, the network may not be training uniformly.

7. Dead-unit fraction.
   For ReLU networks, inspect how many units are persistently inactive.

8. Can the model overfit a tiny subset?
   Take 20 rows and deliberately train until the loss is near zero. A model that cannot memorise a tiny, fixed dataset has a structural problem somewhere in the model, data pipeline, or loss. This is one of the highest-value debugging tests because it sharply narrows the search.

9. Learning-rate sweep.
   Test learning rates across orders of magnitude rather than nudging the value blindly.

10. Regularisation and capacity changes.
    Only after the earlier evidence points in this direction should you change dropout, weight decay, width, depth, or related capacity controls.

The ordering is deliberate. The expensive checks are often the ones that feel most productive, but the cheap checks can eliminate whole classes of explanations before you spend another training cycle.

The reference run passes the protocol:

• losses remain finite;
• AUC is clearly above chance;
• accuracy is above the majority-class rate;
• predictions span a wide range and contain hundreds of distinct values;
• the gradient profile is reasonably flat across layers; and
• the model can memorise 20 rows when explicitly asked to do so.

That gives us a concrete picture of normal.

The six cases that follow each violate a different part of that picture. Your job is not to spot the answer from the section title. First inspect the evidence, then state the diagnosis, then use the proposed fix to test whether the diagnosis was actually correct.

Each case uses the same diagnostic structure:

1. inspect the evidence;
2. state the most likely diagnosis;
3. explain the mechanism that connects the evidence to the diagnosis;
4. change the configuration in the smallest defensible way;
5. verify that the result improves.

Before reading the diagnosis below the output, pause and make your own statement of the evidence. The important skill is not guessing the answer; it is being able to justify it.

Evidence. This is gallery shape D: the loss settles into a flat, stable line. The final loss is 0.6158, close to 0.6109, the loss obtained by a constant prediction at the dataset base rate. That is a crucial clue: the loss has not merely stopped improving; it has settled near the loss of the trivial constant predictor.

The validation AUC is at chance, validation accuracy equals the majority-class rate, and the prediction summary is decisive: only two distinct values occur across 250 validation rows, in a narrow range, with a standard deviation far smaller than the reference run.

This is worth distinguishing from the more extreme collapse shown in Lesson 3.1 §6, where every prediction became literally identical. Collapse is a continuum. A model can retain a tiny amount of variation while still having lost almost all useful predictive behaviour.

Diagnosis. The model has collapsed to an almost-constant predictor. The learning rate is too large, so the parameter updates are too aggressive. As discussed in Lesson 3.3, a sufficiently large update can push many first-layer ReLU units into inactive regions. Once those units stop receiving useful gradients, the network can become effectively unable to recover during the run.

Why the loss misleads. Binary cross-entropy has a simple trivial solution: if the model knows only the overall positive rate, predicting that rate for everyone minimises the loss among constant predictions. Therefore a collapsed classifier can have a smooth, stable, apparently "converged" loss.

The important distinction is:

convergence means the optimisation has stopped changing the solution; it does not mean the solution is useful.

That is why the metric and prediction distribution are more informative here than the flat loss curve.

Fix. Reduce the learning rate and retrain from the beginning. Once the network has collapsed, simply changing the learning rate after the fact does not undo the parameter trajectory or restore units that have become inactive.

This case demonstrates the opposite problem.

A learning rate is the scale factor that controls how far the optimiser moves the parameters in response to the gradient. If it is too large, updates can overshoot or destabilise training. If it is too small, training may remain perfectly stable while making too little progress within the available epoch budget.

That second failure is easy to miss because almost everything looks "healthy."

Evidence. This is gallery shape C. Training and validation loss both descend smoothly, but they are still descending at epoch 40. There is no convincing plateau. Validation AUC is also still climbing: the model is improving, not merely fluctuating around a finished solution. The gradient profile is healthy.

Diagnosis. The run is undertrained. The learning rate is so small that each update moves the parameters only a little. The optimiser therefore needs more epochs than the experiment allows to reach the region that a better learning rate can reach within the same budget.

Why this one is dangerous. There is no dramatic symptom. There is no NaN, explosion, collapse, or obvious instability. If you stop at epoch 40 and report the resulting AUC as the model's capability, you may incorrectly blame the dataset or model class for a result that was caused by the training schedule.

This is the distinction to remember:

A run that has not plateaued has not yet measured its attainable performance.

A low final score is not automatically evidence that the task is hard. First establish whether optimisation has actually finished.

Fix. Raise the learning rate, or use the learning-rate range test from Lesson 3.1 §7 to identify a useful scale directly. Increasing the epoch budget could also allow progress, but if the learning rate is orders of magnitude too small, choosing a sensible learning rate is the more direct test of the diagnosis.

Regularisation deliberately makes a model less free to fit the training data. Techniques such as dropout, weight decay, and reducing model capacity can improve generalisation when a model is overfitting.

But regularisation has a cost: if it is too strong, it can prevent the model from learning a sufficiently useful function at all.

This case is particularly important because the loss curve itself looks healthy.

Evidence. The loss curves are calm and well behaved. Validation AUC is respectable, but validation accuracy is exactly the majority-class rate. Predictions are also concentrated in a narrow middle range and do not cross 0.5.

Diagnosis. The model is over-regularised. The small architecture, aggressive dropout, and strong weight decay collectively restrict the model so much that it can retain some ability to rank examples without producing sufficiently separated probabilities for threshold-based decisions.

To understand this, separate two ideas:

• Ranking: are positive examples generally assigned higher scores than negative examples? AUC measures this.
• Classification at a threshold: after choosing a threshold such as 0.5, how many examples are labelled positive or negative? Accuracy depends on this decision rule.

AUC is therefore threshold-free. A model can rank examples reasonably well while leaving every probability below the chosen classification threshold. In that situation, AUC can look good while threshold-based accuracy remains at the majority-class baseline.

The trap. Looking only at AUC would make this model appear successful. Looking only at the loss would also miss the problem. The useful diagnosis comes from combining AUC + accuracy + prediction range.

How to catch it. If AUC is respectable but accuracy equals the majority-class rate and the predictions are tightly clustered, investigate whether regularisation or capacity is preventing the model from separating the classes strongly enough.

Fix. Reduce the regularisation. As Lesson 3.2 established, first make sure early stopping/checkpoint selection is sensible, then add regularisation only when it measurably improves the relevant validation behaviour.

The first three cases were failures of the model or training configuration. This case is different: the model may be behaving exactly as instructed, but the experiment has invalid evaluation data.

That distinction is fundamental.

A model is evaluated by comparing its predictions with outcomes on data that represents the situation in which the model will be used. If information from the training set leaks into validation, the validation score no longer estimates performance on genuinely unseen data.

The result can look excellent precisely because the evaluation has stopped being independent.

Evidence. The validation curve tracks the training curve almost exactly. There is effectively no generalisation gap. On the 500-row dataset used here, that is suspicious because the model can reach a training AUC of 1.0 while a genuinely unseen validation set should not simply reproduce the same behaviour.

The validation AUC is also far above the reference run.

Diagnosis. The validation rows are drawn from the training rows. The model is therefore being evaluated on examples it has already seen during training. The label "validation" does not make those rows independent.

This is validation leakage: information that should be held out for evaluation has become available to the training process.

Why this is especially dangerous. Cases 1–3 produce poor or limited numbers, which naturally invite investigation. A broken validation split can produce a beautiful number. That makes the bug more likely to survive review and reach deployment.

The exact bug here is blatant, but real projects often leak information in subtler ways:

• duplicate records are removed only after the split, allowing duplicates into both sets;
• rows from the same customer or patient are randomly split, so the same entity appears on both sides;
• time-series rows are randomly split, allowing the model to train on information from the future;
• preprocessing statistics are fitted on the full dataset before splitting.

The last case explains the common pipeline pattern:

pre.fit_transform(X_train)
pre.transform(X_val)

The preprocessing object learns its parameters from training data only. Validation data is transformed using those already-learned parameters.

The tell. A validation score that is suspiciously close to, or even better than, training performance deserves an investigation of the split and data flow. A small generalisation gap can be perfectly normal; the point is not that every small gap is leakage, but that an implausibly perfect evaluation should be explained.

Fix. Rebuild the split so that validation examples are genuinely held out. Where rows have a natural group structure, split by the group rather than by individual row.

A neural network does not begin from a neutral state. Its initial parameter values determine the numerical scale of the first forward pass and therefore the scale of the first backward pass.

Good initialisation aims to keep activations and gradients in a useful numerical range. Poor initialisation can put the network into an extreme state before learning has meaningfully begun.

This case makes that visible by deliberately starting with weights that are far too large.

Evidence. The first loss is in the hundreds rather than near the roughly 0.6–0.7 range expected for a sensibly initialised binary classifier whose initial predictions are near the dataset base rate. The initial gradient norms are also orders of magnitude larger than those in the reference run.

The loss decreases, but never reaches a reasonable region, and AUC remains poor — below both the reference run and the logistic-regression baseline.

Diagnosis. The weights were initialised far too large. As discussed in Lesson 3.3, oversized weights create oversized pre-activations. That pushes the network into an extreme numerical regime and can produce very large gradients and unstable updates. The optimiser then spends the run trying to recover from a poor starting scale.

The Tier 0 tell that costs almost nothing. For a binary classifier whose initial predictions are close to the base rate p, binary cross-entropy should be near the entropy of that base-rate prediction:

-[p·ln(p) + (1-p)·ln(1-p)]

For this dataset that is around 0.6. The exact number is not the point; the diagnostic principle is.

If the first loss is wildly outside a plausible range, investigate initialisation and input scaling before spending time tuning later training behaviour. The first forward pass is a cheap window into whether the model started in a sensible numerical regime.

Every previous case had an obvious statistical or optimisation symptom.

This one is different.

The curves are healthy. The gradient profile is healthy. The generalisation gap looks normal. Predictions are well spread. The model appears excellent.

And the model is still unusable.

This is the case that forces you to recognise a boundary of model diagnostics: some failures cannot be discovered by looking at the training run alone. You must understand what the features mean and when they exist in the real process.

Evidence. There is no obvious internal failure. Every Tier 0 and Tier 1 diagnostic passes. The curves match gallery shape A, gradients are well behaved, predictions are well spread, and the generalisation gap is small and stable.

The only statistical warning is Tier 0 check 4: the result is much better than a sensible baseline and there is no credible explanation for the size of the improvement.

Diagnosis. The problem is the duration feature. It records how long the sales call lasted. Call duration is strongly related to whether the call ultimately succeeds, so it is highly predictive inside the historical dataset.

But the prediction is supposed to answer a question such as "who should we call?" At the moment that decision must be made, the call has not happened yet. Therefore duration does not exist.

This is target leakage: an input contains information that is available only because the outcome, or a process downstream of the outcome, has already occurred.

That makes this different from validation leakage:

• Validation leakage contaminates the separation between training and evaluation data.
• Target leakage uses an input feature that would not be available at the real prediction time.

Within the historical table, duration is simply a very good predictor. Cross-validation can repeatedly confirm that it is predictive. The problem is not that the statistical relationship is false.

The problem is that the relationship is temporally invalid for deployment.

This means leakage is partly a data-science problem and partly a domain-understanding problem. A person has to know where each feature comes from and when it becomes available.

For every feature, ask:

1. At the moment I need to make the prediction, does this value exist yet?
2. Could this value have been influenced by the outcome I am trying to predict?
3. If I deployed the model tomorrow, could I actually obtain this value in time?

A result that is dramatically better than a sensible baseline should therefore be treated as an investigation trigger, not automatically as a success.

Now the answer key disappears.

There are two configurations below. For each one:

1. run the experiment;
2. inspect the curve and Tier 0 output;
3. work through the protocol from Section 3 in order;
4. write the evidence before writing the diagnosis;
5. propose the smallest defensible fix;
6. run the fix and check whether the result changes in the way your hypothesis predicted.

Resist the urge to guess and immediately re-run. The point of the exercise is to practise evidence → hypothesis → test, not trial-and-error tuning.

A note about Mystery B: it contains more than one fault. That means you will need to distinguish the primary symptoms and avoid stopping after finding the first plausible explanation.

A diagnosis that cannot be reconstructed three months later is of limited value. The point of an experiment record is not bureaucracy; it is to preserve the reasoning that connects a configuration to an observed result.

The blueprint makes documentation a cross-cutting practice for exactly this reason, and Module 4 will turn it into a formal experiment protocol.

The format below is deliberately minimal. It captures five questions:

1. What did I try?
2. Why did I try it?
3. What happened?
4. What did I learn?
5. What would I change next?

The log also records the configuration differences from the baseline. That makes the experiment easier to reproduce and makes the causal comparison visible without forcing you to copy a large configuration block every time.

A good record should let your future self reconstruct the reasoning without needing to trust your memory.

Diagnose before you intervene. Changing something plausible and re-running is search, not diagnosis. It can find a working configuration while teaching you very little about the original failure.

Order checks by cost. Tier 0 checks — finite losses, metrics, prediction spread, majority-class comparison, and baseline plausibility — cost almost nothing. Use them before spending another training cycle.

The loss is evidence, not the diagnosis. A flat loss can mean healthy convergence or collapse. A smooth descending loss can mean genuine progress or simply that the run has not trained long enough. Always read the loss alongside metrics, predictions, and the reference run.

Use prediction behaviour as a first-class diagnostic. A model that emits nearly one value has a very different problem from a model whose predictions are spread across the full range. The metric alone may not reveal that distinction.

The tiny-subset overfit test is a powerful structural check. If the model cannot memorise a tiny set of examples when deliberately given enough optimisation, suspect the model, data pipeline, or loss before tuning ordinary hyperparameters.

Good numbers deserve scrutiny too. A broken validation split can produce excellent validation scores. A leaked feature can produce excellent curves and healthy gradients. Statistical diagnostics cannot replace knowledge of how the data was generated and when each feature becomes available.

Write the reasoning down while it is fresh. A short experiment record turns a training run into reusable knowledge rather than another result you will have to rediscover later.

You began the day able to write a training loop. You can now:

• choose an optimiser and learning rate with a reason, and identify a sensible learning-rate scale;
• recognise common learning-rate and loss-curve signatures;
• deliberately create overfitting and choose an intervention based on evidence;
• distinguish a genuine improvement from variation by measuring the relevant noise;
• inspect a network's gradient profile layer by layer;
• identify and explain vanishing gradients, exploding gradients, and dead units;
• work from an unfamiliar symptom to a testable diagnosis and then to a defensible intervention.

That is the Module 3 competency the blueprint asks for:

Explain why a training run is failing and propose a technically defensible intervention.

Module 3 has focused on making a single run work and understanding why it behaves as it does.

Module 4 asks the harder experimental question: how do you run multiple experiments so that the conclusions remain defensible when someone challenges them?

The foundations are already visible here. Lesson 3.2 §9 measured a noise floor and showed that many small differences were indistinguishable. Case 4 showed how a broken split can manufacture a beautiful result. This notebook's config runner already writes structured results to disk, which is the beginning of experiment tracking.

Module 4 turns those ideas into a broader discipline:

• controlled comparisons;
• reproducibility;
• leakage prevention;
• GPU memory and mixed precision;
• and a gradient-boosting versus neural-network comparison in which the neural network is not assumed to win.

The goal is not to make one model look good. It is to make experimental conclusions you can defend.