Overfitting, and What to Do About It

Before swapping datasets, let's test the reason for the swap.

The problem we want to study is overfitting: a model having enough flexibility to fit details that are specific to its training examples rather than details that generalise to new examples.

A useful first diagnostic is the relationship between model capacity and amount of training data.

• A parameter is a trainable number in the model, such as a weight or bias.
• Model capacity is the model's ability to represent different functions. More parameters often give a model more capacity, although parameter count is only a rough proxy for capacity.
• A training example is one row the model is allowed to learn from.

If there are vastly more training examples than parameters, the model has relatively little room to give every individual row its own special treatment. It is pushed towards finding patterns shared across many examples.

If there are many more parameters than training examples, the model has much more freedom. It can potentially fit individual examples or small groups of examples instead of learning only the broad structure.

So parameters per training example is a useful diagnostic here, not a law of machine learning. The experiment below measures it for the Module 2 setup rather than assuming that the ratio alone determines whether overfitting will occur.

There it is, measured rather than asserted.

After forty epochs — far more than anyone would normally need for this small demonstration — a gap opens between training and validation AUC, and validation AUC peaks and then drifts down slightly. So the Bank Marketing model does show some overfitting.

But notice the scale of the effect. Training loss is still around 0.26 rather than approaching zero, and training AUC is still well below 1.0. The model has not memorised the training rows in the way we are about to make the German Credit model memorise its 500 rows.

The parameter-to-example ratio helps explain why. The Bank Marketing model has roughly 0.15 parameters per training example. That does not make memorisation mathematically impossible in every sense, but it gives this particular network far less capacity per example than the German Credit experiment.

This matters for teaching and for experiments generally:

A method cannot look useful if the problem it is designed to solve is barely present.

If we added dropout to the Bank Marketing experiment and saw almost no change, that would not prove that dropout is useless. It would mostly tell us that this experiment is a poor place to demonstrate its effect.

The broader engineering lesson is even more important. Before evaluating a regulariser, architecture, optimiser, or other intervention, first establish that the failure mode it is supposed to address can actually be observed in your experiment.

The dataset is changing; the basic modelling pipeline should not.

This is an important experimental principle: change one major thing at a time when you want to understand causality. Here we keep the preprocessing, tensor construction, MLP pattern, and training-loop structure familiar from Module 2, while changing the dataset and deliberately using a much smaller training set.

A few terms used in the code:

• ColumnTransformer applies different preprocessing to different groups of columns.
• StandardScaler standardises numeric features.
• OneHotEncoder converts categorical values into indicator columns.
• TensorDataset packages tensors so they can be consumed by a PyTorch DataLoader.
• The training set is used to fit the model.
• The validation set is used to make development choices such as model size or when to stop.
• The test set is held back until the end so it can provide a less biased estimate of performance.

The preprocessing is fitted on the training data only. That prevents information from the validation or test rows from influencing the transformation learned from the training set.

Roughly a hundred parameters for every training row, compared with Bank Marketing's fraction of one. That is a very large change in capacity per training example, and we achieved it without introducing new modelling machinery.

The important phrase is “capacity per example.” It does not mean that every parameter stores one row. Neural networks do not work that way. It means that, relative to the amount of data available, the model has much more freedom to construct a complicated function.

That freedom makes memorisation possible. The next section deliberately pushes the model in that direction.

Now we deliberately create the failure we want to study.

The model is large relative to the 500 training examples, and we train it for long enough without an explicit regularisation method. The purpose is not to build a useful credit model. The purpose is to make the difference between training performance and generalisation visible.

The key vocabulary:

• Training performance tells us how well the model fits examples it was allowed to learn from.
• Validation performance tells us how well that same model behaves on examples it did not train on.
• Generalisation means performing well on new examples drawn from the same underlying problem.
• Overfitting occurs when continued training improves the fit to the training data while performance on unseen data stops improving or gets worse.

Watch for the point where those two stories separate.

This is textbook overfitting, and unlike Section 1 it took under three seconds to produce.

Training AUC reaches 1.0 — the model ranks all 500 training rows perfectly. Training loss also falls towards zero, which means the model is not merely getting the training labels right; it is becoming increasingly confident about them.

Meanwhile, validation loss reaches its minimum early and then climbs. Validation AUC also stops improving and begins to drift down.

The left panel shows the classic visual signature:

training and validation curves improve together at first; then the validation curve turns while the training curve continues improving.

That separation is the important event. The model is still becoming better at fitting the training examples, but those extra improvements are no longer transferring to unseen examples.

This is the distinction between fitting and generalising:

• fitting asks, “How well can I explain the examples I was given?”
• generalising asks, “How well does what I learned transfer to examples I have not seen?”

The model does not suddenly become incapable of learning after the turning point. It keeps learning — but increasingly learns details that are specific to the training set.

Now look at the right panel, because the metric there tells a different story.

The loss panel says the model is becoming badly overconfident on validation data. The AUC panel says the deterioration in ranking is much smaller.

Both measurements are calculated from the same predictions on the same validation rows. The reason they disagree is that they measure different properties of those predictions.

Before deciding which curve to monitor, you need to understand exactly what each metric can and cannot see.

Now the disagreement makes sense.

ROC-AUC measures ranking quality. Informally, it asks:

If I randomly choose one bad-credit row and one good-credit row, how often does the model assign the bad row a higher score?

Because AUC cares about the ordering of scores, it is largely insensitive to a transformation that preserves that ordering. For example, making predictions more extreme without changing their order can leave AUC unchanged.

BCEWithLogitsLoss evaluates the quality of the predicted probabilities implied by the logits. A confident mistake is much more costly than a mistake made with moderate confidence.

For example, predicting 0.99 for an event that does not happen is treated much more harshly than predicting 0.60 for the same event.

That gives us two different questions:

• AUC: Are the risky cases generally ranked above the less risky cases?
• Loss: Are the predictions, including their confidence, compatible with the observed outcomes?

The overfitted model has therefore retained much of its ranking ability while becoming increasingly extreme in its predictions. That is why the AUC curve can look relatively stable while validation loss becomes much worse.

The histogram makes this visible: the final model pushes many predictions towards the extremes, while the earlier checkpoint keeps more predictions in the middle.

There is no universal answer. The correct metric depends on what the model's output is used for.

• If the output is used mainly for ranking or prioritisation — for example, reviewing the 100 highest-risk applications — AUC directly describes an important part of the task.
• If the output is used as a probability in an expected-loss calculation, threshold decision, or human-facing risk estimate, probability quality and calibration matter much more.
• If you are deciding when to stop training, validation loss can provide a useful stopping signal here because it reacts strongly to the overconfidence that appears after the best checkpoint.

The important habit is not “always use loss” or “always use AUC.” It is:

Know what failure modes your metric can see, and check another measurement for failures it cannot see.

Once you see overfitting, it is tempting to add dropout or weight decay immediately.

Start with the simplest question instead:

Do I actually need a model this large?

Section 2 showed that the model has much more capacity than the 500 training rows require. The most direct response to excess capacity is to reduce the capacity.

A smaller network has fewer trainable parameters and therefore fewer degrees of freedom. That can make it harder for the model to memorise individual training examples.

This is often called a regularisation effect: we are constraining the set of functions the model can represent so that solutions that fit arbitrary training-set details become harder to reach.

Capacity reduction is different from the other interventions in this notebook because it changes the model itself rather than adding a penalty or changing when training stops.

The trend in validation loss is monotone in this experiment: reducing capacity lowers the validation loss, with the smallest model performing best. Validation AUC is noisier — the (64,32) model is slightly below the largest model — which is our first warning that small numerical differences may be measurement noise rather than genuine improvements.

The clearer evidence is in the overfitting pattern. Shrinking the network greatly reduces the upward drift in validation loss and narrows the gap between training and validation behaviour. The smallest model never reaches a training AUC of 1.0, so it has not memorised the training set in the same way as the oversized network.

That does not, by itself, prove that the smallest model is perfect. A model can fail to reach training AUC of 1.0 for two very different reasons:

1. it is appropriately constrained and is learning the useful structure; or
2. it is too small to represent important structure in the problem.

This is the distinction between regularisation and underfitting. If we reduce capacity too far, both training and validation performance can suffer because the model is no longer expressive enough.

That is why capacity reduction is a useful first experiment: it is cheap, simple, and it tells us how much capacity the problem appears to need.

Why is a small model not always the answer? Some problems genuinely require large models. Images, language, and other high-dimensional tasks can contain structure that a tiny network simply cannot represent. In those settings, we may want to keep a large model while using constraints such as weight decay, dropout, data augmentation, or other methods to reduce harmful memorisation.

Weight decay adds a pressure for model parameters to remain smaller during training.

The simple intuition is that very large weights can help a network create sharp, highly specific responses. A function that bends sharply to pass through many scattered training examples can require stronger parameter values than a smoother function that captures a broad pattern.

Weight decay therefore adds a preference for solutions with smaller parameter magnitudes.

In this notebook, the implementation uses AdamW, which applies weight decay in a way that is decoupled from Adam's gradient-based update. Lesson 3.1 covered that optimisation detail; here the important idea is the modelling effect:

Weight decay does not remove parameters. It makes large parameter values more costly to maintain.

That distinction matters when we compare it with reduced capacity in Section 5.

Weight decay does what the mechanism predicts: as the coefficient increases, the final weight norm falls, and the validation-loss blow-up becomes less severe.

A weight norm is a single number summarising the overall magnitude of the model's parameters. A lower norm therefore confirms that the regulariser is actually changing the parameter values, not merely producing a coincidental change in the validation curve.

Be precise about the result. At the strongest setting, the validation-loss blow-up is reduced, not eliminated. At the smaller coefficients, the effect is barely visible in this experiment.

The validation AUC also changes much less than the validation loss. That is understandable from Section 4: weight decay is affecting the confidence and shape of the learned function, while AUC only cares about the ordering of predictions.

So the correct conclusion is not “weight decay improves AUC” or “weight decay does nothing.” The more precise conclusion is:

In this experiment, weight decay strongly changes parameter magnitude and reduces loss deterioration, while its effect on ranking quality is comparatively small.

Notice another important difference from Section 5. The models can still reach training AUC of 1.0. Weight decay has not removed the model's capacity; it has changed which parameter configurations the optimiser prefers.

Reduced capacity: remove some of the model's ability to represent complicated functions.

Weight decay: keep the capacity, but discourage solutions that require large weights.

That distinction is worth remembering because the two methods can produce similar-looking improvements for different reasons.

Dropout randomly sets some activations to zero during training.

An activation is the value produced by a neuron after its input has passed through the layer and activation function. With dropout, some of those values are temporarily removed on each training step.

This means the model cannot depend too heavily on any one unit being present. Across many updates it is encouraged to distribute useful information across multiple pathways instead of building a fragile solution that depends on a particular collection of units.

There is a crucial asymmetry:

• during training, dropout is active;
• during evaluation, dropout must be disabled.

PyTorch controls this behaviour through model.train() and model.eval().

This is not just a cosmetic switch. It changes the behaviour of layers such as Dropout and BatchNorm. Forgetting to call model.eval() before validation or testing can therefore make your reported metric noisy or systematically wrong.

We will deliberately create that bug before fixing it.

Two symptoms identify this bug, and both are worth remembering.

If dropout is still active, the model is being evaluated with randomly removed activations. You are therefore not evaluating the deterministic full network you intend to deploy.

This is the giveaway.

If you evaluate the same model on the same data twice and the predictions or metrics change, some stochastic training behaviour is probably still active. In this experiment, leaving the model in training mode leaves dropout switched on.

A good development check is therefore simple: evaluate twice. If identical inputs and identical model parameters produce different predictions when you expect deterministic evaluation, investigate the model mode and other sources of randomness.

The same train()/eval() discipline matters for BatchNorm. In training mode, BatchNorm uses statistics computed from the current batch; in evaluation mode, it uses the running statistics collected during training. Lesson 3.3 covers BatchNorm in detail.

The dropout results themselves show a moderate reduction in the validation-loss blow-up, especially at stronger dropout rates. The effect on AUC is comparatively small in this experiment.

Section 3 showed validation loss reaching its minimum early and getting worse when training continued.

Early stopping turns that observation into a training rule:

1. monitor a validation metric;
2. remember the best value seen so far;
3. save the corresponding model parameters;
4. stop training when the metric has failed to improve for a chosen number of epochs.

That waiting period is called patience.

Early stopping is different from dropout and weight decay because it does not directly change the model architecture or add a penalty to the training objective. It changes how long you train and which checkpoint you keep.

In this experiment, that matters because the model is capable of continuing to reduce training loss after validation performance has already peaked.

Three details in the implementation matter more than the basic concept.

Suppose the best validation loss occurred at epoch 12 and your patience is 5. If you simply stop at epoch 17, you are holding the model from epoch 17, not the model that achieved the best validation result.

load_state_dict(best_state) restores the parameters from the best checkpoint.

So early stopping is really two operations:

choose when training should stop + keep the best model found before stopping.

Validation loss does not necessarily improve smoothly every epoch. A small upward movement can happen even while the overall trend is improving.

A patience of 1 reacts to every small upward fluctuation. A patience of 10 requires the model to fail to improve for a much longer period before stopping.

The correct patience therefore depends partly on how noisy your validation measurement is.

The validation set is no longer a neutral measurement if you repeatedly use it to choose among checkpoints.

For example, if you inspect 200 checkpoints and keep the one with the best validation AUC, the final validation AUC is optimistic: you selected the checkpoint partly because that particular validation set liked it.

That is why this notebook keeps a separate test set. The test set is not used to choose the model or stopping point. It is held back for the final measurement.

This distinction becomes part of the experimental protocol in Module 4.

Every regularisation method has a strength setting, and pushing that setting too far can create a different problem: over-regularisation.

Over-regularisation means the constraints have become strong enough that the model is no longer free to represent useful structure.

The visual pattern is often the opposite of ordinary overfitting:

• the model is constrained enough that it cannot fit the training data as closely;
• useful signal can be suppressed;
• both training and validation performance can suffer.

There is a second issue here that is easy to miss: validation measurements are noisy.

If two configurations differ by 0.005 AUC, that difference might be real — or it might simply be caused by which rows happened to be placed in the validation set.

This section therefore asks two questions:

1. What does over-regularisation look like?
2. How large does a validation difference need to be before we should take it seriously?

Here is the trap.

The over-regularised model has the highest best validation AUC in this notebook. If AUC were the only number you looked at, you might conclude that it is the best model.

But the rest of the evidence says something important. Its validation accuracy has collapsed to approximately the majority-class rate, and its predicted probabilities are compressed into a narrow range that rarely crosses the 0.5 decision threshold.

This exposes another limitation of AUC.

AUC is threshold-free: it evaluates how well the model orders examples across possible thresholds. It does not tell you whether a particular threshold, such as 0.5, will produce useful decisions.

So this model can still rank bad-credit cases above good-credit cases reasonably well while producing almost no positive classifications at the chosen threshold.

Compare this with Section 4:

• the overfitted model became too confident while retaining much of its ranking ability;
• the over-regularised model became too cautious while also retaining reasonable ranking ability.

AUC does not directly reveal either calibration problem.

The broader lesson is:

A metric can be excellent at measuring one property while being completely blind to another property you care about.

Now we reach the question that should sit behind every comparison in this notebook:

How large does a difference have to be before we can reasonably treat it as evidence of an improvement?

That number should change how you read every table in this notebook.

The same model configuration and seed, evaluated on different validation splits, produces a noticeable spread in AUC. That spread is the noise floor of this measurement: the amount the reported score can move simply because the validation examples changed.

Two standard deviations are only a rough descriptive yardstick here, not a universal statistical significance test. But the practical message is clear: if the difference between two configurations is smaller than the normal variation of the measurement, the experiment has not established that one configuration is genuinely better.

This is why several close comparisons in Sections 6 and 7 should not be turned into confident rankings. The capacity effect in Section 5 is much larger and more consistent; many of the smaller dropout and weight-decay differences are within the range of ordinary validation variation.

Three habits follow:

• Measure your noise floor before tuning. Then you know what size of improvement your experiment can realistically resolve.
• Report variability, not only a single point. Repeat across splits or seeds when close comparisons matter.
• Be suspicious of small gains after many trials. If you try many configurations, one of them will often obtain an unusually good validation score by chance. Selecting that winner does not prove that the underlying configuration is better.

This is a central idea in empirical machine learning: measurement noise and selection can create apparent improvements even when the underlying models are equally good.

Module 4 turns these ideas into a more formal experimental protocol.

The final table brings the interventions together.

The important question is not “Which technique has the highest number?” Instead, read the table alongside the mechanisms and the measurement noise from Section 9.

The test set is included here because it has been kept separate from model selection. A test score is meaningful only to the extent that we have not repeatedly used the test set to choose the model.

Read that last block of output carefully. The most important result is not a tidy ranking of regularisers.

Once early stopping is applied, the configurations in the final table are statistically indistinguishable within the resolution of this experiment on the untouched test set. The spread is smaller than the noise floor measured in Section 9.

That does not prove that the methods are universally equivalent. It means that this experiment, with this dataset, split, model family, seeds, and sample size, does not provide enough evidence to separate them reliably on the test measurement.

That distinction matters. “We cannot distinguish them here” is different from “they are mathematically identical.”

1. Overfitting requires enough freedom relative to the available data.

The Bank Marketing setup showed only mild overfitting in this experiment. German Credit, with only 500 training rows and a much larger model relative to the dataset, made the failure obvious.

Before choosing a fix, first establish that the failure mode exists.

2. Validation loss and validation AUC answer different questions.

Loss is sensitive to confidence as well as correctness. AUC focuses on ranking. The overfitted model stayed a useful ranker for longer than it remained a well-behaved probability estimator, while the over-regularised model could retain ranking ability while producing predictions that were too compressed for a 0.5 decision threshold.

3. Early stopping mattered strongly in this experiment.

With a fixed 60-epoch budget, capacity appeared to produce large differences because the larger models were allowed to continue past their useful checkpoints. When early stopping is applied, those differences largely disappear on the final test measurement.

So part of what the other regularisation methods appeared to accomplish was simply preventing the model from being trained beyond the point where validation performance was best.

That is a conclusion about this experiment, not a universal claim that early stopping makes every other regulariser unnecessary.

4. Restore the best weights.

Stopping at the end of the patience window is not enough. You must keep the parameters from the best validation checkpoint; otherwise you deliberately return to a later, worse model.

5. Measure the noise floor before trusting a ranking.

This is the habit that turns the notebook from a list of techniques into an experimental method.

If two configurations differ by less than the normal variability of your validation/test measurement, the honest conclusion may be that your experiment cannot yet tell them apart.

That is not a disappointing result. It is useful information about the resolution of your experiment.

The next step is to formalise that discipline: control the split, seeds, model-selection procedure, and final evaluation so that improvements are separated from noise.

Lessons 3.1 and 3.2 both treated the network as a single object with a loss curve attached. When it trains badly you change a global setting — the learning rate, the capacity, the decay — and observe the curve again.

That works until it does not. Some failures are not global: the network trains, the loss goes down, and yet one part of it is receiving no useful gradient at all, or receiving gradients so large they destroy it. From the outside these look like ordinary slow learning. Lesson 3.3 opens the model up, promotes the gradient-norm probe from Module 2 into a per-layer instrument, and makes those failures visible — along with the interventions that prevent them.