Experiments You Can Reproduce

People use one word for three different things. Keep them apart, because each one fails for its own reasons.

Level 1 — repeatable. Same code, same data, same machine, same software. Run it twice, get identical numbers, to the last decimal. If this fails, something in your run is out of your control and nothing built on it can be trusted.

Level 2 — portable. Same code and data, a different machine or a different software version. Numbers that are close, but usually not identical to the last decimal. Level 2 is not about forcing them to match; it is about recording enough that the difference can be explained.

Level 3 — robust. Same method, different random choices: another seed, another split of the rows. The numbers move. The question is whether your conclusion survives the movement. A finding that only holds for one seed is not a finding.

You have already met Level 3 without the name. Module 2 and Lesson 3.1 report a best validation AUC of 0.7965. That came from a split with 80% training and 20% validation, trained with the Adam optimizer. Lesson 3.4 used a different arrangement: 80% training, 10% validation, 10% test, with AdamW. Same model, same data, same learning rate. Section 4 below will show 0.7871.

Neither number is wrong. They were measured on different rows. A score means nothing without the recipe that produced it, and this lesson is about writing the recipe down.

A training run makes several random choices:

• the starting values of the weights;
• the order the training rows are visited in;
• which units dropout switches off, if you use dropout;
• sometimes which rows go into which part of the split.

Those choices come from a random number generator, which is not really random. It is a machine that produces a fixed sequence of numbers that merely look random. Tell it where in the sequence to start — that is the seed — and it produces the same numbers every time.

The awkward part is that Python, NumPy and PyTorch each keep their own generator. Seeding one does not seed the others. So we write one function that seeds all of them.

Two lines in it need a word of explanation:

• torch.use_deterministic_algorithms(True, warn_only=True) asks PyTorch to prefer operations that give the same answer every time. Some fast GPU operations do not. warn_only=True means that if no repeatable version of an operation exists, PyTorch warns instead of stopping.
• os.environ.setdefault("CUBLAS_WORKSPACE_CONFIG", ":4096:8") is a setting some NVIDIA maths routines need before they will behave repeatably. It does nothing on CPU, and it is harmless to set anyway.

Same seed, same numbers. Different seed, different numbers. That is the whole mechanism.

Now the bug that makes notebooks unreproducible. It is not that people forget to seed. It is that they seed once, at the top, and then run cells more than once.

Each call continues further along the sequence. So the numbers a cell gets depend on everything that ran before it. Run a cell twice, add a cell above it, run the notebook out of order, and every later result changes.

The fix is the rule the training function in Section 4 follows: seed at the start of every run, not once per notebook.

There is a second half to the fix. The DataLoader shuffles the training rows each epoch, and by default it draws from the same shared PyTorch generator. So the row order also depends on everything that ran before. You can give a DataLoader its own private generator, and then its shuffling depends on nothing else at all.

With the shared generator the two calls differ. With a private generator seeded to 0 they are identical, no matter what else has run. Every DataLoader from here on gets its own.

• The number of threads, and the hardware. Maths libraries split work across CPU cores and add the pieces up in whatever order they finish. Adding floating-point numbers in a different order can change the last few digits.
• Library versions. A new PyTorch release can change how an operation is computed.
• Some GPU operations, which are fast precisely because they do not guarantee an order.
• Anything outside the process: a data file that changed, a download that returned a new version.

None of these can be seeded away. They are Level 2 problems, and the answer is to record them, which Section 6 does.

This is Lesson 4.1's work, compressed into three cells so this notebook stands on its own. If a step here is unfamiliar, 4.1 explains each one at length.

What the three cells do:

1. download bank-full.csv if it is not already here, and drop duration;
2. cut the rows into training, validation and test, keeping the share of yes answers the same in each part (stratified);
3. fit the scaler and one-hot encoder on the training rows only, then transform all three parts with that same fitted object.

Module 3 wrote a training loop in a cell and edited it whenever something needed changing. That works once. It stops working when you want twenty runs that differ in one setting, because now you cannot tell which version of the cell produced which number.

The fix is a function whose every setting arrives in one dictionary. Then the settings are the experiment, and they can be saved, compared and hashed.

Build it in pieces first, then assemble.

Three linear layers with ReLU between them, exactly the Module 2 and Module 3 model. nn.Sequential just runs the layers in order.

Note the torch.manual_seed at the top: the starting weights are random, so the model has to be built at a known point in the sequence for the run to be repeatable.

After each epoch we want to know how the model is doing. torch.no_grad() says "do not keep the information needed for a backward pass here", which makes scoring faster and uses less memory, since we are not going to train on it.

torch.sigmoid turns the model's raw output into a number between 0 and 1, so it can be read as a probability.

One dictionary holds every choice. Anything you pass in replaces the matching entry; anything you leave out keeps the value below. So an experiment is described by what it changes, which is usually one or two things.

Now the whole thing. Walking through what it does, in order:

1. seed everything, so the run starts from a known point;
2. split the rows and prepare the arrays, using the rules from Lesson 4.1;
3. build the model and the optimizer. AdamW is the optimizer from Module 3; it keeps two running averages per weight, which will matter in Section 7;
4. BCEWithLogitsLoss is the loss for a yes/no problem. It expects raw outputs, not probabilities, and applies the sigmoid itself, which is more numerically stable;
5. DataLoader with its own generator, so the row order is under our control;
6. for each epoch, walk the training rows in batches: clear the old gradients, run the batch forward, measure the loss, run backward, let the optimizer update the weights;
7. after each epoch, score the validation rows and remember the epoch if it is the best so far. copy.deepcopy takes a full copy of the weights, so later training does not overwrite the best ones;
8. at the end, restore the best epoch's weights and score the test rows once. Choosing the epoch on validation and reporting on test is the point of having three parts;
9. return a dictionary with the history, the scores, and the artifacts we might want afterwards.

The function refers to three helpers we have not written yet: run_record, save_checkpoint and load_checkpoint. Sections 6 and 7 build them. Python only looks them up when the function runs, so we define them now, in their simplest form, and replace them properly later.

Best epoch 18, validation AUC 0.7871. That is Lesson 3.4's number reproduced exactly: same rows, same model, same settings.

The test AUC printed beside it is new. It is the first test-set number for Bank Marketing in this course, because Module 3 only ever looked at validation. The difference matters: the validation score helped choose the epoch, so it is slightly flattered. The test score chose nothing, so it is the one to report.

And the comparison from Section 1 is now concrete: Module 2's 0.7965 came from a different split and a different optimizer. This is 0.7871. Same model, different recipe, different number.

The settings are now a Python dictionary. That is already better than editing a cell, but a dictionary lives inside the notebook. To share an experiment, or to rerun one next year, the settings need to be a file.

YAML is a plain-text format for exactly this. It is readable by a person, it can be loaded by a machine, and because it is text, git can show you what changed between two versions of it.

yaml.safe_dump writes a dictionary to text. sort_keys=False keeps our order rather than alphabetising.

Twenty runs produce twenty settings files, many of them nearly identical. Comparing them by eye is slow and error-prone. A hash solves this.

A hash function takes any text and returns a short string of letters and numbers. The same text always gives the same string. Different text almost always gives a different one. So the hash is a fingerprint: if two runs carry the same fingerprint, they used the same settings.

json.dumps(..., sort_keys=True) turns the dictionary into text with the keys in a fixed order, so that writing the same settings in a different order still gives the same fingerprint. [:12] keeps the first twelve characters, which is short enough to read and long enough to be unique here.

Changing one character of the input changes the fingerprint completely. Reordering the keys does not, and neither does renaming the run — because the name says how this execution was labelled, not what was computed. The learning rate does change it, because that changes the result.

YAML has a rule that catches nearly everybody who writes learning rates in scientific notation.

3e-4 comes back as the text '3e-4', not the number 0.0003, because the YAML version PyYAML follows requires a decimal point in scientific notation. Watch what happens if that text reaches the optimizer.

That is the good outcome: it fails immediately and loudly. The bad outcome is code that stores or compares the value silently and leaves you wondering why two runs differ. Write 3.0e-4 or 0.0003.

A score on its own cannot be checked. To make it checkable, each run records the things somebody would need to get it again:

• a run id — a name for this particular execution;
• the settings fingerprint from Section 5 — what was run;
• a data fingerprint — which exact rows and labels were used;
• the code version — which commit of your repository produced it;
• the software versions — the Level 2 details from Section 2.

The settings fingerprint says what you asked for. It says nothing about the data, which can change without anybody noticing: a file gets re-downloaded, a colleague fixes a column, a row is appended.

So we hash the data too. pd.util.hash_pandas_object turns every row into a number; we hash those numbers, the column names, and the labels.

One flipped label out of 45,211 changes the fingerprint. That is what you want: it is a smoke alarm, not a summary.

If your project is in git, the commit id says exactly which code ran. Two things are worth recording: the commit, and whether the working folder had uncommitted changes at the time. A run from a folder with uncommitted changes came from code that exists nowhere except that machine, at that moment.

subprocess.run runs a command as if you had typed it in a terminal and hands back its output.

This folder is probably not a git repository, and the function says so rather than inventing a value. To see it work, here is a small repository created, committed to, and then edited.

Now put it together and replace the placeholder from Section 4.

Every future run now carries this block, because run_experiment calls run_record at the end.

One more thing worth pinning: the versions for the whole project, in a file that can be installed later.

Long runs get interrupted. A Colab session times out, a laptop sleeps, a shared machine takes your job away. A checkpoint is a saved copy of the run so you can continue instead of starting over.

The usual mistake is to save only the weights. That is enough to use the model. It is not enough to continue training it, and the reason is the optimizer.

AdamW does not just subtract the gradient from each weight. For every weight it keeps two running averages: one of recent gradients, one of recent squared gradients. Those decide how big each weight's next step is. Look at them.

exp_avg and exp_avg_sq are those two running averages, and they are the same shape as the weights. Throw them away mid-run and the optimizer starts again from nothing, taking the wrong-sized steps until it rebuilds them.

So a checkpoint that can resume must hold:

• the weights;
• the optimizer state, those running averages and the step count;
• the row-order generator, so the next epoch visits rows in the order it would have;
• the global random generators, for anything else that draws random numbers;
• the bookkeeping: history so far, best score, best weights.

One argument deserves a warning. torch.load(path, weights_only=False) is needed here because the random generator states are not plain tensors. That setting lets the file rebuild arbitrary Python objects, which means a malicious file could run code on your machine. So: only load full checkpoints that you made or trust. For weights downloaded from the internet, keep the safe default, weights_only=True.

Three runs. One straight through, 30 epochs. One that saves a checkpoint every epoch and "dies" after epoch 12. Then a third that resumes from that checkpoint and finishes.

If the checkpoint is complete, the finished weights of the third run should be identical, number for number, to the first.

Identical, to every decimal place.

Now the weights-only version, to see what it costs. We take the same epoch-12 checkpoint and replace two things with fresh ones: the optimizer state and the row-order generator. That is exactly what you get when somebody saves model.state_dict() alone and builds a new optimizer and DataLoader when they restart.

A different model, quietly. Validation AUC 0.7866 instead of 0.7871, test AUC 0.7991 instead of 0.8016. Nothing crashed, nothing warned.

On a 30-epoch run the damage is small. On a long run with a learning-rate schedule, restarting the optimizer's averages part-way causes a visible jump in the loss, and restarting the row order makes the model see examples it has just seen.

This is the most useful measurement in the note, and the one people skip.

Before you can say "this change improved the model by 0.005", you have to know how much the score moves when you change nothing that matters. If it moves by 0.01 on its own, your 0.005 is nothing.

Two different things can wobble, and it is worth separating them:

• seed noise — same rows, different starting weights and row order;
• split noise — same seed, different rows in training, validation and test.

Five runs of each. This takes about a minute and a half.

The two are not the same size.

Changing the seed moves test AUC by a standard deviation of 0.0010. Changing the split moves it by 0.0130 — about thirteen times as much. Validation AUC tells the same story: 0.0020 against 0.0123.

Standard deviation here just means "the typical distance from the average". So with a different split you should expect the score to land about 0.013 away from where it landed this time, in either direction, for no reason at all.

Three things follow.

Almost all the uncertainty comes from which rows you tested on, not from the starting weights. Rerunning with three seeds and averaging is nearly free reassurance that barely touches the number that actually moves.

The average validation AUC (0.7948) is higher than the average test AUC (0.7885). That is expected, not a bug. The best epoch was chosen by validation score, so validation is the best of 30 tries and carries that luck. Test chose nothing. Lesson 4.4 compares models on test only, for exactly this reason.

The best epoch itself is noisy. Look down that column: it ranges from 8 to 22 across the ten runs. The epoch that happened to be best is not a finding.

Finally, notice the last two columns. Every run has the same data fingerprint, because the file did not change. Every run has a different settings fingerprint, because seed and split seed are part of the settings.

The numbers are saved. What is not saved is the thinking: why you ran it, what you concluded, what you would do next. Six months later that is the part you will wish you had.

Lesson 3.4 introduced this habit with five questions. Here they become a fixed format, so records can be skimmed and compared:

1. What did I try? — filled in automatically: which settings differ from the baseline.
2. Why? — your reason, written before you look at the result.
3. What happened? — filled in automatically from the result.
4. What did I learn? — your conclusion.
5. What would I change next? — the next experiment, so the record is also a plan.

The function refuses to write a record without a reason and a lesson, because a record without them is just the numbers again.

What you have built — a run function, JSON results, Markdown records, fingerprints — is a complete tracking system, and you understand every part of it.

In a team, people usually use a ready-made tool instead of a folder of files, because it adds a searchable database, a web page for comparing runs, and a shared server everybody writes to. MLflow is a widely used open-source one. Here it runs on your own machine, storing everything in a single SQLite database file, with no account and no server to set up.

The run function does not change. A small function copies a saved result into MLflow.

The vocabulary MLflow uses:

• parameters — the settings, recorded once per run;
• metrics — numbers that can change as the run progresses, recorded with a step (our epoch);
• tags — short labels for searching, which is where the fingerprints go;
• artifacts — files attached to the run, such as our JSON and Markdown.