Data You Can Trust

Before any code, get one idea clear, because the rest of the note depends on it.

When you split a dataset, you cut the rows into groups and give each group a job:

• training set — the rows the model learns from;
• validation set — rows the model never learns from, used while you are still making choices (which learning rate, how many epochs, which model);
• test set — rows nobody touches until the very end, used once, to report a final number.

Now the important part. The test set has a part to play. It is pretending to be the future: the data your model will meet after you put it to work.

the rows you have                        what they pretend to be

 ┌──────────────┐                         ┌──────────────┐
 │  training    │ ──────────────────────► │  the past    │
 └──────────────┘                         └──────────────┘
 ┌──────────────┐                         ┌──────────────┐
 │  test        │ ──────────────────────► │  the future  │
 └──────────────┘                         └──────────────┘

A split is good exactly as far as that pretending is convincing.

So the question to ask about any split is: who, and when, will this model be used on? For Bank Marketing the honest answer is: customers the bank has not phoned yet, on days that have not happened yet. Hold on to that sentence. Every section below tests one part of it.

You have already seen two ways this pretending fails, in Module 3:

• Lesson 3.4, Case 4 put copies of training rows into the validation set. The validation score went to a perfect 1.0000. The split was pretending the future is the past.
• Lesson 3.4, Case 6 kept a column called duration, the length of the phone call. You only know how long a call lasted after it has finished, so the model was reading something from the future. The score jumped to 0.9262 and meant nothing.

Both of those were deliberately obvious. Real ones are quieter, which is why this lesson measures rather than argues.

Let us make a split with ten rows, so you can see exactly what the words mean before we use 45,211 of them.

The tool is train_test_split from scikit-learn. It takes a list of things and cuts it in two. Its arguments:

• the first argument is what to cut, here the row numbers 0..9;
• train_size=0.6 means 60% goes to the first output, the rest to the second;
• random_state=SEED fixes the randomness, so you and I get the same cut;
• shuffle=True (the default) means it picks rows at random rather than cutting in the middle.

Two things to notice, because both come back later.

First, the function gave back row numbers, not the data itself. That is deliberate and we will keep doing it: if you carry row numbers around, you can apply the same split to the features, to the labels and to anything else you calculate later, and they will always line up.

Second, the rows are picked at random, not in order. Rows 0, 1, 2 did not go together. That is the normal choice, and Section 7 shows where it goes wrong.

Now cut the remaining rows in two, to get the third group:

That is the whole idea: cut once to get training, cut the leftovers again to get validation and test. Sixty percent, twenty, twenty. We will do exactly this to the real data in Section 4, with the same two calls.

The next cell downloads the file if it is not already on disk. Reading it, line by line:

• os.path.exists(path) asks whether we already have the file, so a second run does not download again;
• urllib.request.urlopen(url).read() fetches the bytes from UCI;
• the download is a zip file with another zip inside it, so we open the outer one, pull bank.zip out of it, and open that to reach bank-full.csv;
• io.BytesIO lets zipfile treat those bytes as if they were a file on disk;
• pd.read_csv(path, sep=";") reads the CSV. The sep=";" matters: this file separates columns with semicolons, not commas. Without it you get one wide useless column.

The last column, y, is the label: "yes" if the customer subscribed to the term deposit, "no" otherwise. Everything else is an input.

Two words used throughout the course, in plain terms:

• a numeric column holds numbers you can do arithmetic with, like age or balance;
• a categorical column holds labels from a fixed set, like job or month. The numbers in them, if any, mean nothing arithmetically.

Let us see which is which, and how often each label says yes.

11.7% said yes. Write that number down; Section 10 is entirely about what it does to every score you report.

duration is the length of the phone call in seconds. Think about when you know it. You know it after the call has ended, which is also when you know the answer. A model using it is being told something that does not exist at the moment you would actually want a prediction.

This is called target leakage: an input column that carries information about the answer, information you would not have in real use.

Lesson 3.4 Case 6 measured it: keeping duration took validation AUC from 0.7853 to 0.9262. A jump like that looks like success and is worthless. Here is the reason it happens, in one line.

The usual first check on a dataset is to count missing values. Run it here and the file looks perfect.

Zero everywhere. That is not the same as clean.

Missing information often arrives disguised as a valid value. Two disguises are extremely common:

• a sentinel value: a number chosen to mean "this did not happen". -1, 999 and 0 are the usual suspects;
• a placeholder string: a word like "unknown", "N/A" or "none" sitting in a text column.

Neither is missing as far as pandas is concerned. Both are missing as far as meaning is concerned. Look for them by hand first, then we will wrap the search into a function.

The value -1 appears 36,954 times out of 45,211. pdays means "days since the customer was last contacted in a previous campaign", and -1 is how this file says never contacted before.

So a yes/no fact is hiding inside a number column. That matters for two reasons. A scaler will treat -1 as "one day before zero days ago", which is nonsense. And a model has to waste effort learning that this one value means something completely different from the others.

poutcome is the outcome of the previous campaign, and it says "unknown" on almost exactly the same share of rows. That is suspicious in a useful way: two columns may be recording the same fact. Check whether they are the same rows.

Same fact, written twice, apart from 5 rows. "We have never spoken to this person before."

Now the third column with a placeholder, contact, which records how the customer was reached. It says "unknown" on a lot of rows. Before deciding what to do with a placeholder, always check it against the label.

This is the reason you check. "unknown" is not noise here. Those customers said yes 4.1% of the time, against 14.9% for customers reached on a mobile. The word is telling us something about how the bank was running that part of the campaign, and it predicts the answer.

So the right treatment is to keep "unknown" as its own category, not to replace it with the most common value. Replacing it would throw away a real signal. (Section 8 finds another reason to care about this column.)

We have now done the same few checks by hand three times. That is the moment to write a function. Ours takes a table and returns one row per column, with:

• kind — numeric or categorical;
• n_unique — how many different values appear;
• missing — the share that pandas calls missing;
• unknown — the share equal to the string "unknown";
• sentinel — the share equal to a sentinel value we name ourselves;
• negative — the share below zero, for numeric columns;
• top_value and top_share — the commonest value and how much of the column it covers.

Two pieces of pandas used inside it:

• pd.api.types.is_numeric_dtype(s) answers "is this column numeric?";
• s.value_counts(normalize=True) counts each value and, with normalize=True, turns the counts into shares that add up to 1.

Read down the table and three more things appear that we had not looked for.

• balance is negative on 8.3% of rows. For a bank balance that is an overdraft, so it is valid data. An audit raises questions; it does not pass verdicts.
• default is one value on 98.2% of rows. A column that says the same thing about nearly everybody can carry almost no information.
• pdays is 81.7% negative, which is the sentinel we already found, showing up from a second direction.

A table is easy to skim past, so it helps to turn it into sentences that demand an answer. The function below does that: it walks the audit table and prints a question for anything above a threshold.

One more check before we split. A duplicate is a row that appears more than once. Duplicates matter for a reason that will not be obvious until Section 9: if the same row sits in training and in test, the model can score well by remembering rather than by learning.

Worth checking twice over:

• whole rows that repeat, label included;
• rows whose inputs repeat while their labels disagree, which means the data is telling you two different answers for the same situation.

None at all, which is worth recording precisely because it is often not the case. Module 8 meets duplicates again in fine-tuning data, where they are common and cause real damage.

Now we split the real file, using the same two calls from Section 1. But this time we make the split a choice we can name, because there is more than one sensible way to do it.

Four rules, in plain words:

• stratified — pick rows at random, but keep the share of yes answers the same in every part;
• random — pick rows at random and let the shares fall where they may;
• chronological — the earliest rows for training, the middle for validation, the latest for test;
• group — decide which side a whole group of rows goes to, all together.

The first one is what Module 2 and Module 3 used. The rest arrive later in this lesson, each for a reason.

First, the target. We turn "yes"/"no" into 1.0/0.0 because every model in this course expects numbers.

The word stratified means the split keeps the class balance. Our data is 11.7% yes. Without stratification, a random cut could hand the test set 10.4% and the training set 12.0% just by luck, and part of any difference you measure later would be that luck instead of your model.

train_test_split does it when you pass stratify=y. Watch it work on the real rows.

All four shares agree to three decimals. That is stratification doing its job.

Note the second call passes stratify=y[rest_idx], not stratify=y. It is splitting only the leftover rows, so it must be told the labels of those rows. Passing the whole label array there is a common and silent bug.

Now we collect the four rules into one function. It returns a dictionary with three keys, "train", "val" and "test", each holding an array of row numbers.

The pieces inside:

• fractions=(0.8, 0.1, 0.1) says how big each part should be;
• for "chronological" there is no randomness at all: we take the first 80% of rows in file order, then the next 10%, then the last 10%;
• for "group" we use GroupShuffleSplit, which keeps every row that shares a group label on the same side. Section 9 explains why that matters. Note that for this rule the fractions count groups, not rows;
• int(round(...)) converts a fraction of the row count into a whole number of rows.

Those are the same numbers we produced by hand a moment ago, which is the point: the function is not doing anything new, it is just giving the choice a name.

A split can be broken in ways that do not raise an error. Rows can land on two sides. Rows can vanish. A chronological split can quietly be out of order. So we write the checks down, and we use assert, which stops the notebook when the statement after it is false.

Each check in words:

• no row number appears in two parts (& on two Python sets gives what they share);
• the three parts add up to every row;
• for a chronological split, every training row comes before every validation row, which comes before every test row.

Now break it on purpose, so you can see the check earn its place. We copy five training rows into the validation set, which is exactly the fault of Lesson 3.4 Case 4.

Five rows out of 45,211 is a leak far too small to notice in any score, and the check caught it instantly. That is the trade: a few lines that run in a second, against a result you cannot trust.

Preprocessing is everything done to the inputs between the raw table and the model: scaling numbers, turning categories into numbers, filling gaps. Two jobs in this lesson.

Numeric columns here live on wildly different scales. age holds tens. balance holds thousands. A model that adds up inputs will feel balance far more strongly than age, for no better reason than the units.

Look at the actual ranges first.

balance spans a range roughly ten thousand times wider than age. To a model that adds up its inputs, that difference alone makes balance shout and age whisper.

Standardising a column fixes that: subtract the column's average, then divide by its standard deviation. The result has average 0 and standard deviation 1. Do it by hand once, on five numbers.

StandardScaler from scikit-learn does exactly that, one column at a time. Like most scikit-learn objects it has two methods that matter:

• .fit(data) — learn the numbers it needs, here each column's average and spread;
• .transform(data) — apply what it learned.

Keeping those two steps separate is the whole ballgame, as the rest of this section shows.

A model cannot add up the word "student". Categories have to become numbers, and the obvious idea is the wrong one. If you write admin. = 0, blue-collar = 1, management = 2, you have told the model that management is twice blue-collar and that admin is the smallest. None of that is true.

One-hot encoding avoids inventing an order. Each category gets its own column, holding 1 when the row is that category and 0 otherwise.

Four rows in, four rows out, but now three columns, and each row has exactly one 1. Row 1 and row 4 are both admin. and get identical rows of numbers, which is right.

One argument is worth knowing: handle_unknown="ignore". If a category shows up later that was not there when the encoder was fitted, the encoder writes all zeros instead of raising an error. Useful, because real data brings new categories after you ship.

Our table has both kinds of column, and they need different treatment. ColumnTransformer applies one tool to one list of columns and another tool to another list, then glues the outputs together side by side.

Reading its arguments: each entry is (nickname, tool, columns). Ours says "scale these numeric columns" and "one-hot these categorical columns".

Fifteen columns went in and 50 came out, because each categorical column expanded into one column per category. Those 50 numbers are what a model actually sees, and it is the same 50 you have been training on since Module 2 without looking at them.

The rule: fit the preprocessing on training rows only, then transform everything with that same fitted object.

The reason is that .fit() learns from data. If it learns from the test rows too, then something about the test rows has reached the model before test time, and your final number is a little bit rigged.

Lesson 3.4 Case 4 described this as a realistic version of a broken split. Let us measure it instead of describing it. Two runs, identical except for which rows the scaler learned from.

The model is logistic regression. In one line: it multiplies each input by a learned weight, adds them up, and squashes the result into a probability between 0 and 1. max_iter=3000 just gives it enough steps to settle.

The score is ROC-AUC, the same one from Module 3: the chance that a randomly chosen yes-customer gets a higher score than a randomly chosen no-customer. 0.5 is coin-flipping; 1.0 is perfect ordering.

The planned lesson here was "look how much the leak inflates the score". The measurement says otherwise, so here is the honest version.

The difference is +0.000093 AUC, far too small to matter. And the reason is easy to see: a scaler learns an average and a spread per column. With 36,168 training rows already in hand, adding 4,522 test rows barely moves either number.

So why keep the rule? Because the same mistake with a different preprocessing step is not small at all. The step does real damage when it reads the label, or when the dataset is small:

• target encoding — replacing a category with the average outcome for that category. Fitted on all rows, it copies test labels straight into training inputs. Section 9 measures a close relative of this and it costs about 0.08 AUC;
• choosing features by their correlation with the label, done before splitting, picks columns that happen to suit the test rows;
• filling missing values or scaling on a small dataset, where each test row visibly moves the statistics. German Credit's 250-row test set from Module 3 is a very different situation from 4,522 rows here;
• oversampling before splitting, which puts copies of the same row on both sides.

You follow the rule everywhere, not because every break is expensive, but because you cannot tell in advance which kind you have.

One more consequence. The fitted preprocessor holds learned numbers — 6 averages, 6 spreads, and the list of categories. A model given raw inputs without them produces nonsense. So whatever you ship, you ship both.

joblib saves a fitted scikit-learn object to a file and loads it back.

Everything so far treated the rows as a bag: order did not matter. Now look at the order.

The file has a month column and a day column, but no year. The UCI documentation says the rows are stored in date order, from May 2008 to November 2010. If that is true, then reading down the file the months should climb, and then jump backwards each time a new year starts.

We can check that, and even recover the year, with a small trick. Turn each month name into its position in the calendar, then look at the change from one row to the next.

Read the last line. Moving through the year gives small positive steps. The move from December (position 11) to January (position 0) gives −11: a big jump backwards. That is a new year starting.

So: count the big backward jumps, and you have counted the year boundaries. We use "backwards by more than 6" rather than "backwards at all", which tolerates a little shuffling inside a month.

np.cumsum adds up as it goes, so it turns a list of "is this a new year?" answers into "how many years have started so far", which is exactly the offset to add to 2008.

Only two backward steps in the entire file, and both are year boundaries. Inside a year the file never goes back even one month. So the rows really are in date order, and we now know where 2008 becomes 2009 and where 2009 becomes 2010.

Attach the year and look at the success rate per year. The year is not an input to any model here; it is information about the rows, which we use for splitting and checking.

Three years, starting in May 2008 and ending in November 2010, exactly as documented.

Now look at the middle column, because it is the most important number in this lesson.

The share of customers saying yes goes from about 5% in 2008, to 17% in 2009, to 52% in 2010. Same bank, same product, same columns, and the success rate multiplies by ten.

The plot below makes it plain. It shows the share of yes answers in a sliding window of 1,000 rows as we read down the file. The dashed lines are the year boundaries we just found, and the shaded strip on the right is the last 10% of the file, which Section 7 will use as a test set.

Here is the consequence of Section 6, and it is the heart of the note.

A stratified split picks rows at random, so 2010 rows land in the training set and 2008 rows land in the test set, all mixed together. That answers a real question:

How well does this model score a random customer from somewhere in 2008 to 2010?

But the bank's question is different:

We trained on everything up to today. How well will this model do on next month's campaign?

The split that answers that question is the chronological one: earliest 80% of rows to train on, the next 10% to validate, the last 10% to test. Train on the past, test on the future, exactly as it will happen in use.

stratified split                 chronological split

 2008 2009 2010                  2008        2009   2010
 ▓░▓░▓▒░▓░▒▓░▒▓                  ▓▓▓▓▓▓▓▓▓▓▓▓▒▒▒▒▒▒ ░░░
 mixed everywhere                train ──────► val ─► test

Same file, same model, same preprocessing code. Only the choice of rows changes.

First, a small helper so the two runs are identical in every other way. It fits the preprocessor on the training rows, fits a logistic regression on them, and scores the test rows.

Look at that table before running anything else. The chronological training rows are 6.7% yes and the chronological test rows are 47.1% yes. The model is about to be trained in one world and tested in another.

Test AUC falls from 0.7796 to 0.6108. The same model, the same columns, the same code; only the choice of which rows to test on. That drop of about 0.17 is far larger than any improvement Module 3 found by tuning anything.

Two more lines in that table are traps, and both are worth knowing by name.

PR-AUC (average precision) is another way to score a ranking, and it is often recommended for imbalanced problems. Its weakness is that its floor moves. A model that guesses at random scores roughly the share of positives in the test set. So:

• on the stratified test set, 11.7% are yes, so random guessing already scores about 0.117;
• on the chronological test set, 47.1% are yes, so random guessing already scores about 0.471.

That is why PR-AUC went up, from 0.4331 to 0.5531, while the model actually got worse. Comparing PR-AUC across test sets with different positive rates compares the floors, not the models. Always print the share of positives next to it.

Calibration asks a different question from AUC. AUC asks "does the model put the right customers at the top of the list?" Calibration asks "when the model says 30%, do 30% of those customers actually say yes?"

The chronological model's average prediction is 0.2434 while the true rate is 0.4709. It learnt from a world where 6.7% said yes, so its probabilities are anchored far too low. Any decision written as "phone everybody the model scores above 0.3" is quietly wrong, and AUC will never tell you.

Neither split is the correct one in general. They answer different questions, and the mistake is using the random split's number to answer the chronological question.

Which means being honest about Modules 2 and 3: every Bank Marketing number in this course so far came from a random split. They were fine for what those days were studying, which was how training behaves. They were never an estimate of how this model would do next month.

"The data changed" is not a diagnosis. Two different things can change, and they break a model in different ways, so they have names.

• Label shift — the answer becomes more or less common. Here the yes rate goes from 6.7% in the training rows to 47.1% in the test rows. This is what breaks calibration.
• Covariate shift — the inputs change. Different kinds of customer are being phoned. This breaks the model in places it has barely seen.

Take them one at a time. Label shift is already measured, so start with the inputs.

For each column we want a single number saying "how different are these two periods?". Numeric and categorical columns need different measures.

For a numeric column: how far apart are the two averages, measured in standard deviations? Dividing by the spread makes ages and balances comparable.

For a categorical column: what share of rows would have to change category to make the two periods look the same? Add up the differences in each category's share and halve it; the halving is because every row that leaves one category arrives in another, so each difference gets counted twice.

Both, on toy numbers first.

Half the rows would have to change to line those two up. Now the same measure on every column, comparing the chronological training rows with the chronological test rows.

The three biggest movers are pdays, previous and poutcome. All three describe the same thing: whether this customer had been contacted in an earlier campaign.

In the early period about one customer in ten had been phoned before. In the late period it is nearly six in ten. The bank stopped cold-calling strangers and started calling back people it already knew, who are much likelier to say yes. That single change explains both the label shift and most of the covariate shift.

The table above can only see one column at a time. Shift can also hide in combinations: each column looks normal, but the mixture is different.

There is a neat trick for that, called adversarial validation. The idea:

1. forget the real label completely;
2. label every early row 0 and every late row 1;
3. train a model to tell them apart;
4. see how well it does.

If it cannot tell them apart, the two periods look alike and your split is safe. If it can, they are different, and the score tells you how different.

We need one more tool to do this honestly: cross-validation. Rather than a single split, cross_val_predict cuts the data into 5 parts, trains on 4 and predicts the 5th, five times over, so every row gets a prediction from a model that never saw it.

The last argument, class_weight="balanced", matters because the late period has far fewer rows than the early one; it tells the model to treat the two groups as equally important.

Remember what AUC means here. 0.5 would mean the two periods are indistinguishable. We get 0.9383, so a simple model can nearly always tell which period a customer came from.

Removing month and day barely changes it (0.9030), which rules out the easy explanation that this is just seasonality. The customers themselves are different.

This check is worth running before you trust any split, and again in production against live traffic, where it becomes a drift alarm. Module 9 and the GenAI module return to monitoring.

Two kinds of leak so far: through a row (the same row on both sides) and through time (the future mixed into the past). The third is quieter.

A group is a set of rows that belong together and are more alike than two random rows would be. Several patients from one hospital. Several photos of one person. Several reviews by one author. Several sentences from one document.

Group leakage happens when rows from the same group land on both sides of a split. The model can then do well on test rows by recognising the group, which will not be there in real use.

Bank Marketing has a natural group that is easy to miss: the calendar day. Calls made on the same day share a team, a script, an offer and the mood of the market.