The Bake-Off: Gradient Boosting against a Neural Network

The way model comparisons usually go wrong is not dishonesty. It is drift.

You run both models. One looks worse. You adjust it a little, rerun, it improves. You adjust it again. By the end, one model has had an afternoon of your attention and the other has had five minutes, and what you have measured is effort, not models.

The defence is to write the rules first, run them, and report what happened — including the parts you did not like. This is sometimes called pre-registering a comparison, and it is borrowed from clinical trials, where the same problem is much more serious.

Here are this comparison's rules.

The question. For this dataset, at this size, with this deployment story, which model would we ship?

Data and splits. Bank Marketing, duration dropped. Stratified 80/10/10, with split seeds 0 to 9. Both models see identical rows on every split.

The main measure. Test ROC-AUC, at the settings and epoch chosen on the validation rows. Second measure: test PR-AUC, always printed next to the share of positives that sets its floor (Lesson 4.1 Section 7).

Tuning budget. The neural network is frozen at the Module 3 settings: 50 inputs → 64 → 32 → 1, AdamW at 0.001, batch 256, 30 epochs, best epoch chosen on validation. Gradient boosting gets scikit-learn's defaults, plus one search over twelve settings, scored on the validation rows of split seed 42 only, then applied unchanged to all ten splits.

The decision rule. A difference counts as real only if it is bigger than the split-to-split noise from Lesson 4.2 and consistent in direction across splits. Accuracy alone does not decide: training time, prediction time, artifact size and behaviour under the Lesson 4.1 shift all count.

What would change our mind. If the two land within the noise on accuracy, the decision goes to the cost and robustness columns. If the ranking flips under the chronological split, then neither model is fit to deploy until the shift is dealt with.

One imbalance is worth stating in advance, because no table will show it. The network's settings are the product of two days of work across Lessons 2.3 to 3.4. Gradient boosting gets twelve settings and no architecture search at all. So if the network wins narrowly, that imbalance is a reason for caution. If it loses, the imbalance makes the loss more convincing.

Lesson 4.2 measured two kinds of wobble in this model's score. Changing the seed moved test AUC by a standard deviation of 0.0010. Changing the split moved it by 0.0130.

So a gap of 0.01 between two models measured on different splits means nothing. Measured on the same splits, it might mean a great deal. Here is why, on numbers we control, before we touch the real ones.

Imagine each split has its own difficulty, which both models feel equally — some test sets simply contain harder customers. On top of that each model has a small wobble of its own. Model B is genuinely better by 0.008.

The same ten pairs of numbers, read two ways. Summarised separately, the models are indistinguishable. Subtracted split by split, the advantage is obvious and consistent.

The reason is that the shared difficulty cancels. When both models are scored on the same test rows, part of each score is "how hard were these rows", and that part is identical for both. Subtract, and it disappears, leaving only the part that depends on the model.

This is called a paired comparison, and it is the single most useful trick in this lesson. Keep it in mind for Section 5, which has exactly this shape with real models.

Now the real thing. First the data and the training function — the same ones from Lessons 4.1 and 4.2, brought along so this notebook stands alone.

The training function is Lesson 4.2's, trimmed to what this comparison needs. It seeds everything, splits, fits the preprocessing on training rows only, trains for 30 epochs, keeps the best epoch by validation AUC, and then scores the test rows once.

Best epoch 18, validation AUC 0.7871, which is Lesson 4.2's number reproduced exactly. The test AUC beside it is the number this comparison will use.

Gradient boosting builds many small decision trees, one after another, where each new tree is trained to fix the errors the trees so far are still making. Predictions are the sum of all the trees. It is not a neural network, it has no layers or epochs, and on tabular data it is the model most practitioners reach for first.

We use HistGradientBoostingClassifier from scikit-learn, for two reasons. It comes with scikit-learn, so there is nothing extra to install on any platform. And it handles categorical columns natively, which matters for a fair comparison.

That last point is worth seeing. The network needs Lesson 4.1's pipeline: scale the numbers, one-hot the categories, 15 columns become 50 inputs. Gradient boosting takes the raw columns; you just mark which ones are categories, using pandas' category type.

Now the same shape of function as the network's, so both sides produce the same kind of result and can be compared without special cases.

A few of its settings, in plain words:

• learning_rate — how much each new tree is allowed to change the prediction;
• max_leaf_nodes — how many final branches each tree may have, which is its complexity;
• l2_regularization — a penalty that discourages extreme predictions;
• max_iter — the largest number of trees allowed;
• early_stopping="auto" — stop adding trees when a held-out slice of the training rows stops improving. With 45,211 rows this is on by default, which is why the number of trees actually used will be below max_iter.

Default settings, no preprocessing, under a second. It is already ahead of the network on test AUC, and we have not tuned it at all.

Now the declared budget: twelve combinations of three settings, scored on the validation rows of this one split.

Before going on, look at that last number. The entire search spans 0.0068 on validation, and Lesson 4.2 measured the split-to-split noise at 0.0130. The whole space we just searched is narrower than the wobble between splits. Section 6 comes back to what that means.

The rules say take the validation winner and carry it unchanged, so that is what happens next.

Now the comparison itself. Ten different splits, seeds 0 to 9. On each one, three models: the frozen network, gradient boosting with defaults, and gradient boosting with the tuned settings. All three see identical rows, because all three call make_split with the same seed.

This takes about two minutes.

This is the heart of the note, so read the two panels together with the numbers above them.

Summarised separately, the two models are indistinguishable. The network averages 0.7956 with a standard deviation of 0.0132; boosting averages 0.8034 with 0.0111. The gap between the averages, 0.0078, is smaller than either spread. If you reported one sentence per model, you would have to say the comparison is inconclusive.

Paired, the difference is +0.0078 with a standard deviation of 0.0047, and boosting wins all ten splits. The left panel shows why: the two lines move up and down together, because a split that is hard for one model is hard for the other. Subtracting removes that shared movement, exactly as in the made-up example in Section 2.

So the honest statement is: gradient boosting is better here, by a small and consistent margin of about 0.008 AUC. Not "much better". Not "indistinguishable".

And there is a general lesson underneath it, which Module 3 could not have shown you with single runs: a difference smaller than the noise floor can still be real, as long as you measure it in a way that cancels the noise.

The tuned settings won the search on validation. The rules said carry them to all ten splits, which we did. Now compare them with the untuned defaults, paired, exactly as before.

The tuning moved the paired result by +0.0012, with a standard deviation of 0.0024, winning five splits out of ten. Five out of ten is a coin toss. The twelve-setting budget produced nothing.

The two lines underneath show how the illusion forms. On the split the search was scored on, the tuned settings beat the defaults on validation, 0.7994 against 0.7944 — exactly the kind of gain that gets written up. On the test rows of that same split they were worse, 0.8109 against 0.8160.

Picking the best of twelve noisy numbers partly picks luck, and luck does not transfer to new rows.

This is the same lesson Lesson 3.2 reached from the other direction, where eight different regularisation settings turned out to be indistinguishable once early stopping was in place. Searching a space of settings costs real time. Before spending it, check whether the space you are searching is wider than your noise. Here it was not: the whole search spanned 0.0068 and the split noise is 0.0130.

Accuracy is one column of the decision. Here are the others, measured rather than assumed.

One caution about timing first, and it is the kind of thing that quietly makes comparisons unfair. Our network's timer covers 30 epochs including scoring the validation rows after each one, which is diagnostic work rather than training work. Gradient boosting's timer covers only its fit. To compare like with like, time the bare training loop as well.

Next, how long each one takes to make predictions, and how big each one is to ship.

Latency means the time to produce predictions, measured here for 1,000 raw customer rows, with the preprocessing included, because in production you have to do that too. We take the median of five attempts, because single timings are noisy.

Read that as a set of trades rather than a scoreboard.

Training. Boosting fits in 0.49 seconds against 6.80 for the network's 30 epochs — about 14 times faster on the same machine — with no epoch count to choose and no learning rate to get wrong.

Prediction. The network is the faster one here, 6.1 ms per 1,000 rows against 13.4, because scoring it is a few small matrix multiplications while boosting has to walk dozens of trees. Its files are also far smaller, 28 KB against 344 KB.

Everything else. The network needs a fitted preprocessor that has to be saved, versioned and kept in step with the weights — which is exactly what Lesson 4.2's bundle existed to manage. Boosting ships one file and takes the raw columns.

At this volume of requests none of the prediction numbers is a constraint, which is why the accuracy and training columns carry the decision. At a hundred thousand requests a second, or on a device with a hard size limit, the same table would point the other way.

Everything so far used the stratified split, which answers "how well on a random customer from 2008 to 2010?". Lesson 4.1 showed that the deployment question is different, and that the chronological split answers it: train on the earliest 80% of the file, test on the last 10%, where 47% of customers say yes.

Run both models there.

Both models fall a long way, and the ranking reverses. The network, which lost on every stratified split, is now the better of the two.

Before making anything of that, notice the problem: this is one test set, not ten. Time only runs in one direction, so there is no second chronological split to compare against.

The tool for that situation is the bootstrap. The idea is simple: treat your test set as a stand-in for the population, and draw a new test set from it with replacement — meaning the same row can be drawn twice, and some rows not at all. Score both models on that resample. Do it a thousand times, and the spread of the differences tells you how much your single result depends on which particular rows you happened to have.

The network is ahead by 0.0300, and the interval from +0.0172 to +0.0437 does not include zero, with the network ahead in 100% of the resamples. So the reversal is not an accident of which rows landed in that one test set.

Notice how much larger this gap is than the 0.0078 that separated the models on the stratified splits. The shift changed the ranking by roughly four times the margin the original comparison was arguing about.

The network's best epoch also moved, from 18 down to 2. Its validation rows now come from a later, different period, so the run reaches its best general performance almost immediately and then spends 28 more epochs fitting itself more tightly to a training period that no longer describes the world.

Why would the more flexible model suffer more? Because flexibility is fitted to the training distribution. Gradient boosting carves the old population into many fine regions, and when the population moves, a lot of those regions no longer describe anybody. The smaller, smoother model has less to be wrong about. That is a pattern worth carrying forward, not a fact about these two libraries.

And be careful what this does not say. It does not say the network is fit to deploy. Both models lose most of their skill, and both are badly miscalibrated on the later period, predicting far below the true rate of 47%. The honest conclusion is that the shift is the dominant problem, and it is bigger than the choice of model.

Two models with nearly the same AUC are not necessarily making the same predictions. On the stratified split, compare them customer by customer.

Spearman correlation measures whether two sets of scores put things in the same order, ignoring the actual values. 1.0 means identical order; 0 means no relationship.

The two models rank customers similarly but not identically: a Spearman correlation of 0.8658. Averaging their predictions gives 0.8163 against boosting's 0.8160, so combining them bought nothing worth shipping a second model for.

The slices are more useful. Boosting is ahead on three of the four, and its biggest advantage is on customers never contacted before (+0.0198), which is also the largest and hardest slice — its ability to carve up the raw categorical columns evidently helps there. On customers whose contact type is unknown, the two are level (−0.0036), and that slice is where both models are weakest in absolute terms: 0.6246 for the network and 0.6210 for boosting, against 0.75 to 0.86 everywhere else.

That slice was flagged in Lesson 4.1 as a suspicious 28.8% of rows with a low success rate, and Lesson 4.1 Section 8 found it shrinking sharply over time. Neither model solves it, and no amount of model choice will. That is a data question, and it is the kind of thing Module 9's error analysis is built to chase.

The blueprint asks for a decision rule, not a winner. Here is the memo, written from the measurements above, and then a short record for each side in the Lesson 4.2 format.

And the records, in the five-question format from Lesson 4.2. The function is repeated here so this notebook stands alone.

Three extensions, in increasing order of effort. Each one follows the same rules: declare the budget first, use the same splits, report the paired difference against the noise, and write the record.

1. A third model. Add logistic regression to the ten-split comparison. Lesson 3.1 reported 0.7717 for it on the Module 2 split. Does it survive a paired comparison against the other two, and what does its cost column look like?
2. The imbalance question. Run the network with pos_weight set to 7.5, roughly the ratio of no answers to yes answers, across the same ten splits. Lesson 4.1 Section 10 predicts the ranking will not change. Check it, and check the calibration too.
3. Combining them. Section 9 averaged the two models on one split. Does the average beat the better model across all ten, paired? If it does, is the gain worth shipping two models and maintaining both?