Establishing Baselines

A baseline does three jobs:

• It sets the bar. An adapted model is only worth its training cost, memory and complexity if it clearly beats the best simpler option.
• It catches mistakes. If a sophisticated model scores below a simple baseline, something is often wrong: a bug, leakage, a bad setting.
• It tells you where the gains come from. Comparing rungs that differ in one thing at a time (more labels, a pretrained encoder, adapting the encoder) shows which ingredient matters.

Our ladder has three rungs below fine-tuning, plus a do-nothing floor:

Fine-tuning in Lessons 8.4 and 8.5 uses the same 1,000 labelled reviews as rungs 1 and 3, the same validation reviews for every choice, and the same test reviews for the final number. That shared setup is what makes the comparison fair.

Figure 8.3.1. Baselines share a declared evaluation protocol. Off-the-shelf, frozen-feature and trained alternatives use different upstream training and sometimes different input information; disclose those differences.

All rungs use the fixed split (Lesson 7.1), with fingerprints, and the same subsets: 1,000 training reviews (the first 1,000 of the training split), the 1,000 val_eval reviews for every choice of setting, and the 1,000 test_eval reviews, used once per rung. We also fingerprint the 1,000 training reviews, so Lessons 8.4 and 8.5 can prove they train on the same ones.

We report four numbers for every rung: accuracy, recall for each class, and F1 for the positive class (Lesson 8.2). We start with the floor: always predict the most common label in the 1,000 training reviews.

The training subset has slightly more negative reviews, so the floor always says negative. It scores 0.505 on validation, chance level, with a recall of 1.0 for negatives and 0.0 for positives. Every rung must clear this comfortably.

TF-IDF turns each review into a long vector of word weights (Lesson 7.1). Here we use single words and word pairs (ngram_range=(1, 2)), keep only terms that appear in at least two reviews, and dampen very frequent words (sublinear_tf). Logistic regression then learns one weight per term. The vectoriser is fitted on the training reviews only, never on validation or test (Lesson 8.2, Section 7).

One setting needs choosing: C, which controls regularisation, how strongly large weights are discouraged. Smaller C means stronger regularisation. We try four values and choose by validation accuracy, then evaluate the chosen model once on test. We do this twice: with the 1,000 labelled reviews, and with all 22,404 as a reference for what plenty of labels can do.

With 1,000 labels, the best setting is C=3, with validation accuracy 0.866 and test accuracy 0.828. Note the gap between validation and test. Part of it is that we chose C on validation, so the chosen value is a little lucky there; part is ordinary sampling noise on 1,000 reviews. It's a reminder that the test number is the one to report.

With all 22,404 labels, TF-IDF reaches 0.921 on validation and 0.913 on test. Its vocabulary grows from 29,754 terms to 399,450, and the best C, 10, is the largest we tried, so even weaker regularisation might do slightly better. More data supports a more complex model. Fitting took a fraction of a second with 1,000 reviews, and a few seconds with all of them.

Key terms

• Baseline: a simple reference method, evaluated exactly like the method you hope will beat it.
• Regularisation: a penalty that discourages large weights, reducing overfitting; in scikit-learn's logistic regression, smaller C means stronger regularisation.
• n-grams: sequences of n consecutive words; (1, 2) means single words and word pairs.

The second rung uses no IMDb labels at all: the SST-2 sentiment model from Module 7, exactly as downloaded. It's the baseline every adaptation must clearly beat, because it costs nothing to train. We feed it the same head+tail 128-token inputs the fine-tuned models will use (Lesson 8.2).

The helpers below are shared with rung 3: head_tail builds the 128-token inputs, and batches groups reviews of similar length to cut padding (Lesson 7.4).

The off-the-shelf model scores 0.888 on validation and 0.873 on test, without seeing a single IMDb label. Its time per review is printed above; it depends on your processor, so compare it only with other timings from your own run. The bar for any adapted model trained on 1,000 IMDb reviews is therefore already high: someone else's adaptation, on a related task, is free.

The third rung is feature extraction (Lesson 8.1): DistilBERT's pretrained body stays frozen, mean-pooled into one 768-number vector per review, and a logistic-regression head is trained on the same 1,000 labelled reviews. This is exactly a fine-tuned model minus the fine-tuning: same encoder, same inputs, same labels, but the encoder never changes. Comparing it with Lessons 8.4 and 8.5 will show what adapting the encoder itself adds.

Each review becomes a vector of 768 features. Computing them for the training reviews takes a few seconds, and that one pass is the only use of the Transformer during training. Now we choose the head's regularisation on validation:

The best setting, C=1, gives 0.850 on validation and 0.837 on test. The trained part is tiny: 769 parameters (768 weights and a bias). Everything else comes from pretraining.

Key terms

• Frozen encoder: a pretrained model used only to compute features, never updated.
• Linear head (linear probe): a single linear layer, here logistic regression, trained on frozen features.

Here is every rung on the same 1,000 validation and 1,000 test reviews:

Reading the ladder, from the test column:

• Every rung clears the floor (0.485) by a wide margin, and all have balanced recall on the two classes, so none is succeeding by favouring one label.
• Rung 2, the off-the-shelf SST-2 model, is the strongest method that uses 1,000 labels or fewer, at 0.873 with no IMDb labels.
• With 1,000 labels, TF-IDF (0.828) and the frozen encoder (0.837) are close, and both below the off-the-shelf model.
• TF-IDF with all 22,404 labels is the strongest of all (0.913). Plenty of labels make even a simple model hard to beat.

Differences of a few points on 1,000 reviews can be noise, so we test the important ones with the sign test on paired predictions: the same test reviews, comparing which model gets each one right (Module 5). With 1,000 test reviews, one review is worth 0.1 percentage points.

• The off-the-shelf model beats TF-IDF with 1,000 labels: it is right on 126 reviews where TF-IDF is wrong, and wrong on only 81 where TF-IDF is right (p = 0.0022).
• The frozen encoder and TF-IDF, both with 1,000 labels, are not reliably different (98 against 89, p = 0.56).
• TF-IDF with all labels beats the off-the-shelf model (96 against 56, p = 0.0015).

So the honest summary of the baselines: with 1,000 labels, the bar to beat is the off-the-shelf model at 0.873, and with all labels, TF-IDF at 0.913.

Everything Lessons 8.4–8.7 compare will follow the same protocol:

• Fixed data: the same split, checked by fingerprint, and the same 1,000 training, 1,000 validation and 1,000 test reviews.
• Choices on validation only: regularisation strength, learning rates, epochs, checkpoints.
• Test once per method, after all choices are made.
• The same metrics: accuracy, per-class recall and F1, plus paired sign tests for comparisons.
• Costs recorded alongside quality: labels used, parameters trained, and training and inference time (timings printed with units; they vary between machines and runs, so compare them within one run, not across machines).

To let Lesson 8.7 compare every rung on the same test reviews without retraining anything, we save the results, including each rung's prediction for every test review, to a file. Lessons 8.4 and 8.5 will save theirs the same way.

The file records the split fingerprints, the training-subset fingerprint and, for each rung, its metrics, settings, costs and test predictions. If Lesson 8.7 finds a fingerprint that doesn't match, it will refuse to compare, rather than compare different reviews by mistake.

Key terms

• Evaluation protocol: the fixed rules for how every method is measured: data, metrics, when the test set is used, and how results are compared.
• Paired comparison: comparing two methods on the same examples, example by example.
• Results file: a saved record of a method's settings, costs, metrics and predictions, for later comparison.

Common mistake: comparing your fine-tuned model with a weak baseline, or with one evaluated on different data. Beating "always negative" or a poorly tuned TF-IDF proves little. Tune each baseline on validation as carefully as the model you hope will beat it, and evaluate everything on the same reviews.

Let's reassemble the baseline ladder.

A baseline sets the bar an adapted model must clear, catches mistakes, and shows where gains come from. Our ladder, on the same fixed reviews with fingerprints, tuned on 1,000 validation reviews and tested once on 1,000 test reviews:

Paired sign tests showed that the off-the-shelf model beats both 1,000-label baselines (p = 0.0022 against TF-IDF), that the frozen encoder and TF-IDF are not significantly different in this paired comparison at 1,000 labels (p = 0.56), and that TF-IDF with all labels beats everything (p = 0.0015 against the off-the-shelf model). The protocol (fixed data, choices on validation, test once, the same metrics, costs recorded) is saved with every result in results/8.3.json.

For fine-tuning on 1,000 reviews in Lessons 8.4 and 8.5, the bar is clear: beat 0.873, the free off-the-shelf model, and know that 0.913 is what plenty of labels and a simple model achieve.

Figure 8.3.2. Source test accuracy on 1,000 reviews, with training-label budgets shown. All-label TF-IDF is a separate reference budget; it reads the full text rather than 128 Transformer tokens.