Evaluating Adapted Models

"Worked" has several meanings, and a good evaluation checks each one:

Figure 8.7.1. Evaluation combines training behaviour, held-out metrics, errors, retention, costs, reproducibility and the application decision. A p-value alone does not settle suitability.

The fixed split is rebuilt with the same function as every other note. Then we load the three results files and check that each one used the same split and the same 1,000 training reviews. If not, comparing them would be meaningless, so the cell stops.

All three notes ran on the same data. Here is everything they measured, side by side:

Seven methods, all scored on the same 1,000 validation and 1,000 test reviews. The rest of this lesson works out what these numbers mean.

During training, two kinds of signal tell you whether things are going well. The training loss shows whether the model is fitting the data it learns from. A validation signal, validation loss or validation accuracy on data the model never trains on, shows whether that learning carries over to new data. Overfitting appears as the two pulling apart: training keeps improving while validation stalls or worsens (Lesson 8.1).

Lessons 8.4 and 8.5 saved the training loss at every step and the validation accuracy at each checkpoint:

Both runs started at a training loss near ln 2 (0.692 and 0.688 over the first 8 steps, the untrained-head value from Lesson 8.4) and ended at 0.436 and 0.440 over the last 8 steps. LoRA's loss fell faster early on, mostly because of its larger learning rate, and its first checkpoint was already far above chance, while full fine-tuning's was still at chance. Both peaked on validation at step 48 and dipped slightly at the end: a mild sign that one epoch was about right, and that checkpoint selection mattered.

A note on what we didn't record: validation loss. Validation accuracy only counts right and wrong; validation loss also shows whether the model is becoming over-confident on the reviews it gets wrong, often an earlier sign of overfitting. Recording it costs nothing extra during evaluation, and an experiment record should include it. That's a lesson for the next run, not something to invent now.

Key terms

• Training loss: the loss on the batches being trained on; shows whether the model fits its training data.
• Validation loss / accuracy: the same measures on held-out data; shows whether learning generalises.

Our task's metrics are accuracy, recall on each class, and F1 for the positive class (Lesson 8.2). Any accuracy measured on 1,000 reviews is uncertain, so we add a 95% confidence interval: a range that, by a standard calculation (the Wilson interval), is likely to contain the accuracy we'd see on unlimited reviews like these.

The intervals are roughly four to five points wide. Full fine-tuning and LoRA both score 0.834, with intervals of 0.810–0.856. The off-the-shelf SST-2 model scores 0.873 (0.851–0.892), and TF-IDF with all labels 0.913 (0.894–0.929). Overlapping intervals don't settle a comparison on their own; for that we use the paired sign test (Section 6), which is more sensitive because it compares methods review by review.

The recall columns show a difference the accuracies hide. Full fine-tuning catches negative reviews well (0.874) but misses more positive ones (0.796), while LoRA is balanced (0.829 and 0.839). If the feature's purpose were to catch unhappy customers, full fine-tuning's profile might even be preferable, despite equal accuracy. Metrics should match what the application needs (Lesson 7.5).

Key terms

• Confidence interval: a range likely to contain the true value of a measured quantity; here, accuracy on unlimited data.
• Wilson interval: a standard way to compute a confidence interval for a proportion.

Metrics count errors; error analysis looks at them. First, how much do methods agree on which reviews are hard? The table counts test reviews that both the row's and the column's method got wrong. The diagonal shows each method's own number of errors.

Full fine-tuning and LoRA each get 166 reviews wrong, and 112 of those are shared. They're the same model architecture trained on the same data, so they fail in similar places. LoRA also shares 106 errors with the frozen encoder: all three use the same pretrained DistilBERT representation, which seems to struggle with the same reviews. The SST-2 model and TF-IDF with all labels share far fewer errors with the others: they fail differently.

Of the 112 reviews both fine-tuned models got wrong, the SST-2 model got 46 right, so those are worth reading. Only 12 reviews defeat all six methods, perhaps hard or even mislabelled ones. Splitting by length shows something else. On the 121 test reviews that fit within 128 tokens, the DistilBERT-based methods all do well (0.876–0.917). On the 879 longer reviews, which head+tail truncation had to cut, every method that reads only 128 tokens drops, while TF-IDF with all labels, which reads every word, holds at 0.914. Part of the gap between the transformer methods and that TF-IDF model is simply that TF-IDF saw the whole review.

Here are the first three reviews that both fine-tuned models got wrong but the SST-2 model got right:

All three are positive reviews written with plenty of negative-sounding words. One calls the film "a bad, raunchy, predictable, tacky, salacious soap opera" and then a "guilty pleasure". One praises the film for a reason other than its quality. One opens by comparing it with gory horror films. The fine-tuned models, trained on just 1,000 reviews, seem to have learned that negative words mean a negative review. The SST-2 model, trained on many more labelled sentences, handles the reversal. That points to a concrete improvement: more training examples of exactly this kind of review.

Key terms

• Error analysis: examining which examples a model gets wrong, and why.
• Error overlap: how many errors two models share; low overlap means they fail differently, which also makes them good candidates to combine.

Common mistake: choosing error examples by hand because they "look interesting". Choose them by a rule (here, the first three that meet a condition), or you may be picking the examples that support a story you already believe.

Settings (regularisation, checkpoints) were chosen on validation, so each method's validation score is slightly optimistic, and the test score is the honest one. A large gap between them warns that a method was tuned to its validation data, a form of overfitting to the selection process.

All four methods that chose settings on validation score lower on test, by between 0.013 and 0.046. LoRA has the largest gap (0.880 → 0.834): its validation score looked best among the 1,000-label methods, but on test it is level with the rest. That is why final test reporting must remain separate from selecting settings on validation, and why a single validation result should never be trusted on its own (Section 7 shows how much results vary).

Interpret the evidence carefully. A validation–test gap alone cannot diagnose overfitting: sampling variation and selection on validation can also produce one. Examine training and held-out learning curves, and reserve a fresh final test after any further model selection.

The forgetting measure from Lesson 8.1 tells a clear story: full fine-tuning raised the word-guessing loss from 3.381 to 4.07, while LoRA, even merged, raised it only to 3.461, and with the adapter switched off it is unchanged. For this application we don't need word-guessing, so forgetting doesn't count against either model here. In applications where the adapted model must keep general abilities, it would weigh heavily in LoRA's favour.

Here is every method with what it used and what it cost. Training times were measured in each note on this machine and printed with units; compare them relative to each other, since they vary between machines and runs.

The costs span orders of magnitude:

• TF-IDF trains tens or hundreds of thousands of word weights in seconds.
• The frozen encoder trains 769 weights after one feature pass.
• LoRA trains 739,586 parameters, full fine-tuning all 66,955,010. Their estimated training memory is 280 MB and 1071 MB.
• The SST-2 model trains nothing on IMDb. Note, though, that it isn't free in an absolute sense: someone else fine-tuned it on a large labelled dataset, and we inherit that work.

The strongest method using no more than 1,000 IMDb labels is the SST-2 model off the shelf. We compare everything with it on the same test reviews:

Every 1,000-label method is significantly worse than the off-the-shelf model:

• full fine-tuning: 50 against 89 reviews, p = 0.0012;
• LoRA: 43 against 82, p = 0.00062;
• the frozen encoder and TF-IDF with 1,000 labels likewise.

Only TF-IDF trained on all 22,404 labels beats it (96 against 56, p = 0.0015). The picture in one chart:

The chart makes the lesson hard to miss. Training more parameters on the same 1,000 labels didn't buy accuracy: the frozen encoder (769 parameters), LoRA (739,586) and full fine-tuning (66,955,010) all land at about the same height. The two winners got their advantage from more labelled data: the SST-2 model indirectly, through someone else's large dataset, and TF-IDF directly, from all 22,404 IMDb labels.

Every adapted model here comes from a single training run. How much might a result change with a different random draw? We can't afford to retrain the Transformers many times on a CPU, but we can measure the variation for the cheapest rung: TF-IDF trained on five different random sets of 1,000 training reviews, scored on the same validation reviews.

Validation accuracy ranges from 0.840 to 0.859 across the five draws, about two points, purely from which 1,000 reviews were labelled. Fine-tuning runs also vary with initialisation, shuffling and dropout. So differences of one or two points between single runs, like LoRA's 0.880 against full fine-tuning's 0.860 on validation, shouldn't be read as real without repeated runs or a paired test. The paired test said there was no difference on test (Lesson 8.5).

Each note saved its settings, data fingerprints, costs and results to a file. Collecting them into one table, and saving it, is the beginning of experiment tracking: a permanent, comparable record of every run. Tools such as MLflow and Weights & Biases do this at scale, but the principle is the same.

The table lists every method with its note, settings, validation and test accuracy, and is saved as results/experiments.csv, alongside the environment (Python 3.12.15, PyTorch 2.14.1, Transformers 5.20.1, PEFT 0.21.2). With the fixed split, the fingerprints and the pinned versions in requirements.txt, anyone can rerun these lessons and check the numbers.

Key terms

• Run-to-run variation: differences in results caused by randomness in data sampling, initialisation and training order.
• Experiment tracking: recording each run's settings, data, code versions and results so that runs can be compared and reproduced.

A decision should follow rules written before looking at which model looks best. Otherwise it's easy to find reasons to keep the model you hoped would win. Ours, stated in the cell below before the decision is computed:

• R1: an adapted model must beat the strongest baseline that uses no more labels, on the same test reviews, with sign-test p < 0.05.
• R2: it must fit the deployment budget (CPU inference, one model per task). All candidates do.
• R3: it must not damage abilities we still need. Word-guessing isn't needed here.

Both adapted models fail R1: each is significantly worse than the off-the-shelf SST-2 model, not better. The decision is to reject both and, under a 1,000-label budget, use the SST-2 model as it is. If all 22,404 labels can be used, TF-IDF with all labels is better still.

Rejecting a model you spent effort on is not a failed project. It's the evaluation doing its job. The experiments answered a real question ("does adapting DistilBERT on 1,000 IMDb labels beat what we can get for free?") with a clear, defensible "no, not in this setup". And the error analysis showed what might change the answer.

Finally, we write the decision down, extending Lesson 7.7's decision record with everything this module measured:

The record states the question, the exact data (by fingerprint), every method's test accuracy, the decision with its alternative, the limitations that bound the conclusion, and next steps that could change it:

• Limitations: a single run per adapted model; only 1,000 reviews, one epoch and 128 tokens; a CPU-sized experiment with learning rates not tuned on validation.
• Next steps:
  • more labels, the clearest lever in our results;
  • longer inputs, since every 128-token method lost accuracy on long reviews;
  • repeated runs before claiming small differences;
  • a fresh test sample for any new winner, since this one has now been used to make a decision.

A reader of this record in six months can see what was concluded, how much to trust it, and what to try next.

Common mistake: writing up only the winning number. A useful record includes what was compared, on which data, what was rejected and why, what the conclusion depends on, and what would change it.

Let's reassemble the evaluation, and with it Module 8.

We loaded the results of seven methods, checked by fingerprint that all used the same split and the same 1,000 training reviews, and asked seven questions:

• Training behaved sensibly: the loss fell from about ln 2 to 0.436 and 0.440 in the two fine-tuning runs, and validation peaked at step 48 before dipping slightly.
• Task metrics with 95% intervals: both adapted models scored 0.834 (0.810–0.856), with different recall profiles.
• Error analysis: the two fine-tuned models shared 112 of their 166 errors, struggled on reviews longer than 128 tokens, and misread positive reviews full of negative words, which the SST-2 model handled.
• Overfitting and forgetting: every method lost a little between validation and test (LoRA most, 0.046), and full fine-tuning forgot more (word-guessing loss 4.07) than LoRA (3.461).
• Comparison: across the ladder, training more parameters on the same 1,000 labels didn't help, and every 1,000-label method was significantly worse than the off-the-shelf SST-2 model.
• Variation: TF-IDF varied by about two points across five draws of training data; results were tracked in results/experiments.csv.
• Decision: by rules set in advance, both adapted models were rejected: use the SST-2 model under a 1,000-label budget, or TF-IDF with all labels if they're available. The decision was recorded with its limitations and next steps.

Module 8 asked how to adapt pretrained models without updating every parameter, and how to prepare data for reliable adaptation.

• 8.1 mapped the adaptation spectrum.
• 8.2 prepared the data (and found that keeping the end of a review mattered).
• 8.3 built the baselines.
• 8.4 fine-tuned fully.
• 8.5 showed LoRA matching full fine-tuning with about 1% of the trainable parameters and less forgetting.
• 8.6 carried the same ideas to large language models.
• 8.7 showed how to decide, honestly, whether any of it was worth it.

The adapted models didn't win here, and knowing that, with evidence, is the outcome a careful engineer should want.

Figure 8.7.2. Keep/reject conclusions apply to the supplied one-epoch, 1,000-label experiment and declared requirements. Future tuning needs new validation work and a fresh final test sample.