Understanding Model Adaptation

A pretrained model is a generalist. DistilBERT knows English well, from guessing hidden words in books and Wikipedia (Lesson 7.1). A small chat model knows how to follow instructions. Neither knows what our product team means by a negative review, which reviews matter, or how our users write. Adaptation closes that gap, and the main ways to do it form a spectrum, ordered by how much of the model they change:

Moving down the table, adaptation becomes more powerful, but also costs more data and compute, and risks damaging what the model already knew. The rest of this lesson tries each approach on the same task and data, so the trade-offs are measured, not just described.

Figure 8.1.1. Prompting changes the input; a frozen encoder supports a trained head; LoRA changes selected adapters; full fine-tuning makes the selected base parameters trainable. More complexity need not improve quality.

Everything in Module 8 uses the fixed IMDb split from Lesson 7.1. The function is copied here, and the fingerprints confirm the same reviews.

Many reviews are longer than the 128 tokens we'll feed DistilBERT in this module. Rather than keeping only the start of each review, the helper below keeps the beginning and the end, where reviewers often give their verdict. Lesson 8.2 explains this choice and measures how much it helps.

We'll compare the approaches on the first 100 validation reviews, where every review is worth one percentage point of accuracy.

The cheapest adaptation changes no weights at all. With an instruction-following model, you write the task as a question, and the model answers. We use SmolLM2-360M-Instruct, a small open chat model with about 360 million parameters, loaded in float32 because its stored bfloat16 format is very slow on most CPUs (Lesson 7.6). Its chat template (Lesson 8.6 looks inside) wraps our question in the special markers the model was trained with:

The template added a system message we didn't write ("You are a helpful AI assistant named SmolLM…"). That's this model's default. Then comes our question, and finally the start of the assistant's turn, where the model should write its answer. Let's ask it about a real review:

It answers "Negative", correctly. To score 100 reviews quickly and fairly, we don't generate free text. Instead we compare the model's scores for the two possible answer words at the first position of its reply, and take whichever is higher. This avoids having to parse answers like "Negative." or "It is negative".

With no training and no labels at all, prompting gets 0.84 of the 100 reviews right, and it says "positive" for 0.49 of them, about the right balance. That's a strong start for zero effort. The costs are elsewhere: every prediction runs a 360-million-parameter model over a long prompt, and the only way to improve it is to rephrase the prompt, which is fragile.

Feature extraction keeps a pretrained model completely frozen, meaning its weights are never changed, and uses it only to turn each review into a vector of numbers, its features. A small, fast model, here logistic regression, then learns to map those vectors to labels. This is the encoder-plus-linear-probe idea from Lesson 5.1, with a pretrained encoder. We use DistilBERT's body, average its token vectors (mean pooling, Lesson 7.5), and train on 1,000 labelled reviews.

With 1,000 labels, feature extraction reaches 0.844 on 500 validation reviews, and 0.86 on the first 100, the same reviews prompting saw. On those 100 reviews the two approaches are within two reviews of each other. Neither one changed a single weight of a pretrained model.

The last two approaches do change the pretrained model, by training it further on labelled examples. That's fine-tuning (Lesson 7.1, Section 2).

• Full fine-tuning updates every weight. It's the most flexible, but it needs the most memory and compute, and every task gets its own complete copy of the model. Lesson 8.4 runs it properly.
• Parameter-efficient fine-tuning (PEFT) freezes the pretrained weights and trains a small number of extra ones. The most popular method, LoRA, adds small "correction" matrices beside some of the layers. Often around 1% of the parameters are trained, which makes training cheaper, keeps the original model intact, and lets one base model serve many tasks. Lesson 8.5 builds it.

We'll see full fine-tuning at small scale in Section 4.

Key terms

• Adaptation: making a general pretrained model better at a specific task.
• Prompting: adapting a model only by describing the task in its input; no weights change.
• Chat template: the format of special markers an instruction-tuned model expects around system, user and assistant messages.
• Feature extraction: using a frozen pretrained model to produce features, and training a separate small model on them.
• Frozen: describes weights that are not updated during training.
• Fine-tuning: further training of a pretrained model's own weights on labelled task data.
• PEFT (parameter-efficient fine-tuning): fine-tuning that trains only a small number of new or selected weights while the rest stay frozen.
• LoRA (low-rank adaptation): a PEFT method that adds small trainable matrices to some layers (Lesson 8.5).

Fine-tuning is powerful, but it's often not the first thing to try. Before you start, it helps to ask these questions in order, from cheapest to most expensive:

1. Does a ready-made model already do the task? In Lesson 7.7, a downloadable sentiment model met our requirement without any training. If one exists with a suitable licence, measure it first (Lesson 8.3 makes it a baseline).
2. Can prompting do it? For a capable instruction model, a clear prompt, perhaps with a few examples in it, may be enough, as Section 2.1 suggests.
3. Is a simple model on frozen features enough? Feature extraction is cheap to train and easy to retrain when labels change.
4. Only then: fine-tune. It is worth it when you have enough good labelled data, when the task or its data differ from anything the model saw (special vocabulary, formats or labels), or when you need a small, fast model to do well what only a large, slow one can do by prompting.

Fine-tuning is the wrong tool when:

• you have very few labels. A few dozen examples can't adjust millions of weights reliably. Section 5 shows how quickly a small training set leads to overfitting.
• the task changes often. Each change means new data and a new training run. A prompt or a retrained simple classifier adapts faster.
• what's missing is knowledge, not behaviour. Fine-tuning shapes how a model responds, but it is a poor way to teach it new or changing facts, which it may then recall wrongly (Lesson 7.1, Section 3). Feeding the relevant documents into the prompt usually works better.
• a simple baseline already meets the requirement. If TF-IDF or an off-the-shelf model is good enough, the extra cost and risk of fine-tuning buy nothing.

Common mistake: jumping straight to fine-tuning because it feels like the "real" machine-learning solution. Start from the cheapest approach that could work, measure it, and move along the spectrum only when the measurements say you must.

Let's watch full fine-tuning happen at small scale. We take DistilBERT's pretrained body, add a new, randomly initialised classification head (Lesson 7.1, Section 4, hence the LOAD REPORT), and train everything on a deliberately small set of 160 reviews for three passes (epochs). After each epoch we check accuracy on the training reviews themselves and on 200 validation reviews. The loop is the familiar one from Module 2: forward pass, loss, backward pass, optimiser step. Lesson 8.4 takes it apart.

Three things happen.

• Epoch 1 learns almost nothing useful. The training loss barely moves (0.697), and the model calls every validation review positive (1.00), which scores 0.525, the share of positive reviews among them. The new head starts random, and 10 small steps are not enough.
• Epoch 2 is where it learns the task. Accuracy jumps to 0.925 on the training reviews and 0.850 on validation, from only 160 labelled examples. The pretrained body already understands language; training mostly has to connect that understanding to the new head.
• Epoch 3 keeps improving on the training reviews (0.969) while validation drops to 0.800, with the model now leaning positive (0.69). Section 5 returns to this.

How much did the pretrained weights move? We compare the body after training with the body we started from, layer by layer, using the relative change from Lesson 7.1, Section 7:

Every one of the 66,955,010 parameters was trainable, including 592,130 in the new head, and every part of the body moved, but only slightly: between 0.395% and 0.624% on average. The last Transformer layer, nearest the new head, changed most. Full fine-tuning is gentle nudging of a model that already works, not rebuilding it. Even so, those small nudges have consequences, which we measure next.

Key terms

• Epoch: one pass through all the training examples.
• Classification head: the new layer(s) on top of a pretrained body that map its features to the task's labels; it starts random.
• Relative change: the size of the change in a weight tensor divided by the size of the original.

Two things can go wrong when you fine-tune, and they're easy to confuse: the model can fit its small training set too closely, and it can lose abilities it had before.

You met overfitting in Module 3: a model learns the quirks of its training examples instead of the general pattern, so it keeps improving on them while getting worse on new data. Here are the numbers from Section 4 as a picture:

Through epoch 2, both curves rise together. In epoch 3 the training loss falls sharply and training accuracy keeps climbing, but validation accuracy drops from 0.850 to 0.800, and the gap between the two widens. With only 160 examples, the model has started to memorise them. One caution: the drop is 10 of 200 validation reviews in a single training run, so treat it as a warning sign rather than proof. But it is exactly the pattern to watch for, and it's why the fine-tuning notes choose their checkpoint on validation (Lesson 8.4).

Overfitting is about the task we're training on. Catastrophic forgetting is about everything else: when a model's weights are adjusted for a new task, it can lose some of what it could do before. DistilBERT was pretrained to guess hidden words. Does our fine-tuned body still do that as well?

To find out, we hide 15% of the words in 200 validation reviews used nowhere else, and measure how well the original word-guessing head does. First on top of the pretrained body, then with the fine-tuned body swapped in. The loss is the cross-entropy of its guesses; lower is better.

With the pretrained body, the word-guessing loss on the 3557 hidden words is 3.381. With the fine-tuned body, it rises to 3.920. Bodies that moved by less than 1% on average have measurably lost some of their original skill, after training on just 160 reviews. That's catastrophic forgetting in miniature. It matters when you want the adapted model to keep doing other things too, and it's one reason PEFT methods, which leave the original weights untouched, are attractive. Lesson 8.5 measures how much less LoRA forgets.

Figure 8.1.2. Source small-run measurements illustrate an overfitting warning and degraded masked-word retention. The retention experiment uses a tied output projection and is not a complete capability audit.

Key terms

• Overfitting: improving on the training examples while getting worse on new data.
• Catastrophic forgetting: losing previously learned abilities when a model is trained on a new task.
• Masked-word (word-guessing) loss: the cross-entropy of a masked language model's guesses for hidden words; used here as a measure of the pretrained skill.

Common mistake: checking only the new task after fine-tuning. A model can score well on the task you trained it for and still have lost abilities your application relies on. If other behaviours matter, measure them before and after.

Interpret the evidence carefully. This is one masked-word retention probe on 200 IMDb reviews, not a complete test of general knowledge. DistilBERT ties its vocabulary projection to its input embeddings: loading a changed body also changes those shared output weights. The other masked-word head layers remain original. This setup measures the resulting combined system; it is not a strictly fixed-decoder experiment.

Putting the measurements together gives a practical way to choose. Three resources trade against each other:

• Data. Prompting needs no labels. Feature extraction got 0.844 from 1,000. Full fine-tuning learned the task from 160 examples but immediately began to overfit. More good labels make the heavier methods safer and stronger. Lesson 8.2 is about making those labels good.
• Compute. Prompting costs nothing to set up but runs a large model on every request. Feature extraction costs one pass to compute features and a quick fit. Fine-tuning costs real training time and memory. Full fine-tuning needs memory for every weight's gradient and optimiser state (Lesson 8.5 counts them), PEFT far less.
• Quality and risk. The heavier methods can fit a task most closely, but they risk overfitting on small data and forgetting old skills, and each fully fine-tuned task needs its own complete copy of the model.

A reasonable default path, which the rest of Module 8 follows:

1. establish baselines: a simple model, an off-the-shelf model and frozen features (Lesson 8.3);
2. try PEFT (Lesson 8.5) and full fine-tuning (Lesson 8.4) on the same data, with checkpoints chosen on validation;
3. evaluate everything on the same test reviews, including forgetting, and keep an adapted model only if it beats the baselines by enough to justify its costs (Lesson 8.7).

Let's reassemble the picture.

Adaptation turns a general pretrained model into a task-specific one, and the options form a spectrum by how much of the model they change. Prompting changes nothing but the input: SmolLM2-360M answered 0.84 of 100 reviews correctly with no training and no labels. Feature extraction freezes the model and trains a small classifier on its outputs: 0.844 on 500 reviews from 1,000 labels. PEFT trains a small set of added weights, and full fine-tuning trains them all.

Watching full fine-tuning at small scale showed both its power and its risks. From only 160 reviews, the model went from predicting "positive" for everything to 0.850 validation accuracy in two epochs, while its pretrained weights moved by less than 1% on average. But a third epoch showed overfitting (training accuracy 0.969, validation down to 0.800), and the fine-tuned body had suffered degradation on this retention probe: its word-guessing loss rose from 3.381 to 3.920.

So the choice of strategy is a trade-off between data, compute, quality and risk, made with measurements: start from the cheapest approach that could work, set baselines, and only move along the spectrum when the evidence says it's worth it.

Figure 8.1.3. Measure simpler options before adding training cost. The order is a decision aid, not a guaranteed quality ranking or a claim that every task requires an LLM prompt.