Understanding Pretrained Models

Let's start with the whole thing working, then take it apart. The cell below downloads a ready-made sentiment model from the Hugging Face Hub, a public website where people share trained models (Lesson 7.2 tours it properly). The model is called DistilBERT, a compact member of the BERT family of Transformer encoders. The version we load has already been adapted to judge whether a sentence about a film is positive or negative.

The pipeline function wraps the whole process, from text in to a label out, in one object. Note the device=device argument: without it, the library may pick a device on its own, and your numbers might not match these lessons.

Now let's give it four short reviews. I wrote these before running the model, so they weren't chosen to make it look good.

Three are right, with a confidence score of 1.000. The fourth, "Two hours of my life I will never get back", is a complaint, but the model calls it positive with a score of 0.745. There's no unhappy word in the sentence; the complaint lives in the idiom. Keep this in mind: a working model is not a perfect one, and Lesson 7.5 tests this model much harder.

We did no training at all. Every weight in that model was set by someone else, on someone else's computer, before we arrived. That is what a pretrained model is: a model whose weights have already been learned, published so that others can use them without training from the start.

Four sentences prove nothing, so we need a proper test. We'll use the IMDb movie-review dataset: reviews from the Internet Movie Database, each labelled positive or negative, with an official training half and test half. The same split will be used in every note of Modules 7 and 8, so that results from different notes can be compared fairly.

The function below builds that split. It does three things, for reasons you met in Module 4:

• It removes duplicate reviews before splitting. If a review appears twice, one copy could land in training and the other in evaluation, and the model would be tested on something it has already seen.
• It also removes test reviews that appear in the training half. That's the same leak, across the official split.
• It carves a validation set out of the training half, with a fixed seed. Settings are chosen on validation and reported once on test.

It also cleans one quirk: the reviews contain HTML line-break tags (<br />), which we replace with spaces. Finally, fingerprint turns a set of texts into a short code. If two notebooks print the same fingerprint, they are using exactly the same reviews. That gets checked rather than assumed.

The first run downloads the dataset into a data/imdb folder next to this notebook. You'll see "Generating train split" progress lines.

Each official half has 25,000 reviews, and both contained duplicates: 96 were removed from train and 322 from test. That leaves 22,404 reviews for training, 2,500 for validation and 24,678 for testing, all close to half positive. Running a Transformer over 24,678 reviews on a CPU takes a long time, so most evaluations in these lessons use two fixed samples of 1,000 reviews: val_eval (the first 1,000 validation reviews) and test_eval (a seeded sample of the test set). Their fingerprints will appear again in later notes.

Common mistake: trusting an official split to be clean. IMDb's halves share some reviews, so a model trained on one half has seen part of the other. Always deduplicate before splitting, and check across any split you are given.

Here is the experiment that shows where the value lies. We'll score the pretrained model on the first 500 validation reviews. Then we build a model with the identical architecture, the same layers and the same number of weights, but with random weights, and score it on the same 500 reviews.

The pipeline cuts each review to its first 256 tokens (max_length=256). As you saw in Lesson 6.2, a Transformer can only look at a limited number of tokens at once, and many reviews are longer than that. Lesson 7.4 explains truncation in detail.

To build the untrained twin, we load only the model's configuration, the description of its architecture without any weights (Section 6 looks inside one), and ask the library to build a fresh model from it.

The difference is stark. The pretrained model scores 0.874 on 500 reviews held out from our own training. Its untrained twin scores 0.486, which is chance: it calls almost every review negative (only 8.4% positive) and is right only when the review happens to be negative. Nothing about the code differs. Same architecture, same tokeniser, same pipeline. With architecture and tokeniser held fixed, the trained weights make the difference in this experiment.

Figure 7.1.1. The architecture and tokenizer are held fixed. The saved CPU run scores 0.486 with random weights and 0.874 with pretrained/adapted weights on the same 500 validation reviews; this does not audit all upstream exposure.

Key terms

• Pretrained model: a model whose weights were learned by someone else, usually on a very large dataset, and published for others to use or adapt.
• Weights (parameters): the learned numbers inside a model. Training means adjusting them.
• Architecture: the structure of a model, meaning which layers it has, their sizes and how they connect. Two models can share an architecture and behave completely differently.
• Configuration: a small file that describes an architecture in words and numbers, so the library can rebuild it.
• Pipeline: a Hugging Face helper that bundles tokenising, running the model and turning its output into a readable answer.

The next section steps back to ask where a model like this comes from.

Interpret the evidence carefully. The comparison fixes architecture, preprocessing and tokeniser; it isolates the benefit of trained weights in that setup. These reviews are held out from our own training. Upstream documentation is evidence about earlier training data, not an exhaustive proof that a public checkpoint never encountered related material.

The sentiment model we loaded did not begin life as a sentiment model. Its story has two stages, and almost every pretrained model you'll meet follows the same pattern.

Stage one: pretraining. A large model is trained on a huge collection of general data, such as books and web pages for text. Nobody labels this data. Instead, the model is given a task that the data can grade by itself. In Lesson 6.6 your Transformer predicted the next character. DistilBERT was pretrained on a close cousin of that task, masked language modelling: words in a sentence are hidden behind a [MASK] token, and the model must guess them from the words on both sides. Guessing hidden words well, across billions of sentences, forces a model to learn a lot about how language works. The result of this stage is a base model: general-purpose, but not yet built for any particular job.

Stage two: downstream use. A downstream task is the job you actually care about, such as sentiment, spam detection or question answering. "Downstream" because it comes after pretraining. The base model is adapted to it, usually by adding a small new output layer and training a little more on labelled examples. That extra training is called fine-tuning. Our sentiment model is DistilBERT fine-tuned on SST-2, a dataset of short sentences from film reviews labelled positive or negative.

We can watch the base model do its own pretraining task. The fill-mask pipeline hides nothing itself; we write [MASK] where we want a guess.

All five guesses are positive adjectives that fit the sentence: "fantastic" with probability 0.110, then "fabulous", "successful" and so on. The base model was never told about sentiment, but filling in blanks taught it that "absolutely" before a blank about a movie is usually followed by strong praise.

Now let's compare the two models' parts. In Lesson 5.1 we split networks into an encoder (often called the body) that builds a representation, and a head that uses it. Hugging Face names the body's weights distilbert.…, so we can count them separately.

The bodies are exactly the same size, 66,362,880 parameters each. Only the heads differ. The base model's head (622,650 parameters as counted here) turns each position's vector back into a score for every word in the vocabulary. It doesn't need its own copy of the big word table: the last line shows it reuses the body's word-embedding table, a common trick called weight tying, so the shared table is counted only once, in the body. The SST-2 head (592,130 parameters) turns the whole text into just two scores.

The output shapes make the difference concrete:

For 1 sentence of 8 tokens, the base model returns 30522 scores at every position: one per vocabulary word, a guess for what could stand there. The SST-2 model returns 2 numbers for the whole sentence, one per class. Same body, different head, different job.

Figure 7.1.2. Pretraining produces reusable weights. Inference can use a checkpoint directly; optional adaptation changes selected weights for a downstream objective.

Key terms

• Pretraining: the first, expensive training stage on large, general, usually unlabelled data.
• Masked language modelling: a pretraining task in which some words are hidden and the model must predict them from the surrounding words on both sides.
• Base model: the result of pretraining, before any task-specific adaptation.
• Downstream task: the task you actually want to solve with a pretrained model.
• Fine-tuning: further training of a pretrained model on labelled data for a downstream task.
• Body and head: the body (encoder) builds representations and is shared across tasks; the head is a small task-specific layer on top.

So pretraining on blanks produced something useful. The next section asks what, exactly, the model picked up.

Masked-word guessing is a convenient window into a base model, because we can design blanks that only make sense if the model has learned something specific. Here are four probes. Each tests a different kind of knowledge: grammar, a fact about the world, everyday cause and effect, and who typically does what.

The results are uneven, and that is the lesson.

• Grammar is strong. "She [MASK] to the cinema … yesterday" gets "went" with probability 0.62. The model has learned that "yesterday" calls for a past tense and that "to the cinema" calls for a verb of movement.
• Everyday cause and effect is strong. "So boring that I fell [MASK]" gets "asleep" at 0.71.
• Facts are shaky. "The capital of France is [MASK]" gets "marseille", "nantes" and "toulouse", all French cities, but not the right one, and with low confidence (0.14 for the top guess). The model learned what kind of word fits, but not the fact.
• Typical agents are only partly there. "The [MASK] barked" does get "dog" first, but with only 0.06, barely ahead of "clerk" and "sheriff".

Pretraining teaches a model the patterns of its training text: grammar, typical word combinations and common associations. It may also absorb facts, but less reliably, and a small model like DistilBERT holds fewer of them. A pretrained language model is not a database. Treat any fact it produces as a guess to check.

The deeper thing pretraining builds is the representation itself. In Lesson 5.3 we met static embeddings, where each word gets one fixed vector, and saw their limit: "bank" gets the same vector whether it is a river bank or a savings bank. A Transformer builds contextual representations instead: after each layer, a word's vector has been mixed with information from the words around it.

Let's measure that. We take the base model's body, without any head, and feed it six sentences: three use "bank" in the money sense and three in the river sense. For each sentence we collect the vector for "bank" at the input (layer 0) and after each of the six Transformer layers.

Each sentence gives 7 vectors of 768 numbers each. Now we compare "bank" with "bank" using cosine similarity (Lesson 5.3): averaged over all pairs that share a meaning, and over all pairs that don't, at every layer.

At layer 0 there is no gap at all. The two groups are equally similar (0.952 and 0.961), because at the input "bank" is mostly just its embedding-table entry plus a position, the same as a static embedding. After the first layer the meanings pull apart, and by the last layer the gap is 0.237: money-sense "bank"s average 0.880 with each other, but only 0.642 with river-sense ones. The plot shows the whole picture.

Nobody told the model that "bank" has two meanings. It worked that out because guessing hidden words well requires it: the words around "bank" decide what can fill nearby blanks. This is why pretrained representations are so useful downstream. The hard work of turning text into meaningful vectors has already been done.

These are six hand-written sentences, so treat the exact values as an illustration. The pattern of a gap that is absent at the input and opens up inside the network is the general behaviour of contextual models.

Key terms

• Probing: testing what a model has learned by giving it inputs designed to need a specific piece of knowledge.
• Contextual representation (contextual embedding): a vector for a word that depends on the sentence around it; the same word gets different vectors in different contexts.
• Hidden states: the vectors a Transformer produces after each layer; hidden_states[0] is the input embedding, and the last is the final output of the body.

Common mistake: asking a pretrained language model for facts and trusting the answer. Its training rewards text that sounds right, so a fluent wrong answer, like a French city that isn't the capital, is a normal outcome rather than a rare glitch.

Because these representations are general, one pretrained body can serve many tasks. The next section shows that directly.

A model pretrained on broad data and then adapted to many different tasks is often called a foundation model. The name says what it is for: you build on it. Large language models used in chat assistants, image–text models like CLIP (which you met in Lesson 5.5 and will use for real in Lesson 7.5) and the encoder we are using here all play that role, at very different scales.

We can show the idea in miniature. The library has a different model class for each kind of head. Let's load the same base checkpoint four times, each time asking for a different head.

The library printed a LOAD REPORT for each model. This is not an error, and you will see these tables often, so it's worth reading one carefully:

• UNEXPECTED lists weights that are in the downloaded file but not needed by the model you asked for. Here they are the masked-word head (vocab_transform, vocab_projector and so on), which none of the other three tasks use, so they are skipped.
• MISSING lists weights the model needs but the file doesn't contain. Here they are the new heads (classifier, qa_outputs). The library creates them with random values, and the note tells you to train them on your task.

Now let's check what the four models have in common:

All four start from the same pretrained body, down to the last number (we compared one weight matrix exactly with torch.equal). Only the small heads differ: the masked-word head we've already met, a two-layer pre_classifier + classifier for whole-text classification, a qa_outputs layer that marks where an answer starts and ends in a passage, and a classifier that labels each token separately (for example, as a person's name, a place, or neither).

The heads on three of them are random, so those models can't do their tasks yet. That's the point: the expensive part, the body, is shared, and only a small head has to be trained for each new task. Module 8 does that training.

Figure 7.1.3. A pretrained body can support different output interfaces. Newly initialised heads require training, and task adaptation can update the body as well as the head.

Key terms

• Foundation model: a model pretrained on broad data and designed to be adapted to many downstream tasks.
• Transfer learning: reusing what a model learned on one task or dataset to help with another.
• LOAD REPORT: the table Hugging Face prints when a checkpoint's weights don't exactly match the model class requested. UNEXPECTED weights are skipped; MISSING weights are created at random.

A shared body is the promise. The next section tests how much it is actually worth, by comparing it with the alternatives.

Suppose you need a sentiment classifier for film reviews. You have three broad options:

• Traditional machine learning: turn each review into word counts and fit a linear model. The classic version is TF-IDF + logistic regression. TF-IDF scores each word by how often it appears in a review, down-weighted if it appears in almost every review (like "the").
• Training a neural network from scratch: start from random weights and learn everything from your own labelled data. We'll use the small bag-of-embeddings network from Lesson 5.3. A full Transformer trained from scratch would need far more data and computing than we have.
• Using a pretrained model: here, the SST-2 model off the shelf, with no IMDb training at all.

The fair way to compare them is to vary how many labelled IMDb reviews the first two get: 100, 1,000, or all 22,404. Small samples are noisy, so for 100 and 1,000 we repeat each experiment on five different random draws of training reviews and report the spread. Everything is scored on the same 500 validation reviews as in Section 1.

One more detail keeps the comparison honest. The neural network needs a setting chosen: how many epochs to train. We choose it on a different slice of validation (reviews 1,000 to 2,000), never on the 500 we report.

The small network is the one from Lesson 5.3, redefined here so this notebook stands alone. Its vocabulary is built only from the reviews it is given.

Now we run all of the combinations. The 100-label and 1,000-label rows each train five models, one per draw.

Read the table from top to bottom:

• With 100 labels, both methods are weak and unstable. TF-IDF averages 0.627 but ranges from 0.508 to 0.748 depending on which 100 reviews it got. The network from scratch is barely above chance (0.541).
• With 1,000 labels, both improve a lot (0.826 and 0.798) but are still below the pretrained model.
• With all 22,404 labels, both beat the off-the-shelf pretrained model: TF-IDF reaches 0.918 and the network 0.890, against 0.874.

That last result deserves a check, because 500 reviews is not a large test. The sign test (Lesson 5.6) looks only at reviews where exactly one of two models is right, and asks whether the split between them could plausibly be a coin flip.

With 500 reviews, one review is worth 0.2% of accuracy. TF-IDF was right on 43 reviews where the pretrained model was wrong, and wrong on only 21 where it was right; a split that lopsided would happen by chance with probability 0.008. So TF-IDF with all the labels really is better here. The from-scratch network's edge (40 against 32, p = 0.410) could easily be luck.

The picture shows the real trade-off. The dashed green line is what you get for free: no labels, no training, 0.874. The other two methods have to climb towards it with labelled data, and they need several thousand labels to reach it.

So why didn't the pretrained model win outright? Because it was adapted to a different dataset: SST-2's short single sentences, not IMDb's long reviews, which we also cut to 256 tokens. With enough labels of exactly the right kind, even a simple method can beat a powerful model that was tuned for something slightly different. That points to the natural next step, which Module 8 takes: start from the pretrained model and adapt it to your data. You then get its head start and the benefit of your labels.

Common mistake: assuming the biggest or most famous model will win. A pretrained model is only as good as its match to your task. Always compare it against a simple baseline on your own data. Here, a TF-IDF model that trains in seconds beat a Transformer with tens of millions of parameters once it had enough labels.

Key terms

• TF-IDF (term frequency–inverse document frequency): a way of turning a text into numbers that counts each word, giving less weight to words that appear in most documents.
• Training from scratch: training a model starting from random weights, using only your own data.
• Baseline: a simple reference method that a more complex one must beat to justify its cost.

We've treated the pretrained model as a black box that comes with weights. Now let's open the box and look at what you actually download.

When you download a model from the Hub, you mainly get two kinds of file: a configuration that describes the architecture, and a weights file that holds the learned numbers. The code that turns them into a working network lives in the Transformers library on your computer. (A third group of files, for the tokeniser, comes alongside; Lesson 7.2 lists them all.)

The configuration is a small JSON file, readable as plain text. It was downloaded with the model and is already in the local cache.

Most of these lines describe the architecture in Module 6's terms:

• dim: 768 is the width of every token's vector (called d_model in Module 6);
• n_layers: 6 and n_heads: 12 are the number of Transformer blocks and the number of attention heads in each;
• hidden_dim: 3072 is the width inside each feed-forward network;
• vocab_size: 30522 and max_position_embeddings: 512 set the size of the word table and the longest input the model can take;
• dropout values are used only during training.

Two lines belong to the task, not the architecture: id2label maps output 0 to NEGATIVE and output 1 to POSITIVE, which is how the pipeline knew which word to print, and finetuning_task records that it was adapted on SST-2. This is the file we used in Section 1.3 to build the untrained twin.

Let's count the parameters in each part of the model.

The model has 66,955,010 parameters. More than a third of them, 35.6%, are in the embedding tables, before any Transformer layer is reached. The six Transformer layers hold 63.5%, and the sentiment head only 0.9%. We can check the embedding count by hand from the configuration:

The word table alone is 30,522 rows of 768 numbers, which is 23,440,896 parameters, and the position table adds 393,216. Together with two small LayerNorm vectors, that makes the 23,835,648 above. This split matters in practice: a large vocabulary is expensive, and Lesson 7.6 will see it again when we shrink models.

Each parameter is stored as a 32-bit floating-point number, which takes 4 bytes. So the weights file should be about four times the parameter count:

It is: 66,955,010 parameters × 4 bytes is 267.8 MB, exactly the size of the file. The .safetensors format stores raw numbers with a short index and nothing else. Unlike Python's general-purpose pickle format, loading it cannot run hidden code, which makes it the safe choice for files downloaded from strangers. Lesson 7.6 builds on this rule, parameters × bytes per parameter, to estimate memory.

The weights themselves are stored as a state dict: a dictionary from each weight's name to its tensor, exactly as in your own PyTorch models.

There are 104 named tensors. The names spell out where each one lives (distilbert.embeddings.word_embeddings.weight has shape (30522, 768), one row per vocabulary word), and the last four are the head: a 768 × 768 layer followed by a 2 × 768 layer that produces the two class scores.

Figure 7.1.4. Configuration describes the architecture, weights supply learned numbers, and the matching tokenizer defines the text interface. These schematic file objects are not a screenshot.

Key terms

• State dict: a dictionary mapping each weight's name to its tensor; this is what gets saved and loaded.
• Safetensors: a file format for storing weights as plain numbers, safe to load because it cannot contain executable code.
• Weight tying: reusing one weight matrix in two places, such as the word-embedding table and the output layer that predicts words.

A weights file is a snapshot of a model at one moment. The last section looks at those snapshots, and at how models change over time.

A checkpoint is a saved set of weights (together with its configuration) from a particular moment in a model's training. In Lesson 5.3 you kept a copy of the weights from the best epoch; that copy was a checkpoint. On the Hub, each published model is a checkpoint, and models are often built from one another: the SST-2 model is a checkpoint made by fine-tuning the base checkpoint.

How much did that fine-tuning change? We can measure it directly. Below we load the base model's body and compare every tensor with the SST-2 body, using the relative change: the size of the difference divided by the size of the original.

All 100 tensors in the body changed (none is identical), but only a little: on average by 1.85% in the embeddings and between 2.14% and 2.41% in the Transformer layers. Fine-tuning didn't rebuild the model. It nudged every weight slightly, and that was enough to turn a word-guesser into a sentiment classifier. Module 8 makes those nudges itself.

Models on the Hub are stored in repositories that track their history, like code on GitHub. Each change is a commit with its own ID, and a model's files can be updated after you first use them. When the library downloads a model, it stores the files in a cache folder named after the exact commit.

The folder our files came from is named 714eb0fa89d2…, the same ID as the latest of the repository's 26 commits. The three latest commit titles describe changes to the documentation and an extra file format, but a commit could just as easily replace the weights. If your application has to give the same answers next month, ask for an exact version with revision=:

Pinning a revision is the model equivalent of fixing a library version in requirements.txt. Lesson 7.6 returns to this when we look at reproducibility in deployed systems.

Key terms

• Checkpoint: a saved snapshot of a model's weights and configuration at a particular point in training.
• Repository (repo): a Hub storage space for one model, with its files and their full history.
• Commit and revision: a commit is one recorded change to a repository, identified by a long ID; a revision is the version you ask for, which can be a commit ID or a branch name such as main.

Common mistake: loading a model by name only in a system that must stay reproducible. from_pretrained(name) gives you whatever the latest version is on the day you download it. Pin the commit with revision= and record it next to your results.

Let's reassemble the whole picture.

A pretrained model is a model whose weights someone else has already learned. It reaches us as a blueprint (the configuration) and a large file of numbers (the weights), and the code that assembles them comes from a library. We measured that trained weights matter with the architecture and tokeniser held fixed: the SST-2 model scored 0.874 on 500 IMDb validation reviews, and the same architecture with random weights scored 0.486, which is chance.

Those weights come from a two-stage lifecycle. Pretraining on large unlabelled text, here by guessing masked words, produces a base model. It learned grammar and common patterns well (it filled "she [MASK] to the cinema" with "went", and "so boring that I fell [MASK]" with "asleep"), but facts unreliably, and it built contextual representations in which "bank" by a river and "bank" with a loan pull apart layer by layer. Downstream, the same body is shared by many tasks, each with its task-specific output head; fine-tuning may update the body too. The mean relative tensor-norm changes in the body were around 2%; this is not a percentage change for every scalar weight.

In this experiment, the pretrained checkpoint was strongest at the smaller label budgets: it gave, for free, what simpler methods needed thousands of labels to reach. It is not automatically the best: with all 22,404 labels, TF-IDF beat it (sign test p = 0.008), because the model had been tuned on different data. The lesson for the rest of Modules 7 and 8 is to treat a pretrained model as a strong starting point, always measured against a baseline, and adapted to your data when it doesn't fit.

Finally, models change. A checkpoint is one snapshot, and a Hub repository holds many revisions; pinning one makes your results reproducible.

Figure 7.1.5. Author CPU reference results on the same 500 validation reviews. Label budgets and prior training differ; these measurements do not establish a universal winner.