Loading and Using Pretrained Models

Every use of a pretrained text model follows the same five steps, whether a pipeline does them for you or you write them yourself:

1. Load the tokeniser and the model from the same repository (Lesson 7.2).
2. Preprocess: turn texts into token IDs, cut long ones to a maximum length, pad short ones so a batch forms a rectangle, and build an attention mask that marks which positions are real.
3. Run the model in inference mode, on the right device.
4. Decode the outputs: raw scores (logits) → probabilities → labels.
5. Save the model and tokeniser if your application needs its own copy.

Figure 7.4.1. Tokenization, model computation, logits, probabilities and label mapping are separate steps. Weights stay fixed during this inference flow.

We'll use the fixed IMDb split from Lesson 7.1 (its function is copied here so the notebook stands alone; the fingerprints confirm the reviews are the same) and work with the first 32 validation reviews.

A pipeline bundles a tokeniser, a model and the decoding step for one task. We create one by naming the task and the model. Always pass device. If you don't, the library picks a device itself, and on some machines (for example a Mac with an Apple GPU) it will choose the GPU even though the rest of your code runs on the CPU. Your results could then differ slightly from someone else's for no obvious reason.

Inside are the two objects we'll load ourselves in Section 3: a BertTokenizer (Lesson 7.2 explained why a DistilBERT model gets a BERT tokeniser) and a DistilBertForSequenceClassification model.

By default, a text-classification pipeline returns only the best label and its probability. top_k=None returns every class:

The model is sure this mixed sentence is positive: 0.9997346997261047 for POSITIVE and 0.0002653122355695814 for NEGATIVE, which add up to 1.

Give a pipeline a list and it returns a list. For real reviews you should always set the truncation length, and for many texts a batch_size, so they are processed several at a time (Section 6 measures why).

The first review comes out NEGATIVE with probability 0.9940033555030823, and the model gets 0.906 of these 32 reviews right. We keep each review's probability of POSITIVE (for a NEGATIVE result, that is 1 minus the score) to compare against our own code in the next section.

Key terms

• Pipeline: a Hugging Face object that bundles a tokeniser, a model and output decoding for one task.
• top_k: how many of the highest-scoring labels a classification pipeline returns; None means all of them.
• batch_size: how many inputs are processed together in one forward pass.

Common mistake: creating a pipeline without device=. The library's automatic choice can differ between machines, so the same notebook can run on different hardware for different people, with slightly different numbers. State the device explicitly.

The explicit route uses the two Auto classes from Lesson 7.2. For a classification model we use AutoModelForSequenceClassification, which builds the body and the classification head. Two more steps follow loading: .to(device) moves the weights to the chosen device, and .eval() switches the model to inference behaviour (Section 5 shows why).

The configuration's id2label tells us that output 0 means NEGATIVE and output 1 means POSITIVE. We need it to decode the outputs.

Now we process the same 32 reviews ourselves, in batches of 8, and compare the probabilities with the pipeline's. Each step is one line: tokenise with truncation and padding, run the model, and apply softmax to the logits.

The largest difference between our probabilities and the pipeline's is 5.96e-08, a rounding-level difference in the last digits of 32-bit arithmetic, and all 32 labels agree. Our explicit code does exactly what the pipeline does. The rest of this lesson looks at each of its steps in turn.

Key terms

• AutoModelForSequenceClassification: the Auto class that builds a model with a whole-text classification head.
• id2label: the configuration's map from output position to label name.

A model processes a batch as one tensor, and a tensor must be a rectangle: every row the same length. Texts are not. Padding fixes this by adding a filler token to the end of each shorter text until all match the longest. Let's tokenise two texts of different lengths.

The tokeniser returns three tensors. Look at input_ids: both rows start with 101 ([CLS]) and the real text ends with 102 ([SEP]), as in Lesson 7.2. The first text needs only 6 tokens, so it is padded with six zeros: the padding token [PAD] has ID 0, added on the right. (token_type_ids is the extra BERT output from Lesson 7.2, which DistilBERT ignores.)

The attention mask has a 1 for every real token and a 0 for every padding position. The model uses it inside attention: positions with a 0 are blocked, so no real token can attend to padding. It is the same idea as the causal mask in Lesson 6.3, which blocked future positions. Without it, the meaningless [PAD] vectors would be mixed into the real tokens' representations.

The opposite problem is a text that is too long. DistilBERT can read at most 512 tokens, because it has only 512 learned position embeddings (Lesson 7.1, Section 6). Truncation cuts a text down to a maximum length. Here is the longest of the 1,000 val_eval reviews, tokenised with and without it:

Without truncation, the review is 1602 tokens long, and the tokeniser prints a warning that this is longer than the model's limit of 512. With max_length=256 it becomes exactly 256 tokens, and it still ends with a proper [SEP] token: truncation cuts off the end of the text, but keeps the special tokens. Everything after the cut is simply never seen by the model.

What happens if we ignore the warning and pass the full review to the model?

The model fails: a RuntimeError complaining that a tensor of size 1602 doesn't match one of size 512, the position-embedding table. (With some library versions, the error is an IndexError instead.) That warning from the tokeniser was worth reading. Always set truncation=True and a max_length for real-world text.

Padding should change nothing about a review's prediction: a review should get the same answer whether it is processed alone or in a batch next to a longer one. Let's test that on our 32 reviews. We process them as one padded batch, once with the attention mask and once without it, and compare each against processing the review alone, where it needs no padding at all.

In this batch, 15 of the 32 reviews needed padding, and 18.4% of all positions are padding. With the mask, every review's probability matches the one-at-a-time result to within 2.21e-06, rounding noise. Without the mask, probabilities move by as much as 0.764, and 2 reviews flip to the wrong side of 0.5. Same model, same reviews; the only difference is whether the model was told which positions were real. Passing the whole tokeniser output with model(**enc) includes the mask automatically. The danger is code that passes only input_ids.

The model's raw output is a pair of logits, unnormalised scores, one per class. To turn them into probabilities we apply softmax, as in Module 2: exponentiate each logit, then divide by the sum so they add up to 1. Here it is by hand for the first review:

The logits are 2.772 for NEGATIVE and −2.338 for POSITIVE. Exponentiating gives 15.9923 and 0.0965, which sum to 16.0887. Dividing each by the sum gives probabilities 0.994 and 0.006. The larger one wins, and id2label turns position 0 into the word NEGATIVE. That's exactly the pipeline's answer (NEGATIVE, 0.9940033555030823), and it matches the true label.

Two practical points. First, only the difference between logits matters: adding the same number to both leaves the softmax unchanged. Second, a probability of 0.994 is the model's confidence, not a guarantee. Lesson 7.1 showed this model confidently wrong about an idiom, and Lesson 7.5 looks at how far to trust these scores.

Figure 7.4.2. Padding adds positions, an attention mask restricts access to padded positions, and truncation discards input. This illustration is not a causal future-token mask.

Key terms

• Padding: adding filler tokens ([PAD], ID 0 here) so all texts in a batch have the same length.
• Attention mask: a tensor of 1s (real tokens) and 0s (padding) that stops attention from using padding positions.
• Truncation: cutting a text down to a maximum number of tokens; truncation=True, max_length=….
• Logits: a model's raw, unnormalised output scores, one per class.
• Softmax: the function that turns logits into probabilities: exponentiate, then divide by the sum.

Common mistake: calling the model with input_ids only, for example model(enc["input_ids"]), on a padded batch. Nothing fails, but padded reviews get different, wrong predictions. Pass the whole encoding with model(**enc) so the attention mask goes in too.

Visual walkthrough: Follow padding and attention masking.

1. Tokenize two inputs. A: [101, 42, 102] · B: [101, 17, 25, 102]. IDs are arbitrary vocabulary indices in this toy example; 101/102 mark boundaries and 0 is padding.

2. Pad this batch. A: [101, 42, 102, 0] · B unchanged. The longest example has four tokens, so the shorter example receives one padding position.

3. Construct masks. A: [1, 1, 1, 0] · B: [1, 1, 1, 1]. The mask allows attention to real tokens and prevents attending to the padded key. This is not a causal future-token mask.

4. Run fixed weights. IDs + masks → logits → label. Padding creates a batch shape; it does not restore content discarded by truncation. Inference changes no weights.

The app adds a paused animation of this same supplied example.

In Module 3 you met dropout, which randomly switches off a share of the network's units during training to reduce overfitting. That randomness is useful while learning but harmful when you want a stable answer. PyTorch models have two modes: model.train() turns dropout on and model.eval() turns it off. from_pretrained already returns a model in eval mode, but it is good practice to call .eval() explicitly, because any code that calls .train() (for example a fine-tuning loop in Module 8) switches dropout back on.

Here is the same review, run three times in each mode:

In train mode, the POSITIVE logit for one and the same review comes out as -2.3716, -2.78 and -2.0684: dropout makes every run different. In eval mode it is -2.3384 every time. A model left in train mode gives answers that change on every call, and slightly worse ones on average.

During training, PyTorch records every operation so that it can compute gradients later (Module 2's autograd). At inference time there is nothing to learn, and that recording only wastes memory and time. Two context managers switch it off:

A plain call produces logits that require gradients and carry a grad_fn (here AddmmBackward0), the record PyTorch keeps for backpropagation, along with every intermediate tensor it needs. Under torch.no_grad() nothing is recorded. torch.inference_mode() goes slightly further: its outputs are marked as inference tensors, which lets PyTorch skip some extra bookkeeping. For pure prediction, inference_mode is the best choice. Use no_grad if you need to use the outputs later in code that records gradients.

A model's weights and its inputs must be on the same device. If the model is on a GPU and the inputs are on the CPU, PyTorch stops with an error saying they are on different devices. The pattern is always the same: move the model once with model.to(device), and move every batch with enc.to(device), as all our cells have done.

In this notebook both are on the CPU. On a machine with an NVIDIA GPU, the setup cell would choose cuda, and the same code would place both on the GPU. Lesson 7.6 compares the two.

Key terms

• Eval mode / train mode: model.eval() disables dropout (and other training-only behaviour); model.train() enables it.
• torch.no_grad() / torch.inference_mode(): context managers that stop PyTorch recording operations for gradients; inference_mode is the faster, stricter one.
• Device placement: putting the model's weights and its input tensors on the same device (CPU or GPU).

Common mistake: forgetting .eval() after training or fine-tuning. The model keeps dropout on, so evaluation scores are noisy and slightly too low, and the same input gives different answers each time.

Batching lets the model process several texts in one forward pass, which uses the hardware more efficiently. But a padded batch is as long as its longest member, so every shorter review pays for tokens it doesn't have. How much waste depends on which reviews end up together. Here are the first 256 validation reviews in batches of 32, in the order given and sorted by length.

The 256 reviews contain 50,878 real tokens. Batched in the order given, the model processes 65,536 positions, so 22.4% of its work is spent on padding. Sorted by length, similar lengths share a batch, and the total drops to 53,984, with only 5.8% padding. Same reviews, same model, a fifth less work. (Every batch in the given order was padded to the full 256 tokens, because each batch of 32 happened to contain at least one long review.) If you sort, remember to put the results back in the original order afterwards.

Let's time it. We run the 256 reviews with batch sizes 1, 8 and 32, then batch size 32 sorted by length. The timings are printed with units; yours will differ, since they depend on your processor.

On this CPU, larger batches are somewhat faster per review than one review at a time, and sorting by length gives a further saving on top, roughly in line with the padding it removes. The gain from batching on a CPU is modest. A GPU can process many more inputs in parallel, so there the gain from batching is usually much larger (Lesson 7.6).

Batching has a cost: memory. While a batch passes through the model, every layer holds the vectors of every token, and attention computes a score for every pair of tokens in every head. We can estimate both for DistilBERT from its configuration (768 numbers per token, 12 heads, 4 bytes per number):

One review of 256 tokens needs about 0.8 MB of token vectors per layer, and 3.1 MB if the attention scores are stored in full. A batch of 32 multiplies both by 32 (25.2 MB and 100.7 MB). Doubling the length to 512 doubles the token vectors but quadruples the attention scores, to 402.7 MB, because scores grow with length squared (Lesson 6.3). A batch of 256 reviews at 512 tokens would need 3221.2 MB just for one layer's scores. Libraries often avoid storing the full score matrix, but the lesson holds: memory grows with batch size × length, and faster still with length when attention is computed in full. If a large batch runs out of memory, reduce the batch size or the maximum length.

Key terms

• Batching: processing several inputs in one forward pass.
• Padding waste: computation spent on padding positions in a batch.
• Sorting by length (length bucketing): grouping inputs of similar length into the same batch to reduce padding.

An application that downloads its model from the Hub every time it starts depends on the Hub being available, and on the model not having changed (Lesson 7.1, Section 7). Often you want your own copy. save_pretrained writes a model or tokeniser to a folder in the same format as a Hub repository, and from_pretrained accepts a folder path as well as a repository name. In Module 8 this is how you'll save your fine-tuned models.

The folder holds the configuration, the weights (267.833 MB, the same size as on the Hub) and the tokeniser. The library saved it as one tokenizer.json file, the Tokenizers library's complete description of the tokeniser, rather than the vocab.txt file the Hub repository had. "Writing model shards" refers to the weights file: very large models are split into several files, called shards. This one fits in one.

Now let's reload from the folder and check that nothing changed:

The reloaded tokeniser produces the same token IDs, and the reloaded model produces outputs identical to the original, bit for bit. We then delete the copy, because it takes as much disk as the original model. In a real project you would keep it, and record which repository and commit it came from.

Key terms

• save_pretrained: saves a model or tokeniser to a folder in Hub format.
• from_pretrained(path): loads from a local folder as well as from a Hub repository name.
• Shard: one of several files that together hold a large model's weights.

Common mistake: saving the model but not the tokeniser. The folder then can't be used on its own, and someone may pair the model with a different tokeniser. Save both into the same folder.

Let's reassemble the whole path from repository to prediction.

A pipeline does five steps for you, and you can do each yourself. Load the tokeniser and model from the same repository, with AutoTokenizer and AutoModelForSequenceClassification, moving the model to a device and into eval mode. Our explicit code matched the pipeline to within 5.96e-08 on 32 reviews. Preprocess with truncation (a review of 1602 tokens crashed the model until it was cut to fit), padding (so a batch is a rectangle) and the attention mask (without it, padded reviews' probabilities moved by up to 0.764 and 2 labels flipped). Run in eval mode (train mode gave three different answers for one review) and under inference_mode (no gradient recording), with model and inputs on the same device. Decode logits with softmax: [2.772, −2.338] became [0.994, 0.006], and id2label turned position 0 into NEGATIVE. Batch sensibly: sorting by length cut padding from 22.4% to 5.8% of the work, and memory grows with batch × length, faster with length when attention scores are stored in full. Save with save_pretrained and reload with from_pretrained(folder), which gave identical outputs.

Figure 7.4.3. Evaluation mode controls layers such as dropout; no-gradient inference controls autograd. Model and inputs must use compatible devices. Save preprocessing and settings too.