Building Applications with Pretrained Models

Every application in this lesson has the same structure. A pretrained model supplies one capability, a function from an input to an output. The application wraps it with everything a user needs:

• Input handling: what counts as a valid input, and what happens to one that isn't (empty text, the wrong type, a 100-page document).
• Fit checks: is this input the kind of thing the model was built for? A film-review classifier will happily score a password-reset request; the application should notice.
• Decision rules: when to answer, and when to hold back, for example by passing uncertain cases to a person.
• Evaluation: measured on the whole system, with the numbers a product owner cares about: how often it answers, and how often it's right when it does.

Figure 7.5.1. An application adds input validation, decision rules, error handling and measured quality to a pretrained capability. The OOD check is a heuristic to validate, not a guarantee.

All the IMDb work uses the fixed split from Lesson 7.1. The function is copied here, and the fingerprints confirm the same reviews.

We start from the sentiment model and measure it on the 1,000 val_eval reviews. The helper positive_prob returns each text's probability of being positive. It sorts texts by length before batching, to cut padding (Lesson 7.4, Section 6), and puts the results back in the original order.

The model scores 0.882 on the validation reviews. That's the aggregate. An application meets individual inputs, so let's look at those.

We test five groups of inputs, written before running the model: clearly positive, clearly negative, ambiguous (mixed or lukewarm), the four negation sentences from Lesson 5.3, and inputs that are not film reviews at all.

Read the groups one at a time.

• Clear inputs get confident, correct answers (1.000 and 0.000).
• Negation, which defeated Lesson 5.3's model, is handled correctly. There, a bag-of-embeddings model averaged word vectors, so "not good" still looked positive. Here, "the movie was not good" gets 0.000 and "the movie was not bad" gets 0.999. A Transformer reads words in context, so "not" changes the meaning of the word after it.
• Ambiguous inputs get confident answers too, and that's a problem. "The acting was superb, but the story made no sense" gets 0.001: the model reads the "but" clause as decisive, which is defensible. But "It was fine, I guess." gets 1.000 positive, and "Not as bad as the critics said, but not good either" gets 0.003. A human would call these lukewarm. The model, fine-tuned on sentences labelled only positive or negative, has no way to say "in between", and pushes them to an extreme with near-total confidence.
• Out-of-distribution inputs get answers anyway. The gibberish "asdkjh qwe zzz" is called negative with P(positive) = 0.004, a confidence of 0.996. "Please reset my password." gets 0.011. Only "The parcel arrived on Tuesday." gets an uncertain-looking 0.586. (The French review gets 0.928, positive, which happens to be correct. The model was trained on English, so don't count on that: it may simply be reacting to words like "film" that look the same in both languages.)

The lesson for the application is the most important point in this lesson: a high confidence score does not mean the input made sense. The model's probabilities describe how strongly an input resembles one class compared with the other, not whether it belongs to the model's world at all. Section 6 builds a separate check for that.

Key terms

• Capability: what a model can compute from an input, such as a probability of positive sentiment.
• Out-of-distribution (OOD) input: an input unlike the data a model was trained on.
• Confidence: the probability a classifier gives to its chosen class; here, the larger of P(positive) and 1 − P(positive).

In Lesson 6.6 you generated Shakespeare-like text with your own small Transformer and compared decoding strategies. Here we do the same with distilgpt2, a pretrained language model trained on web text. Generating text is a loop: predict a probability for every possible next token, choose one, append it, repeat. The interesting choice is how to choose.

First, let's look at the model's probabilities for the token after "The movie was", and how temperature changes them. Logits are divided by the temperature T before the softmax: T below 1 sharpens the distribution towards the top choices, T above 1 flattens it.

At T = 1.0, the model's own distribution, the top choice " a" has probability 0.067, and the top five tokens together hold only 0.267. Most of the probability is spread over thousands of other tokens. At T = 0.7, the same five tokens hold 0.540; at T = 1.5, only 0.074. The order never changes, only how sharply the probability is concentrated.

We compare four strategies from Lesson 6.6, each continuing "The movie was" for 40 tokens:

• greedy: always take the most likely token;
• temperature 0.7: sample from the sharpened distribution;
• top-k 40: sample from the 40 most likely tokens only;
• top-p 0.9: sample from the smallest set of tokens whose probabilities add up to 0.9 (also called nucleus sampling).

Sampling is random, so we fix the seed before each call. That way you get the same text as these lessons.

Greedy decoding produces one fluent clause ("a hit in the United States and was a hit in the United Kingdom", already repeating itself), then gets stuck emitting blank lines (\n) until it runs out of tokens. Once a blank line is the most likely next token, it stays the most likely next token. The sampled versions are more varied and readable, though not exactly sensible: distilgpt2 invents a "gritty pilot" and a quote from someone called Laffer. The model has learned what film writing sounds like, not what is true (the same lesson as Lesson 7.1, Section 3).

One sample per strategy proves little, so let's measure repetition over several prompts and seeds. We count the share of 3-word sequences that repeat earlier ones in the same text: 0 means no repetition.

Greedy decoding repeats itself in its texts (a mean of 0.18, up to 0.30), while none of the 9 sampled texts for any sampling strategy repeats a 3-word sequence. (Our measure counts words, so greedy's runs of blank lines aren't even included; its real repetition is worse.) For open-ended writing, sampling with a moderate temperature, top-k or top-p is the usual choice. For tasks with one right answer, such as extracting a date, greedy is often better, because variety is not wanted.

Key terms

• Temperature: a number that divides the logits before softmax; below 1 sharpens the distribution, above 1 flattens it.
• Top-k sampling: sampling only from the k most likely tokens.
• Top-p (nucleus) sampling: sampling only from the smallest set of tokens whose probabilities sum to at least p.
• Degenerate repetition: a generator getting stuck repeating the same words or tokens.

Common mistake: comparing decoding strategies on a single sample each. Sampling is random: one lucky or unlucky sample says little. Fix the seeds and compare several prompts and samples.

A sentence-embedding model turns a whole text into one vector, so that texts with similar meanings get nearby vectors. We use all-MiniLM-L6-v2 (Lesson 7.3), with the pooling it was trained for: the average of its token vectors, ignoring padding. Each vector is then scaled to length 1, so a dot product between two vectors equals their cosine similarity.

In Lesson 5.3 we found that embeddings learn what their objective rewards: vectors trained to predict sentiment grouped words by sentiment, while vectors trained to predict neighbouring words grouped them by topic. Let's ask what MiniLM's vectors reward. We compare one sentence with four others:

The paraphrase scores 0.734, as expected. But "A dull, badly acted film" scores 0.737, as high or higher, even though it says the opposite. Meanwhile "A delicious, beautifully cooked meal", which shares the positive sentiment and even the word "beautifully", scores only 0.344. MiniLM was trained to bring together texts about the same thing, such as a question and its answer, or two descriptions of the same event. So its vectors capture topic far more than opinion. That's exactly what search needs, but it means these embeddings, used as they are, are a poor way to measure sentiment.

Search is the classic use: embed a collection of documents once, embed each query, and return the documents whose vectors are closest. We embed the first 1,000 training reviews and try three queries written before running.

The results match the meaning of the queries, not their exact words. The best match for "terrible special effects and wooden acting" says the special effects "should be named special defects" without using the word "terrible". For "a heartwarming story about a family", all three are moving, positive reviews. As in Section 4.1, the search finds topics, not opinions: the horror query returns one review that loved its film and two that hated theirs, and one of the "terrible special effects" results is a positive review that mentions bad acting at the start.

Key terms

• Sentence-embedding model: a model that maps a text to one vector, placing similar meanings close together.
• Mean pooling: averaging a text's token vectors (ignoring padding) to get one vector for the whole text.
• Semantic search: retrieving documents by similarity of meaning, using embeddings, rather than by matching words.

In Lesson 5.5 you built a toy version of CLIP: an image encoder and a text encoder trained so that matching image–caption pairs had similar vectors. It worked, but it could only recognise what its small training set described. Trained without sneakers, it matched sneaker images to "sandal" and "ankle boot". The real CLIP was trained the same way on an enormous collection of images and captions from the web. That lets it do zero-shot classification: we write each class as a sentence ("a photo of a sneaker."), embed the sentences, and give each image the class whose sentence is most similar. No training on our classes at all.

We use the Fashion-MNIST test file from Module 5. Fashion-MNIST images are tiny (28 × 28), greyscale and drawn light-on-dark, very unlike CLIP's web photos, so two settings may matter: whether to invert the images to dark-on-white, and how to word the class template. To choose them fairly, we use two separate sets of images: the first 300 to choose settings, and 1,000 others to report the result.

We try both settings on the 300 choosing images and keep the best combination.

Both changes help: inverting the images to dark-on-white, like a product photo, and describing them as black-and-white photos. The best combination scores 0.750 on the choosing images, against 0.633 for the plain setup. Now we use that combination, once, on the 1,000 reporting images:

CLIP classifies 0.733 of the 1,000 new images correctly, against 0.100 by chance, with no training on Fashion-MNIST at all. The result on the reporting images is close to the 0.750 seen on the choosing images, as it should be when settings are chosen fairly.

The per-class results tell the real story:

• Sneakers, the class Lesson 5.5's toy could not recognise, are recognised 0.79 of the time. CLIP learned what a sneaker is from the web, not from our data.
• Trousers and bags are almost perfect (0.98). They have distinctive shapes.
• Shirts nearly always fail (0.15), most often called pullovers, and dresses manage only 0.48. At 28 × 28 pixels in greyscale, a shirt, a T-shirt, a pullover and a coat really are hard to tell apart. Their differences (buttons, collars, fabric) are mostly lost at this size.

Because images and text share one space, we can also search images with words: embed a query sentence and return the closest images.

For "sneaker", "bag" and "dress", 4 of the top 5 images are right. The mistakes are understandable: a sandal for "sneaker", a T-shirt for "dress", and a plaid shirt for "bag". For "shirt", only 2 of the top 5 are shirts and the other three are T-shirts, the same confusion the per-class numbers showed. In the dataset's labels a T-shirt and a shirt are different classes, but in everyday English the word "shirt" covers both, and CLIP learned its idea of a "shirt" from everyday English.

Key terms

• Zero-shot classification: classifying into classes the model was never trained on, by describing each class in words.
• Prompt template: the sentence pattern used to describe each class, such as "a photo of a {}.".
• Text-to-image retrieval: finding the images whose embeddings are closest to a text query's embedding.

Common mistake: choosing a template or preprocessing step on the same images you report. Trying several settings and reporting the best one on the same data overstates accuracy. Choose on one set, report on another, as we did with the 300 and the 1,000.

Section 2 showed two weaknesses an application must handle: the model answers inputs that aren't film reviews, and it is sometimes unsure. We'll handle each with a rule, and set each rule's threshold on validation data before looking at the test set.

• Review-likeness check. For any input, we compute its MiniLM embedding's similarity to the closest of our 1,000 known training reviews (Section 4). Genuine reviews resemble other reviews; a password-reset request doesn't. We set the threshold so that 99% of 500 unused validation reviews pass. An input scoring below that is flagged as out of domain.
• Abstention. The model's confidence on the val_eval reviews from Section 2 tells us how sure it usually is. We set the threshold at the 10th percentile, so the service holds back on its least confident tenth of reviews and leaves them for a person.

Genuine reviews almost all score above 0.364 in review-likeness, and the confidence threshold is 0.948. Both were set without touching the test set.

The class below wraps everything. It returns one result per input, as a dictionary with a status: ok (with a label), abstain (unsure), out_of_domain (not a film review), rejected (invalid input, with the reason) or error (the model failed). The design choices worth copying:

• Check inputs before the model sees them. Wrong types, empty or whitespace-only text, and absurdly long text are rejected with a reason, instead of crashing or wasting computation.
• Never let one bad input break a batch. Invalid items are set aside, the rest are processed together, and a model failure is caught and reported as a status rather than raised.
• Return reasons, not just answers. A caller can show "low confidence" or "not a film review" to a user, or route the item to a person.
• Keep the thresholds as settings, so they can be re-tuned without changing code.

Let's try it on a mix of inputs: good reviews, the problem inputs from Section 2, and invalid ones.

Every input gets a sensible outcome. The two clear reviews and "the movie was not bad" are answered. The gibberish, the parcel and the password request are all flagged out_of_domain, with review-likeness scores of 0.238, 0.16 and 0.141, below the 0.364 threshold. The sentiment model alone had been 0.996 and 0.989 sure about two of them. Empty and whitespace-only text, None, a number and a 25,000-character string are rejected with clear reasons, and none of them crashed anything.

Notice one remaining weakness: "The acting was superb, but the story made no sense." is answered NEGATIVE with confidence 0.999. Abstention only catches uncertainty the model admits; it can't catch confident mistakes on genuinely mixed reviews.

Key terms

• Inference service: code that wraps a model with input checks, decision rules and error handling, and returns structured results.
• Abstention: declining to answer an input, typically when confidence is low, so that a person or another system can handle it.
• Out-of-domain detection: checking whether an input resembles the data a model was built for.
• Threshold: the cut-off value in a decision rule; set it on validation data.

Interpret the evidence carefully. Review-likeness is a heuristic demonstrated on these inputs, not a guarantee of robust out-of-domain detection. Validate false accepts and false rejects on representative in-domain and out-of-domain examples before deploying it.

Visual walkthrough: Trace three service decisions.

1. Validate the input. Empty input → reject. Validation happens before model inference. A model score cannot repair an invalid request.

2. Check the declared domain. Flagged out of domain → hold back. This course uses a review-likeness heuristic, which can falsely accept or reject inputs.

3. Apply a confidence threshold. Illustration: 0.88 ≥ 0.80 → answer. These are invented decision-rule values, not a measurement of model calibration.

4. Abstain when required. Illustration: 0.63 < 0.80 → abstain. The source run answers 909, abstains on confidence for 79, and flags 12 as out of domain. All 91 non-answered cases form the held-back group.

The app adds a paused animation of this same supplied example.

Now the test set, used once. We run the service on the 1,000 test_eval reviews. With 1,000 reviews, each review is worth 0.1 percentage points of accuracy.

The service answered 909 reviews, abstained on 79 and flagged 12 genuine reviews as out of domain: about 1%, matching the rate we designed on validation. Three numbers describe the application:

• Accuracy if we answered everything: 0.875. This is the model's raw accuracy.
• Coverage: 0.909 of reviews were answered.
• Accuracy on the answered reviews: 0.899.

The held-back reviews really were the hard ones: the model would have got only 0.637 of them right. Abstention bought a higher accuracy on what we do answer at the cost of answering fewer reviews, and that is a product decision, not a technical one.

Accuracy hides which mistakes a model makes. A confusion matrix counts each combination of true and predicted label:

Of 485 negative reviews (434 + 51), 434 were recognised; of 515 positive reviews (74 + 441), 441 were. The model misses positive reviews more often than negative ones: a recall of 0.856 for positive against 0.895 for negative. If the feature exists to surface unhappy customers, missing 51 negative reviews matters more than the positive mistakes. If it exists to quote praise on a poster, the 74 missed positives matter more. The same model can be good enough for one feature and not the other.

Our threshold is one point on a continuum. If we rank the test reviews from most to least confident and answer only the top share, we get a curve:

The curve falls steadily as we answer more. Answering only the most confident half gives 0.984 accuracy; 80% gives 0.932; our service's point, about 90%, gives 0.899; answering everything gives 0.875. (The curve is jagged at the far left, where only a few reviews are answered.) This is the tool for a product conversation: "If people can review one review in ten, the model is right 0.899 of the time on the rest; if they can review half, 0.984."

Finally, how long does one request take? A request runs two models, the classifier and the review-likeness embedding.

Each request takes tens of milliseconds on this CPU (the printed time will differ on your machine). Lesson 7.6 looks at how latency, batching and hardware trade off in deployment.

Figure 7.5.2. Of 1,000 source test reviews, 909 were answered, 79 received a confidence abstention and 12 were flagged out of domain. Held-back accuracy 0.637 applies to all 91 non-answered reviews.

Key terms

• Coverage: the share of inputs for which the system gives an answer.
• Selective accuracy: accuracy measured only on the inputs the system answers.
• Confusion matrix: a table counting each combination of true and predicted label.
• Recall (per class): the share of a class's real examples that the model labels correctly.

Common mistake: reporting only the model's accuracy for an application with abstention or filtering. Always report coverage alongside accuracy on answered inputs. A system that answers only its easiest 10% can show near-perfect accuracy and still be useless.

Let's reassemble what we built.

Four pretrained models gave four capabilities with a few lines each. The sentiment model scored 0.882 on validation, handled negation that defeated Lesson 5.3's model, but gave near-certain answers to lukewarm and nonsensical inputs: a confidence of 0.996 for "asdkjh qwe zzz". The text generator wrote fluent but invented text. Greedy decoding looped (repeated 3-word sequences: 0.18 on average), while temperature, top-k and top-p sampling didn't repeat in any of their 9 texts. The sentence embeddings found reviews by meaning, but rated "A dull, badly acted film" as close to "A thrilling, beautifully acted film" (0.737) as a true paraphrase was (0.734): they capture topic, not opinion. CLIP classified Fashion-MNIST zero-shot with 0.733 accuracy, recognising sneakers that Lesson 5.5's toy could not, once we chose inversion and a template on held-out images. It still failed on shirts.

Capabilities become features through an application layer: input checks, an out-of-domain check (embeddings, with a threshold set so 99% of genuine validation reviews pass), and abstention (the least confident tenth, set on validation). On 1,000 unseen reviews, the service answered 909 with 0.899 accuracy, against 0.875 for answering everything; the 91 it held back (79 abstentions plus 12 out-of-domain flags) would have been right only 0.637 of the time. The confusion matrix showed which way the errors go (positive recall 0.856, negative 0.895), and the coverage–accuracy curve turns the threshold into a product decision.

Figure 7.5.3. Each capability has failure modes. The notebook measures particular examples; similarity, confidence and fluent generation do not by themselves establish correctness.

Interpret the evidence carefully. Here the 0.637 held-back accuracy applies to all 91 non-answered reviews, not just the 79 low-confidence abstentions. The semantic-similarity and repetition observations are results of these particular examples, models and settings, not universal properties of embeddings or sampling.