Practical Model Selection and Integration

A vague goal ("a good sentiment model") can't be checked. A requirement turns it into criteria that a measurement can pass or fail. Writing it down before looking at any results also protects you from choosing whichever model happens to look best and then inventing reasons afterwards. Here is ours:

Each line comes from a real constraint:

• Accuracy at least 0.85 on our validation reviews: the product team's minimum for the feature to be worth having.
• p95 latency within 100 ms for one review on a laptop CPU, because users wait for the answer (Lesson 7.6).
• At most 600 MB of float32 weights, so the model fits beside the rest of the application.
• An allowed licence: Apache 2.0, MIT or CC-BY 4.0, all of which permit commercial use (CC-BY requires credit to the authors).
• Documented training data that doesn't include our evaluation reviews, so that our measurements mean something (Lesson 7.3, Section 4.3).

The decision rule, also fixed in advance, is: among candidates that pass every check, choose the one with the highest validation accuracy; then tune its settings on validation, and confirm once on test.

We evaluate on the fixed IMDb split from Lesson 7.1 (the function is copied here; the fingerprints confirm the same reviews).

Searching the Hub for text-classification models about sentiment (Lessons 7.2 and 7.3) turns up thousands. Three kinds are worth testing, and we take one of each:

• a general sentiment model: the SST-2 DistilBERT we've used throughout this module;
• a model fine-tuned on exactly our task: a DistilBERT trained on IMDb by the TextAttack project;
• a widely used model from another domain: a RoBERTa model for sentiment in tweets, from Cardiff University.

All three are tagged text-classification. The two DistilBERTs have weight files of 267.8 MB; the RoBERTa model's is 501.0 MB. Two of them only offer the older pytorch_model.bin format, not safetensors (Lesson 7.2), which is acceptable for well-known publishers but worth noticing.

Key terms

• Requirement: a set of concrete, checkable criteria a model must meet, written before any measurement.
• Decision rule: the procedure, fixed in advance, for choosing among candidates that meet the requirement.
• Candidate: a model shortlisted for testing.

Before running anything, we read what each candidate says about itself.

The three cards differ sharply:

• SST-2 DistilBERT: licence apache-2.0, training data declared (sst2, glue), a 3990-character card, and named labels (NEGATIVE, POSITIVE).
• TextAttack IMDb DistilBERT: no licence (None), no declared datasets, a 615-character card, and unnamed labels (LABEL_0, LABEL_1), so we don't even know which output means positive.
• Twitter RoBERTa: licence cc-by-4.0, trained on tweet_eval, and three labels: negative, neutral and positive.

The TextAttack card is short enough to read in full:

It says the model was fine-tuned on "the imdb dataset" for 5 epochs with a maximum sequence length of 128, and that its best "eval set accuracy" was 0.88. But which parts of IMDb? This repository also contains the training script's settings, train_args.json, and reading it answers the question:

This settles it. The model was trained on IMDb's train split, all of it, and its dev_split, the data used to choose its best epoch, was IMDb's test split. That has two consequences for us:

• Our validation set is inside its training data. We carved validation from IMDb's training half (Lesson 7.1), so this model has seen every one of our validation reviews, with their labels. Its validation score will measure memory, not skill.
• Even the test split is not perfectly clean for it. Choosing the best of 5 epochs by test accuracy is a mild form of tuning on test, which nudges its test score upwards.

This is the contamination problem from Lesson 7.3, found in the model's own records. Without train_args.json we'd have had to guess. Many repositories don't include such records, which is itself a reason for caution.

Finally, the Twitter model's card, on its data and labels:

It was pretrained on about 124 million tweets and fine-tuned on the TweetEval benchmark, with labels 0 negative, 1 neutral and 2 positive. Tweets are short, informal and full of usernames and links, which is a different domain from long film reviews. And our task has no "neutral" class, so we must decide how to use its three outputs (Section 3).

Key terms

• Label scheme: the set of output classes and what each output position means.
• Training records: files that document how a model was trained, such as arguments, logs and data splits.
• Evaluation leakage: evaluation data that a model saw during training or model selection, which inflates its score.

Common mistake: assuming a model with unnamed labels (LABEL_0, LABEL_1) uses the order you expect. The mapping is whatever its trainer used. Check it with inputs whose answer is obvious before trusting any accuracy number.

We load all three with the Auto classes from Lesson 7.4. RoBERTa prints a LOAD REPORT with two UNEXPECTED weights, roberta.pooler.dense.*: a pooling layer the checkpoint contains but this classification class doesn't use. As in Lesson 7.1, Section 4, that is harmless.

The two DistilBERTs have 66,955,010 parameters each; the RoBERTa model has 124,647,939.

Each candidate speaks a different label language, so we need one function that turns any of them into P(positive). First, the TextAttack model's unnamed labels. Two reviews with obvious answers tell us the mapping:

LABEL_1 gets 0.998 for the glowing review and LABEL_0 gets 0.998 for the scathing one, so LABEL_1 is positive.

For the Twitter model, we decide before measuring to compare only its positive and negative outputs and ignore neutral: P(positive) = P(pos) / (P(pos) + P(neg)). Our task forces a choice between two answers, so the honest comparison is between those two outputs. One function now serves all three models:

Aggregate accuracy comes later. First, we run the inputs that matter for the feature: clear cases, the negation and lukewarm cases from Lesson 7.5, two inputs that aren't reviews, and the start of a real validation review.

All three get the clear cases right, and all three call "the movie was not bad" positive. The differences are in how they handle uncertainty:

• SST-2 DistilBERT is extreme everywhere: 1.000 for "It was fine, I guess." and 0.004 for gibberish, as we saw in Lesson 7.5.
• TextAttack IMDb DistilBERT is far more moderate on unclear inputs: 0.787 for "It was fine, I guess.", 0.351 for the "not bad, not good" sentence, and 0.401 and 0.641 for the two non-reviews. Its scores are closer to "unsure" where a person would be unsure, which is a useful property.
• Twitter RoBERTa sits in between, and with its neutral option ignored it is also confident on the lukewarm sentence (0.971).

On the real review (a negative one), all three are right. Representative inputs don't choose the model on their own, but they show which behaviours to watch for, and they would catch a broken label mapping immediately.

All three candidates are measured on the same 500 validation reviews, truncated to the same 256 tokens.

The SST-2 model scores 0.874 and the Twitter model 0.834. The TextAttack model scores 0.982, far ahead, and we know why: it was trained on these very reviews (Section 2). Its number is real, but it measures the wrong thing. Without having read train_args.json, this is exactly the result that would have won the decision.

Next, p95 latency for single reviews on this CPU (Lesson 7.6), and weight memory from parameters × 4 bytes:

All three are well inside the 100 ms limit on this machine, and the two DistilBERTs are about equally fast, as expected, since they share an architecture. The larger RoBERTa model is noticeably slower and needs 498.6 MB of weights, against 267.8 MB for the DistilBERTs. All fit under 600 MB.

Now we apply every criterion from Section 1 to every candidate:

Only the SST-2 model passes every check. The TextAttack model fails twice, on the licence and on training data that overlaps our evaluation. Either failure alone rules it out, whatever its accuracy. The Twitter model has an acceptable licence and clean data, but its 0.834 falls short of 0.85: tweets are not film reviews.

The chosen model has one important setting, max_length, the truncation length. Lessons 7.3 and 7.4 showed that 256 tokens cuts off many reviews. Lesson 7.3 couldn't separate the effect of truncation from the fact that long reviews are harder, because it compared different reviews. Here we run the same 300 validation reviews at three lengths, and use the sign test to see whether the difference is real:

Interpret the evidence carefully. The documentation check rejects a known contaminated candidate. For other candidates, “no documented overlap found” is more accurate than a guarantee of clean upstream training. An absent licence field is missing evidence; inspect original publisher terms before concluding that no licence exists.

Accuracy rises with the amount of review the model can read: 0.843 at 128 tokens, 0.873 at 256 and 0.903 at 512. Comparing 512 with 256 on the same reviews, 12 reviews are right only at 512 and 3 only at 256. That split would arise by chance with probability 0.04, so the gain from reading more is real, not noise. More tokens cost more time (Lesson 7.3), so we check that 512 tokens still meets the latency requirement:

It does, with room to spare on this machine. So the validation evidence chooses max_length=512.

The model and its setting are now fixed, so we use the test set once to confirm the result. For comparison, we also run the rejected TextAttack model on the same test reviews. Its test score is the closest we can get to a fair measure of what IMDb fine-tuning achieves.

The chosen setup scores 0.888 on 500 test reviews, a little below its 0.903 on validation. That is normal: some of a validation score's height comes from having chosen the best setting on it.

The TextAttack model falls from 0.982 on validation to 0.916 on test. Most of its apparent lead was leakage. On the same test reviews it was right on 36 that our model got wrong, and wrong on 22 that ours got right; a split like that would occur by chance with probability 0.087. So on these 500 reviews we can't be confident it's better at all, and its test score is itself a little optimistic, since it picked its best epoch on IMDb test. If it is better, the most likely reason is clear: it was fine-tuned on IMDb itself. That's the idea Module 8 takes up, adapting a model to your data yourself, with a licence and records you control.

Figure 7.7.1. Source validation and CPU measurements under declared requirements. The IMDb-trained candidate has known evaluation exposure and missing licence evidence; other candidates are screened by documented information.

Key terms

• Screening: eliminating candidates on non-negotiable criteria (licence, data, size) before comparing performance.
• Setting (hyperparameter) tuning: choosing values such as max_length by their validation results.
• Confirmatory test: a single use of the test set after all choices are made, to estimate real-world performance.

Common mistake: letting a spectacular number override the rules. The TextAttack model's 0.982 was the most impressive result in this lesson, and it was the least trustworthy. A number that looks too good is a signal to check what the model was trained on, not a reason to skip the checks.

A decision isn't complete without knowing how the chosen model fails. Here are the first three validation reviews (in validation order, not chosen by outcome) that it gets wrong at the chosen setting:

The model gets 29 of 300 validation reviews wrong. The first three show recognisable patterns:

• A positive review that reads like a complaint. "Low budget Brit pop melodrama … cannot hide the fact that we have all been here before" is labelled positive, but most of its first sentences criticise the film. The model gives it 0.011.
• A negative review that opens with neutral plot summary. The "Romulus, My Father" review spends its opening describing the story; the model, unsure, leans positive at 0.613.
• A review with mixed feelings. "I really liked this movie … Great story with great actors but with a terrible end" is labelled positive, but the model fixes on the "terrible end" and gives it 0.003.

None of these is a bug. They are the hard cases of sentiment: mixed opinions, sarcasm and summary. Knowing them tells a product team what to expect, for example that mixed reviews are where human review adds most.

Finally, we write the decision down, with its evidence, in a form the next person can check. It builds on the manifest from Lesson 7.6, extended with what was rejected and why, and with known limitations.

The record pins the model to commit 714eb0fa…, states the setting (max_length 512), gives the validation and test results with the fingerprints of exactly which reviews they came from, records the latency on this machine, explains why each other candidate was rejected, and lists known limitations. Those limitations come from this lesson and earlier ones: confidence on lukewarm reviews and on non-reviews (Lesson 7.5), truncation after 512 tokens (Section 4.4), and the country bias reported in the model's card (Lesson 7.2). If the feature misbehaves in six months, this record says where to look first, and it makes the decision easy to revisit when a better candidate appears.

Key terms

• Failure analysis: examining the inputs a model gets wrong to find patterns.
• Decision record: a document stating what was chosen, why, on what evidence, and with which known limitations.

Common mistake: recording only the winning accuracy. Without the settings, the exact data, the model commit and the reasons other candidates were rejected, nobody, including you in six months, can check or revisit the decision.

Let's reassemble the selection, and with it Module 7.

We started from a requirement written before any measurement: binary sentiment of English movie reviews, validation accuracy of at least 0.85, p95 latency within 100 ms, at most 600 MB of weights, an allowed licence, and clean training data. We chose three candidates of three kinds: a general sentiment model, one fine-tuned on our exact task, and one from another domain.

Reading the documentation did half the work. The SST-2 model had an Apache 2.0 licence and declared data. The Twitter model had CC-BY 4.0 and three labels from a different domain. The TextAttack model had no licence and unnamed labels, and its train_args.json showed it was trained on IMDb's train split and tuned on its test split. Representative inputs fixed its label mapping (LABEL_1 is positive) and revealed different behaviours on unclear inputs. Measurement on 500 validation reviews gave 0.874, 0.982 and 0.834. Applying the requirement left only the SST-2 model: the TextAttack model's 0.982 was leakage, and the Twitter model fell short. Tuning max_length on validation raised accuracy from 0.873 at 256 tokens to 0.903 at 512 (sign test p = 0.04), within the latency limit, and one look at the test set confirmed 0.888. The leaked model fell from 0.982 to 0.916 on test, and its remaining edge was not significant on 500 reviews (p = 0.087). Finally, failure analysis and a decision record captured what the model gets wrong and why it was chosen.

This is Module 7's answer to its key question, how can AI engineers leverage models that have already learned useful representations? Find candidates by task (7.2–7.3), load and run them correctly (7.4), wrap them in an application that handles messy inputs (7.5), fit them to deployment constraints (7.6), and choose among them with rules, fair measurement and honest records (7.7).

Figure 7.7.2. Requirements, documentation, representative inputs, validation choices and final reporting belong in a recorded process. Repeated selection on a test sample requires a new final test.