Finding and Selecting the Right Model

A common way to pick a model is to search for something popular and try it. Sometimes it works. More often it wastes an afternoon on a model that was never right for the job. A better habit is to describe the problem first, in a few concrete questions, and let the answers filter the search:

1. What is the task? What goes in and what must come out: a label for a whole text, a label for each word, a span copied from a passage, new text, a vector, a label for an image? This decides the kind of model (Section 2).
2. What do the inputs look like? Language, domain (tweets, legal contracts, film reviews), and above all length. A model that reads 512 tokens can't read a 20-page document in one go (Section 3).
3. What hardware will run it, and how fast must it answer? This sets an upper limit on model size (Section 3).
4. How good must it be, measured how? And on whose data? A benchmark number is evidence, not a guarantee (Section 4).
5. Where will it run, and what are you allowed to do with it? On your own machine or through someone else's service (Section 5), and under what licence (Lesson 7.2).

Only after these questions does it make sense to look at specific models. The questions turn "find a good model" into "find a model that does this task, on inputs like these, within this budget, this well, under these terms", which is far easier to answer.

Figure 7.3.1. Requirements narrow a shortlist; documentation and representative evaluation determine whether a candidate actually fits. Popularity and parameter count alone do not decide.

The first filter is the task itself, so that is where we start.

Every model on the Hub is tagged with the task it was built for, its pipeline tag, and the Hub's search can filter by it. Here are seven well-known models, one per common task, with the tag the Hub gives each:

These tags are the vocabulary of model search. What each means in practice:

Notice how the same architecture serves different tasks. The first two models are both DistilBERT, one fine-tuned for classification and one for question answering. What makes them different is the head and the fine-tuning (Lesson 7.1, Section 4). Let's run three of these tasks with small models. Question answering and summarisation stay on the map only, to keep downloads down, and the image tasks get their turn in Lesson 7.5.

Text classification, which you already know:

Text generation with distilgpt2, a small version of GPT-2. With do_sample=False, the pipeline always picks the most likely next token: greedy decoding, from Lesson 6.6.

The model continues the prompt fluently but falls into a loop: "It's a very good movie" twice in 20 tokens. Greedy decoding often repeats like this, exactly as you saw with your own model in Lesson 6.6. Lesson 7.5 tries the sampling strategies that fix it.

Embeddings with all-MiniLM-L6-v2, a small model trained so that sentences with similar meanings get similar vectors. The pipeline returns one vector per token; averaging them gives one vector per sentence.

Each sentence becomes a vector of 384 numbers. The two film sentences share almost no words ("thrilling, beautifully acted film" against "exciting movie with superb performances"), yet their cosine similarity is 0.734, while each scores about zero against the sentence about a parcel (−0.007 and 0.008). The model compares meanings, not words, which is what makes semantic search possible. Lesson 7.5 builds a search engine on it.

Key terms

• Pipeline tag (task tag): the Hub's label for the task a model was built for, such as text-classification; it is also the name you pass to pipeline(...).
• Greedy decoding: generating text by always choosing the single most likely next token.
• Sentence embedding: one vector representing a whole sentence, built so that similar meanings give nearby vectors.

Knowing the task narrows the search to a few kinds of model. Within a kind, the next filter is whether a model fits your hardware and your inputs.

You can find out how big a model is without downloading it. A .safetensors file begins with a short header that lists every tensor's name, shape and number type, and huggingface_hub can read just that header. Combined with each model's configuration (a few kilobytes), it tells us how many numbers a model stores and how long an input it accepts.

That's a size ladder spanning several hundred times, from 22,713,728 stored numbers (MiniLM) to 7,615,616,512 (Qwen2.5-7B, a general-purpose language model).

Three things to read from this table:

• "Stored as" matters. The BERT models store 32-bit floats (F32, 4 bytes each). The Qwen models store BF16 (bfloat16), a 16-bit format, 2 bytes each. So Qwen2.5-0.5B, with more numbers than BERT-large, takes less disk: 0.99 GB against 1.34 GB. MiniLM also stores a few integers (I64), which are position indexes rather than learned weights. (The header counts everything stored, so its total can differ slightly from the number of trainable parameters.)
• Context length varies enormously. All four BERT-style encoders accept at most 512 tokens. The Qwen models accept 32,768 and 131,072. For long documents, that alone can decide the choice.
• Disk size predicts memory. To run a model you must at least hold its weights in memory, so the file size is a lower bound on the memory you need. Running it needs more: activations, and for generation, the cache of past keys and values (Lesson 7.6).

Using the rule from Lesson 7.1, parameters × bytes per parameter, we can estimate the weight memory in different number formats:

Qwen2.5-7B needs 30.46 GB just for its weights in float32, or 15.23 GB in a 16-bit format. On a laptop with 16 GB of RAM, even the 16-bit version leaves almost nothing for the operating system and for the computation itself. The small models on this ladder take well under a gigabyte. This kind of arithmetic takes seconds and saves you from downloading 15.23 GB only to watch your machine run out of memory. Lesson 7.6 covers ways to shrink these numbers, such as quantisation.

How does input length affect cost? In Lesson 6.3 you saw that attention compares every token with every other token, so its cost grows with the square of the length. But a Transformer layer also contains linear layers, the Q, K, V and output projections and the feed-forward network, whose cost grows only in proportion to the length. Which part dominates depends on the length. We can count the multiplications per token in one DistilBERT layer:

• the linear parts need about 4·d² + 2·d·d_ff, with d = 768 and d_ff = 3072, the same for every token;
• attention needs about 2·L·d per token: one dot product with each of the L positions to get the scores, and one weighted sum over L value vectors.

At 128 tokens, attention is only 2.7% of the work; at 512 tokens, the most DistilBERT accepts, it's 10.0%. The two parts become equal only at 4608 tokens, and attention dominates beyond that (64.0% at 8192). So for short-context models like DistilBERT, cost grows roughly in proportion to length. For long-context models like the Qwen family, which accept inputs of tens of thousands of tokens, the quadratic part takes over. That is why very long inputs are so expensive, and why so much engineering goes into making attention cheaper.

Let's check the prediction by timing the SST-2 model on the same long text, truncated to different lengths, in batches of 16:

Interpret the evidence carefully. The 4,608-token crossover is from this simplified multiplication count for DistilBERT's width and feed-forward expansion. It is not a universal Transformer boundary or a measured runtime crossover. Configured position limits also do not guarantee equal quality at every length.

The measured time per review rises in a near-straight line with length: each doubling of the token count roughly doubles the time. That's what the multiplication count predicted for this range, where the linear parts dominate. (Your timings will differ from the ones printed here, since they depend on your processor, but the shape of the curve should be the same.) Two practical consequences follow. Truncating reviews to 256 tokens instead of 512 roughly halves inference cost. And if your inputs are long, context length and compute have to be judged together: a model that can read 32,768 tokens will also charge you for reading them.

Key terms

• Context length (context window): the maximum number of tokens a model can take as input at once.
• bfloat16 (BF16): a 16-bit number format, 2 bytes per number, widely used for storing and running large models.
• Weight memory: the memory needed just to hold a model's parameters: parameters × bytes per parameter.
• Truncation: cutting an input down to a maximum number of tokens.

Common mistake: choosing a model by parameter count alone. Two models with similar parameter counts can differ in storage format (float32 or bfloat16), context length and speed. Check the file sizes, the number types and the context length too.

A model that fits your task and your hardware still has to be good enough. That is what model cards and benchmarks claim to tell you.

A benchmark is a fixed, public dataset with an agreed way of scoring, so that different models can be compared on equal terms. SST-2 is one such benchmark, and it is part of a larger collection called GLUE. Our SST-2 model's card makes its claim in two places:

The card's text says "an accuracy of 91.3 on the dev set" (dev set is another name for validation set). Its metadata says 0.9106 on the GLUE validation split. These don't quite agree: 91.3 in the text against 0.9106 in the metadata. Small inconsistencies like this are common, because card text and metadata are often written at different times or from different runs. They are a first hint to verify rather than copy.

Let's verify it ourselves. The SST-2 validation set is small and public:

The validation set has 872 sentences. Look at the first one: it is lowercase, and there are spaces before the punctuation (it 's, journey .). Benchmark text has often been preprocessed, and may look quite different from what your users will type. Now we run the model on all 872:

We get 0.9106, exactly the figure in the card's metadata. So the claim is honest, for SST-2. That's the first step of critical reading: confirm that a claim can be reproduced on its own terms. The next step is to ask whether it holds on yours.

Our data is IMDb, with the fixed split from Lesson 7.1. The function is copied here so this notebook stands alone, and the fingerprints confirm it gives exactly the same reviews.

The fingerprints match Lesson 7.1's. Now the same model on the 1,000 val_eval reviews, truncated to 256 tokens:

Accuracy drops from 0.9106 on SST-2 to 0.8820 on IMDb validation. Same task (positive or negative), same domain (films), a few points lower. Why? The obvious difference is length:

The median SST-2 sentence is 24 tokens long; the median IMDb review is 214. The model was fine-tuned on single sentences and is now reading whole reviews, often with mixed opinions: a reviewer may praise the acting and still hate the film. On the 601 reviews that fit within 256 tokens, accuracy is 0.9068, close to the card's figure. On the 399 that were cut off, it is 0.8446.

Be careful about what this does and doesn't show. It's tempting to say "truncation costs about six points", but the two groups differ in more than truncation: long reviews may also be more nuanced, more sarcastic or simply harder. This comparison shows that the model does worse on long reviews; it doesn't separate why. Lesson 7.7 measures the effect of the truncation length directly, by running the same reviews at different lengths.

There's one more question to ask of any benchmark number: could the model have seen the test data during training? If so, its score measures memory, not ability. This is called contamination, and it can happen at either training stage.

DistilBERT was pretrained on bookcorpus (a collection of books) and wikipedia, so IMDb reviews are unlikely to be in its pretraining data, and it was fine-tuned on SST-2's training split, not on its validation split. For this model, both our measurements are clean. But the question needs asking every time:

• Models pretrained on large web crawls may have read IMDb pages, and popular benchmarks themselves often end up in crawled data. A suspiciously high score on a famous dataset is a warning sign.
• Models fine-tuned on a dataset will score well on any split they were trained on. If you carve your validation set from IMDb's training half, as we did, a model fine-tuned on IMDb's training half has seen it. Lesson 7.7 meets exactly this case.

Read the card's training-data section, and if it doesn't say what the model was trained on, treat its benchmark scores with suspicion.

Figure 7.3.2. The card text, structured metadata, and reproduced SST-2 score refer to a different evaluation context from the IMDb run. All displayed numbers come from the source reference results.

Key terms

• Benchmark: a fixed public dataset and scoring rule used to compare models.
• GLUE: a widely used collection of English language-understanding benchmarks, including SST-2.
• Dev set: another name for a validation set.
• Domain shift (distribution shift): a difference between the data a model was trained or evaluated on and the data it meets in use.
• Contamination: test or benchmark data appearing in a model's training data, which inflates its scores.

Common mistake: treating a lower score on your data as a sign that you made a mistake. A drop from a card's benchmark to your data is normal; it measures the distance between the two datasets. Verify your pipeline on the benchmark itself, as we did, and then believe your own number.

So far every model has run on our own computer. The last filter is where the model should run at all.

There are two ways to use a pretrained model:

• Locally: you download the weights and run them on hardware you control, a laptop, a server or a cloud machine you rent. Everything in this module so far has been local.
• Through a hosted API: someone else runs the model on their servers. Your code sends the input over the internet and receives the output. Hugging Face, the major AI labs and cloud providers all offer this. An API (application programming interface) is simply a defined way for one program to ask another for something.

A hosted call looks like this. The listing is shown but not run, because it needs an account and an access token, a secret key that identifies you:

from huggingface_hub import InferenceClient

client = InferenceClient(token="hf_...")        # your access token, from your Hugging Face account settings
result = client.text_classification(
    "The soundtrack alone is worth the price of the ticket.",
    model="distilbert/distilbert-base-uncased-finetuned-sst-2-english",
)
print(result)                                   # labels and scores, computed on a remote server

Which models are available this way, from which providers and at what price changes often, so check the provider's current documentation. The trade-offs, though, are stable:

Neither is better in general. A prototype, or a feature that needs a model far bigger than your hardware, often starts on a hosted API. A product with steady high volume, strict privacy rules or a need for exactly reproducible behaviour often ends up local. Many teams use both: a hosted model to explore, and a local model once they know what they need. Lesson 7.6 returns to this choice with measured costs.

Key terms

• Hosted API: a service that runs a model on the provider's servers and returns results to your code over the internet.
• Access token: a secret key that identifies you to a service; keep it out of notebooks and code you share.
• Rate limit: a cap on how many requests a service accepts from you in a given time.

Common mistake: pasting an access token directly into a notebook and then sharing the notebook. Anyone who reads it can use your account. Store tokens in an environment variable, or log in once with hf auth login, and keep them out of your code.

Let's reassemble the selection method.

Start from the problem, not the model. Five questions (task, inputs, hardware and speed, quality on whose data, deployment and licence) filter the Hub's huge catalogue down to a shortlist.

The task maps to a pipeline tag: classification, question answering, summarisation, generation, embeddings, image classification or zero-shot image classification. Models of the same architecture family can serve different tasks with different heads. We ran three: the classifier, a generator that fell into a loop under greedy decoding, and an embedding model that rated two film sentences with no words in common at 0.734 similarity.

Size and context can be checked before downloading. Safetensors headers gave stored-number counts from 22,713,728 to 7,615,616,512, and parameters × bytes turned them into memory: Qwen2.5-7B needs 15.23 GB in a 16-bit format, too much for a typical laptop. Context lengths ranged from 512 to 131,072 tokens. For a 512-token encoder, attention is only 10.0% of the multiplications, so cost grows roughly in proportion to length, as our timings confirmed. Beyond 4608 tokens, attention's quadratic cost dominates.

Quality claims need two checks. First, reproduce the claim on its own terms: our run on SST-2 validation gave 0.9106, matching the card's metadata (though not its text, which says 91.3). Then measure on your data: on IMDb validation the same model scored 0.8820, with 0.9068 on reviews that fit in 256 tokens and 0.8446 on those that didn't. And always ask what the model saw during training, because contamination inflates scores.

Where it runs is the final choice: local models give control, privacy and fixed costs; hosted APIs give convenience and access to bigger models, at a per-use cost and with your data leaving your hands.

Figure 7.3.3. Check interface, parameter storage versus runtime memory, context, benchmark protocol, training exposure and deployment terms before adopting a model.