The Hugging Face Ecosystem (Essentials)

Let's begin with the whole workflow in three short cells. First, we ask for a model by its repository name, an owner and a model name separated by a slash, just like a GitHub project.

Two lines, two libraries. AutoTokenizer gives us the tokeniser; its fast text-splitting engine comes from the Tokenizers library. AutoModelForSequenceClassification comes from Transformers, and so does the code that builds the network. Both looked up the name on the Hub, downloaded what they needed, and stored it. Now we use them:

The model is sure this one is positive: 1.0 against 0.0, rounded. Where did the downloaded files go?

Every download is stored in a local cache, a folder that by default is ~/.cache/huggingface/hub in your home directory. (You can move it by setting the HF_HOME environment variable, which is what was done when these lessons were built.) The second time you load the same model, nothing is downloaded: the library finds the files in the cache.

That is the whole ecosystem in one picture:

• the Hub stores and publishes models and datasets, with their documentation;
• the libraries (Transformers, Datasets and Tokenizers, with a helper library called huggingface_hub underneath) fetch them and turn them into working Python objects;
• the cache keeps local copies so you download each file only once;
• your code sits on top and decides what to do with the model.

Figure 7.2.1. The Hub and local cache supply compatible artifacts. Tokenizers, Datasets and Transformers serve different stages of the application; cards supply evidence to inspect.

Key terms

• Hugging Face Hub: a website hosting openly shared models, datasets and demos, each in its own repository.
• Repository name: the owner/name identifier of a Hub repository, such as distilbert/distilbert-base-uncased-finetuned-sst-2-english.
• Local cache: the folder where downloaded files are kept so they are not downloaded again.
• Transformers, Datasets, Tokenizers: Hugging Face's main libraries, for models, data and text splitting respectively.

The next section looks at the top layer: what is actually stored in a repository.

A model repository is a folder of files with a history, much like a code repository on GitHub. The huggingface_hub library can list a repository's files and their sizes without downloading anything.

The repository has 17 files, and they fall into four groups.

• The configuration: config.json, the architecture blueprint you read in Lesson 7.1.
• The weights, five times over: model.safetensors, pytorch_model.bin, tf_model.h5, rust_model.ot and onnx/model.onnx are each about 268 MB (267.833 MB, 267.844 MB, 267.950 MB, 267.846 MB and 267.956 MB). They hold the same learned numbers, saved in formats for different software: the safe format we used in 7.1, the older PyTorch format, TensorFlow's format, a Rust library's format, and ONNX, a format for running models outside Python. The onnx/ folder also has its own copy of the tokeniser files, so that version works without the rest of the repository.
• The tokeniser: vocab.txt lists the vocabulary, one token per line, and tokenizer_config.json holds tokeniser settings.
• Documentation: README.md is the model card, and map.jpeg is a picture the card displays. .gitattributes is a bookkeeping file for the version-control system.

Transformers does not download all of this. It picks only the files it needs for PyTorch, and prefers .safetensors when a repository offers it. We'll check exactly what landed on our disk at the end of this section.

The model card is the repository's README.md: a document describing what the model is, how it was trained, what it is for and where it fails. The top of the file holds machine-readable metadata, and the rest is written for people.

The metadata says the model was trained on sst2 (the dataset is also part of a benchmark collection called glue), works on English (en), and is released under the apache-2.0 licence (Section 4). The headings show a well-organised card: details, how to use it, intended uses and misuses, risks, and how it was trained.

Cards can also declare evaluation results in a structured block called model-index. Here are its accuracy entries:

Two numbers, and only one of them is useful. On the glue validation split, a set of SST-2 sentences the model did not train on, the card reports an accuracy of 0.9106. It also reports 0.9886 on its own sst2 training split. A model's score on the data it was trained on says almost nothing about new data, as you saw when diagnosing overfitting in Module 3, so ignore that number. Note too that both numbers are on SST-2's short sentences, not on reviews like IMDb's. Lesson 7.3 measures what that means in practice.

The most valuable part of a good card is often its honest list of problems:

The authors report that the model treats sentences about different countries very differently: 0.89 probability of positive for "This film was filmed in France", but 0.08 for Afghanistan. Nothing in such a sentence says anything good or bad about the film. We will test this claim ourselves in Section 4.

Now let's look in the cache for this repository and see which of the 17 files we actually have.

We have 5 of the 17 files: the configuration, the safetensors weights, the two tokeniser files and the card (fetched just now, when we read it). That's 268.1 MB rather than more than a gigabyte for all five weight formats. The path shows how the cache is organised: one folder per repository (models--owner--name), then a snapshots folder with one subfolder per commit (the 714eb0fa… ID we pinned in Lesson 7.1), then the files.

Figure 7.2.2. A repository can contain documentation, configuration, weights and tokenizer artifacts in several formats. Download only the compatible files required by your framework.

Key terms

• Model repository: a Hub folder holding a model's configuration, weights, tokeniser files and documentation, with version history.
• Model card: the repository's README.md, describing the model, its training, intended uses, limitations and evaluation.
• Card metadata: the machine-readable header of a model card, with fields such as license, language, datasets and model-index.
• ONNX: a file format for running models outside the framework that trained them.

Common mistake: quoting a model card's benchmark number as if it applied to your data. A card's numbers describe its evaluation data; this card's are for short SST-2 sentences. Measure on your own data before relying on them.

With the Hub covered, the next section turns to the libraries that use it.

The Transformers library contains code for hundreds of model architectures. The Auto classes (AutoConfig, AutoTokenizer, AutoModel…) save you from knowing which class each repository needs: they read the repository's config.json, look at its model_type, and build the matching class. Downloading only the configuration (a few kilobytes) is enough to see this:

Four repositories, four different architectures: DistilBERT, GPT-2, BERT and CLIP. Yet the same line of code, AutoConfig.from_pretrained(name), handled all of them. The architectures field records which class the weights were saved from: a sequence classifier, a language model with a word-prediction head (GPT2LMHeadModel), a bare BERT body, and a combined image–text model. The rest of Module 7 loads all four. Lesson 7.4 covers the Auto classes for loading models in detail.

The Datasets library does for data what Transformers does for models: it downloads datasets from the Hub by name, and gives them a common interface. We'll load IMDb's official train and test files, as in Lesson 7.1.

The result is a DatasetDict: a dictionary of splits, each a Dataset with named columns (features) and a number of rows. Each split has 25000 reviews. Let's look inside the training split:

The label column is a ClassLabel with names neg and pos, stored as 0 and 1, and the split is exactly balanced: 12500 of each. Indexing a row gives a plain Python dictionary. (This first review shows the official files' quirks: no cleaning, and the IMDb split from Lesson 7.1, with duplicates removed, is built on top of this raw version.)

How does a library handle datasets far larger than memory? It doesn't load them into memory at all:

The table is a MemoryMappedTable, backed by a 33.4 MB file on disk in the Apache Arrow format, a column-based layout designed for fast reading. Memory mapping means the operating system reads parts of the file only when they are needed, so a dataset bigger than your RAM still opens instantly. This is also why the library writes cache files: every operation produces a new Arrow table on disk.

Datasets works best through a few operations that each produce a new dataset. filter keeps the rows that pass a test:

Only 49 of the 25000 training reviews are under 30 words. You'll see "Filter" progress bars as these operations run.

map applies a function to every row, or with batched=True, to batches of rows at a time, which is much faster. The most common use is tokenising. Here we count how many DistilBERT tokens each review needs, without cutting any off. Expect one warning, "Token indices sequence length is longer than the specified maximum sequence length…": it is the tokeniser pointing out that some reviews are longer than the model can accept. That's true, and exactly what we want to measure.

The median review is 233 tokens long, the longest is 3127, and 44.3% of reviews are over 256 tokens, with 14.8% over 512, the most DistilBERT can read at once. (These are raw reviews; the <br /> tags still in them add a few tokens each.) The histogram shows the long tail:

Most reviews are a few hundred tokens long, but a long tail stretches far past the 512-token limit. Every model in this module has to decide what to do with that tail: cut it off, which is called truncation, or split the review into pieces. Lesson 7.4 covers the choice; the 256-token setting used in Lesson 7.1 cuts off a large share of reviews.

To prepare data for a model, we tokenise with truncation and remove the raw text column. We'll use the first 1,000 reviews to keep this quick:

Each row now holds input_ids (the token IDs), an attention_mask (Lesson 7.4) and a token_type_ids column we'll explain in a moment. Finally, set_format("torch") makes rows come out as PyTorch tensors, and a DataLoader batches them, just as in your own training loops. Because reviews have different lengths, the batches need padding: a data collator is a function that assembles a batch, and DataCollatorWithPadding pads each batch to its longest member.

A batch of 8 reviews, each padded to 256 tokens (the length of the longest review in the batch, which hit the truncation limit), plus 8 labels. The collator also renamed label to labels, the name Transformers models expect when they compute a loss. This dataset is ready for training, which is exactly where Module 8 picks it up.

Figure 7.2.3. Inspect examples, transform them under a declared policy, then dynamically pad and mask each batch. Learned preprocessing must be fitted on training data only.

Key terms

• Dataset / DatasetDict: a table of examples with named columns / a dictionary of such tables, one per split.
• Features: the named, typed columns of a dataset, such as text (a string) and label (a class label).
• Apache Arrow: a column-based file format for fast reading of large tables.
• Memory mapping: reading a file on disk as if it were in memory, fetching only the parts that are used.
• map and filter: Datasets operations that transform every row, or keep only some rows, producing a new dataset.
• Data collator: a function that combines individual examples into a batch, for example by padding them to equal length.

Which tokeniser did AutoTokenizer give us? The answer is a small surprise.

We asked for DistilBERT's tokeniser and got a BertTokenizer. Recall from Section 2.1 that this repository's tokenizer_config.json is tiny: it doesn't name a tokeniser class, so the library fell back to the general BERT tokeniser, which works because DistilBERT uses BERT's vocabulary. The BERT tokeniser also produces token_type_ids, which BERT uses to tell two sentences in one input apart. DistilBERT doesn't use them, and the check above shows its outputs are identical with or without them. Repositories are not all equally tidy, and knowing what each file does helps you spot why a tokeniser behaves unexpectedly.

The Tokenizers library does the actual splitting, in fast compiled code written in Rust. Its most important lesson is one you met in Lesson 6.2: a model and its tokeniser are a pair. Let's tokenise the same sentence with the tokenisers of two different models.

The same 42-character sentence becomes 17 IDs for DistilBERT and 9 for distilgpt2, and the IDs share nothing. DistilBERT's vocabulary has 30,522 entries and distilgpt2's 50,257. The splits differ too:

• DistilBERT lowercases everything (GOOD became good), marks the start and end with [CLS] and [SEP], and marks word-continuation pieces with ##: "unbelievably" became five pieces, un ##bel ##ie ##va ##bly.
• distilgpt2 keeps capitals, adds no special tokens, and marks a piece that starts with a space with Ġ. "unbelievably" is a single token in its larger vocabulary.

The last line makes the point directly: ID 2204 is good to DistilBERT but a completely different piece of text to distilgpt2. Feeding one model's IDs to the other may not raise any error. It silently produces nonsense.

The difference in algorithm shows when we look inside:

Both are "fast" tokenisers, backed by the Tokenizers library. DistilBERT uses WordPiece and distilgpt2 uses BPE (byte-pair encoding, which you built a version of in Lesson 6.2). Before splitting, DistilBERT's tokeniser runs a normaliser that lowercases and strips accents: "Café FÊTE" becomes 'cafe fete'. This is what "uncased" means in the model's name.

One more feature of fast tokenisers is invaluable when you build applications: offset mapping, which records which characters of the original text each token came from.

Each token knows its source characters, even after lowercasing: the token good maps back to characters 37–41, which read 'GOOD' in the original. Special tokens like [CLS] map to nothing (0–0). Offsets let you highlight words in the original text, for example to show a user which part of a review a model reacted to.

Key terms

• Auto classes: Transformers classes (AutoConfig, AutoTokenizer, AutoModel…) that read a repository's configuration and build the matching class.
• Fast tokeniser: a tokeniser backed by the compiled Tokenizers library.
• WordPiece and BPE: two subword algorithms; WordPiece marks continuation pieces with ##, GPT-2's BPE marks pieces that begin with a space with Ġ.
• Normaliser: the step before splitting that cleans text, for example by lowercasing and removing accents.
• Offset mapping: for each token, the start and end positions of the characters it came from.

Common mistake: loading a tokeniser from one repository and a model from another. The shapes may match, so nothing crashes, but the IDs mean different things to each, and the predictions are meaningless. Load both from the same repository name.

Knowing how to load a model is half the job. The other half is knowing whether you're allowed to, and whether you should.

Every model is someone's work, released under some terms. A licence states what you may do with it: use it commercially, change it, redistribute it, and under what conditions. On the Hub, the licence is usually declared in the card's metadata, so we can check it in code.

Three different situations appear in this table.

• A standard open licence. Four models use apache-2.0, a widely used permissive licence: you may use, change and redistribute the model, including commercially, as long as you keep the licence and copyright notices.
• A custom licence with gated access. meta-llama/Llama-3.2-1B and google/gemma-2-2b declare their own licences, llama3.2 and gemma. These are custom terms written by the model's publisher, with their own conditions, for example on how the model may be used. Both are gated (manual): you must sign in, read and accept the terms on the Hub page, and wait for approval before downloading.
• No declared licence. openai/clip-vit-base-patch32 shows None.

Let's see what access looks like for a gated model without an account:

The Hub refuses: a GatedRepoError with HTTP status 401, "not authorised". The metadata was public (we read the licence above), but the files are not until you accept the terms. To use such a model, you would log in with an access token (hf auth login) after accepting its terms on the website. Nothing in Modules 7 and 8 needs one.

What about CLIP's missing licence?

The card's metadata contains only a tag and a demo-widget setting, and no line of the card mentions a licence at all. That doesn't mean the model is free to use. In most countries, work published without a licence gives you no automatic right to reuse it. The card does say a lot about intended use: its headings include "Primary intended uses" and "Out-of-Scope Use Cases". What to do in practice is to look for the licence where the model was originally published (the authors' own code repository), use the model only for the purposes its creators describe, and get advice before building a product on it. Lesson 7.7 makes this part of a model-selection checklist. In these lessons we use CLIP for learning, which is within its stated research purpose.

A licence tells you what you may do; the card's limitations tell you what you should worry about. Section 2.2's card warned that sentences about different countries get very different sentiment. Let's test that claim with six countries, chosen before running the cell.

The missing Hub metadata field is not proof that the original project has no licence. The original CLIP repository includes an MIT licence; verify the terms and provenance applicable to the exact weights and distribution you intend to use. Licence terms and intended-use recommendations are different questions.

The warning holds on today's model. "This film was filmed in Afghanistan." gets a positive probability of 0.165, against 0.965 for France and 0.963 for the United States. Nigeria gets 0.736. The exact values differ from the card's (0.89 and 0.08), which is a good reason to measure rather than copy, but the pattern is the same: the model's judgement of a neutral sentence depends on the country named in it. If this model screened reviews written about films from around the world, some would be judged more harshly purely because of where they were made.

Responsible selection means doing this kind of check as a matter of routine:

1. read the licence and any usage restrictions;
2. read the card's intended uses, out-of-scope uses and limitations;
3. test the warnings that matter for your application, on inputs like your users' inputs;
4. record what you found, and decide whether the model is acceptable for the use you have in mind.

Key terms

• Licence: the legal terms under which a model may be used, changed and shared.
• Permissive licence: a licence such as Apache 2.0 that allows broad use, including commercial use, with few conditions.
• Custom licence: terms written by a model's publisher for that model, often with specific use restrictions.
• Gated model: a model whose files can be downloaded only after you sign in and accept its terms.
• Intended use / out-of-scope use: what a model's creators designed it for, and uses they advise against.

Common mistake: treating "downloadable" as "allowed". A model can be public on the Hub and still carry a licence that forbids your use, or carry no licence at all. Check the terms before you build on a model, not after.

Let's reassemble the ecosystem.

When you write AutoModelForSequenceClassification.from_pretrained("owner/name"), you are working with every layer at once. The Hub stores the model's repository: 17 files for the model we studied, including its configuration, the same weights in five formats, its tokeniser files and its model card. The huggingface_hub library downloads only what your framework needs, 5 files and 268.1 MB here, into the local cache, where they are kept by repository and commit. Transformers reads the configuration and its Auto classes build the right architecture, whether that is DistilBERT, GPT-2, BERT or CLIP. Tokenizers turns text into token IDs, and every model needs its own tokeniser: the same sentence was 17 IDs for DistilBERT and 9 for distilgpt2. Datasets loads data in memory-mapped Arrow tables and turns it into model-ready batches through filter, map, set_format and a data collator. Along the way we measured that 14.8% of IMDb reviews are longer than DistilBERT's 512-token limit.

Around the code sit two questions that no library answers for you. Is the model good enough for my data? The card's 0.9106 accuracy is on SST-2 sentences, not your data. Am I allowed to use it, and should I? Licences ranged from permissive (apache-2.0) to custom and gated (llama3.2, gemma) to missing (CLIP). The card's bias warning held when we tested it: 0.965 for France against 0.165 for Afghanistan, for the same neutral sentence.

Figure 7.2.4. The repository name resolves to artifacts; the tokenizer prepares IDs and masks, and compatible model code loads the architecture and weights before inference.

This appendix collects the parts of the Hub website and the matching Python calls that you are most likely to need. The website changes its layout from time to time, but these elements have been stable.

Finding models (huggingface.co/models)

On a model's page

Downloading and the cache

Datasets have the same structure under huggingface.co/datasets, with a Dataset viewer that shows rows in the browser, and load_dataset(repo_id) to load them.