Glossary — Deep Learning Lab

`AutoModelForSequenceClassification`: The Auto class for a model with a whole-text classification head.; Taught in 7.4 On the map
`id2label`: The configuration's map from output position to label name.; Taught in 7.4 On the map
`max_length`: The maximum number of tokens kept when truncating.; Taught in 7.4 On the map
`num_labels` / `id2label` / `label2id`: Configuration settings for the number of classes and their names.; Taught in 8.4 On the map
`save_pretrained`: Saves a model or tokeniser to a folder in Hub format.; Taught in 7.4 On the map
`save_to_disk`: Saves a Hugging Face dataset in Arrow format.; Taught in 8.2 On the map
`top_k`: How many top labels a classification pipeline returns; None returns all.; Taught in 7.4 On the map
`torch.inference_mode()`: A context manager that disables gradient recording for faster inference.; Taught in 7.4 On the map
`Trainer` / `TrainingArguments`: Hugging Face's ready-made training loop and its settings.; Taught in 8.4 On the map

Abstention: Declining to answer, usually because confidence is low.; Taught in 7.5 · used in7.5 On the map
Access token: A secret key that identifies you to a service.; Taught in 7.3 · used in7.2 · 7.3 On the map
Activation function: A simple non-linear function (such as ReLU, sigmoid or tanh) applied to each neuron's weighted sum. Without it, stacked layers collapse into a single linear model.; Taught in 1.2 · used in1.2 · 1.3 · 3.2 · 5.1 · 6.1 · 6.3 · 6.4 · 8.5 On the map
Adam: A popular optimizer that gives every parameter its own adaptive step size, using running averages of recent gradients and their squares. AdamW is Adam with weight decay applied correctly.; Taught in 3.1 · used in1.1 · 1.2 · 1.3 · 2.3 · 2.4 · 3.1 · 3.2 · 3.3 · … On the map
AdamW: Adam with decoupled weight decay; the standard fine-tuning optimiser.; Taught in 8.4 · used in1.3 · 3.1 · 3.2 · 3.3 · 3.4 · 4.2 · 4.3 · 4.4 · … On the map
Adaptation: Making a general pretrained model better at a specific task.; Taught in 8.1 · used in7.1 · 8.1 · 8.3 · 8.5 · 8.7 On the map
Adaptation spectrum: The range of adaptation methods, ordered by how much of the model they change.; Taught in 8.1 · used in8.1 · 8.7 On the map
Adapter (PEFT): The saved set of trained PEFT weights for one task.; Taught in 8.5 · used in8.5 On the map
Apache Arrow: A column-based file format for fast reading of large tables.; Taught in 7.2 · used in7.2 On the map
Application layer: The code around a model: input checks, fit checks, decision rules, error handling.; Taught in 7.5 · used in7.5 On the map
Architecture: The design of a network — which layers, how many, how wide, and how they connect. It encodes built-in assumptions about the data.; Taught in 2.3 · used in1.1 · 1.3 · 2.3 · 2.4 · 3.1 · 3.2 · 3.3 · 3.4 · … On the map
Attention mask: 1s for real tokens and 0s for padding, so attention ignores padding.; Taught in 7.4 · used in7.4 · 8.2 · 8.4 · 8.6 On the map
Auto classes: Classes that read a configuration and build the matching architecture.; Taught in 7.2 · used in7.2 · 7.4 · 7.7 On the map
Autoregressive: Predicting each next item conditional on earlier items. Known training positions can be scored in parallel with a causal mask, while ordinary generation selects new tokens sequentially.; Taught in 6.6 · used in5.4 · 6.1 · 6.6 On the map

Backpropagation: The algorithm that computes the gradient of the loss with respect to every parameter, by applying the chain rule backwards through the computational graph. It assigns "credit" or "blame" to each parameter.; Taught in 1.2 · used in1.1 · 1.2 · 1.3 · 2.3 · 3.1 · 3.3 · 3.4 · 4.2 · … On the map
Base model: The model produced by pretraining, before adaptation to any task.; Taught in 7.1 · used in4.3 · 7.1 · 8.1 · 8.4 · 8.5 · 8.6 On the map
Baseline: A simple reference result (e.g. always predicting the majority class, or a logistic regression) that any serious model must beat to be worth its cost.; Taught in 2.3 · used in2.2 · 2.3 · 2.4 · 3.4 · 4.2 · 4.4 · 5.3 · 6.4 · … On the map
Baseline ladder: A set of baselines of increasing sophistication, measured identically.; Taught in 8.3 · used in8.3 On the map
Batch: A small group of training examples processed together in one forward and backward pass. The batch size is how many examples it contains.; Taught in 2.2 · used in1.2 · 1.3 · 2.1 · 2.2 · 2.3 · 2.4 · 3.1 · 3.2 · … On the map
Batch size: How many inputs are processed together in one forward pass.; Taught in 7.4 · used in2.1 · 2.2 · 2.3 · 2.4 · 3.1 · 3.3 · 4.2 · 4.3 · … On the map
Benchmark: A fixed public dataset and scoring rule for comparing models.; Taught in 7.3 · used in7.2 · 7.3 · 7.7 · 8.2 On the map
bfloat16 (BF16): A 16-bit number format, 2 bytes per number.; Taught in 7.3 · used in7.3 On the map
Bias (parameter): A learned number added to a neuron's weighted sum, letting it shift its output independently of the inputs — like the intercept in linear regression.; Taught in 1.2 · used in1.1 · 1.2 · 1.3 · 2.1 · 2.3 · 3.3 · 5.2 · 5.4 · … On the map
Blockwise quantisation: Quantising small blocks of numbers, each with its own scale.; Taught in 8.6 · used in8.6 On the map
Body (encoder): The shared part of a model that builds representations.; Taught in 7.1 · used in7.1 On the map
Bottleneck adapter: A small down/up projection inserted into each layer.; Taught in 8.5 · used in8.5 On the map
BPE (byte-pair encoding): A subword algorithm that merges frequent character pairs; used by GPT-2.; Taught in 7.2 · used in7.2 On the map
Break-even volume: The volume at which self-hosting and a hosted service cost the same.; Taught in 7.6 · used in7.6 On the map
Broadcasting: The rule that lets tensors of different shapes be combined element-wise by virtually stretching size-1 dimensions. Powerful, and a source of silent bugs when shapes are not what you think.; Taught in 2.1 · used in1.2 · 1.3 · 2.1 · 2.4 · 6.5 On the map
Byte-pair encoding: Tokenisation by repeatedly merging frequent adjacent pairs. Character-based versions require known characters; byte-level coverage can provide a fallback for arbitrary text.; Taught in 6.2 · used in6.2 · 7.2 On the map

Cache hit rate: The share of requests served from the cache.; Taught in 7.6 · used in7.6 On the map
Cache key: The information identifying a cached answer: input, model version, settings.; Taught in 7.6 · used in7.6 On the map
Candidate: A model shortlisted for testing.; Taught in 7.7 · used in5.4 · 5.5 · 7.3 · 7.5 · 7.7 On the map
Capability: What a model computes from an input.; Taught in 7.5 · used in3.4 · 7.5 · 8.1 · 8.7 On the map
Card metadata: Machine-readable fields at the top of a model card, such as license and model-index.; Taught in 7.2 · used in7.2 · 7.3 On the map
Catastrophic forgetting: Losing earlier abilities when trained on a new task.; Taught in 8.1 · used in8.1 · 8.4 On the map
Causal LM loss: Cross-entropy of predicting each token from those before it.; Taught in 8.6 · used in8.6 On the map
Causal mask: A restriction blocking future keys for a query position, so next-token prediction cannot read its answer. Padding restrictions are a separate mask concern.; Taught in 6.3 · used in6.1 · 6.2 · 6.3 · 6.5 · 6.6 · 7.4 On the map
Chat template: The special-marker format an instruction-tuned model expects around messages.; Taught in 8.1 · used in8.1 · 8.2 · 8.6 On the map
Checkpoint: A saved snapshot of training — model weights, optimizer state, epoch and config — from which you can resume or reproduce a run.; Taught in 2.3 · used in1.3 · 2.3 · 2.4 · 3.1 · 3.2 · 3.4 · 4.1 · 4.2 · … On the map
Checkpoint selection: Keeping the checkpoint with the best validation score.; Taught in 8.4 · used in3.4 · 8.4 · 8.7 On the map
Class imbalance: When one class is much rarer than another (e.g. 11.7% positives). Plain accuracy becomes misleading, so metrics like ROC-AUC and average precision are used.; Taught in 2.3 · used in1.1 · 2.3 · 4.1 · 8.2 On the map
Classification head: The task-specific layer(s) on top of a pretrained body.; Taught in 8.1 · used in7.4 · 8.1 · 8.4 · 8.5 On the map
Cleaning: Removing content with no signal for the task.; Taught in 8.2 · used in7.2 · 8.2 On the map
CLIP: Contrastive Language–Image Pretraining: paired image and text encoders trained to produce comparable projected representations. Similarity supports retrieval and candidate-based classification rather than directly generating an image.; Taught in 5.5 · used in1.2 · 1.3 · 3.3 · 4.4 · 5.5 · 7.1 · 7.2 · 7.5 · … On the map
Commit / revision: A recorded change to a repository / the version you ask for when loading.; Taught in 7.1 · used in7.1 On the map
Computational graph: The record of every operation used to compute the loss from the inputs and parameters. Backpropagation walks it backwards to compute gradients.; Taught in 1.2 · used in1.2 · 1.3 On the map
Confidence: The probability a classifier gives to its chosen class.; Taught in 7.5 · used in1.2 · 1.3 · 2.3 · 3.2 · 5.5 · 7.1 · 7.4 · 7.5 · … On the map
Confidence interval: An interval from a method designed to cover the target quantity at a stated rate over repeated samples; this is not a probability assigned to the fixed unknown value.; Taught in 8.7 · used in3.2 · 8.7 On the map
Configuration: A file describing a model's architecture (and labels) so a library can rebuild it.; Taught in 7.1 · used in1.3 · 2.2 · 2.3 · 2.4 · 3.2 · 3.4 · 7.1 · 7.2 · … On the map
Confirmatory test: A single use of the test set after all choices are made.; Taught in 7.7 · used in7.7 On the map
Confusion matrix: A table counting each combination of true and predicted label.; Taught in 7.5 · used in7.5 · 8.2 On the map
Contamination: Evaluation data appearing in a model's training data, inflating its scores.; Taught in 7.3 · used in7.3 · 7.7 · 8.2 On the map
Context length: The maximum number of tokens a model accepts at once.; Taught in 7.3 · used in6.2 · 7.3 · 8.5 On the map
Context window: The configured amount of sequence context a model can use under its architecture and implementation. A permitted length does not guarantee equal prediction quality at every position.; Taught in 6.2 · used in4.3 · 6.2 · 6.3 · 6.5 · 6.6 · 7.3 On the map
Contextual representation: A word vector that depends on the surrounding sentence.; Taught in 7.1 · used in5.3 · 6.1 · 6.3 · 7.1 On the map
Contrastive learning: Learning representations by favouring matched pairs relative to selected alternatives. What counts as a positive or negative pair depends on the training objective and data.; Taught in 5.5 · used in5.5 · 5.6 On the map
Convolutional neural network: A network that applies learned local filters with shared weights across spatial locations. It provides a useful inductive bias for many image tasks.; Taught in 5.2 · used in3.3 · 5.1 · 5.2 · 5.5 · 5.6 · 6.1 · 6.4 · 7.5 On the map
Cosine similarity: The dot product of two nonzero vectors divided by their lengths. It compares direction, lies between −1 and 1, and is not a calibrated probability of semantic equivalence.; Taught in 5.3 · used in2.2 · 5.3 · 5.4 · 5.5 · 6.1 · 6.2 · 6.3 · 7.1 · … On the map
Coverage: The share of inputs the system answers.; Taught in 7.5 · used in4.3 · 5.2 · 6.6 · 7.5 On the map
Cross-attention: Attention with queries from one sequence and keys and values from another, such as decoder target vectors reading encoded source memory.; Taught in 6.5 · used in6.1 · 6.5 On the map
Cross-entropy loss: Minus the log-probability of the correct class, averaged over a batch.; Taught in 8.4 · used in1.2 · 3.1 · 5.3 · 6.5 · 6.6 · 8.4 On the map
CrossEntropyLoss: The standard loss for classification. It heavily penalises confident wrong predictions and expects integer class labels (long) in PyTorch.; Taught in 1.2 · used in1.2 · 1.3 · 2.1 · 2.3 · 3.4 · 5.3 · 5.5 · 6.5 · … On the map
Custom licence: Terms written by a model's publisher specifically for that model.; Taught in 7.2 · used in7.2 On the map

Data collator: A function that combines examples into a batch, for example by padding them.; Taught in 7.2 · used in7.2 · 8.2 · 8.4 On the map
DataLoader: A PyTorch utility that takes a Dataset and serves shuffled mini-batches, optionally in parallel worker processes.; Taught in 2.2 · used in2.2 · 4.2 On the map
Dataset (PyTorch): In PyTorch, an object that knows its length (__len__) and how to return one example (__getitem__).; Taught in 2.2 On the map
Dataset / DatasetDict: A table of examples / a dictionary of tables, one per split.; Taught in 7.2 · used in7.2 On the map
Dataset card: The documentation page of a dataset on the Hub.; Taught in 8.2 · used in8.2 On the map
Datasets: Hugging Face's library for loading and processing datasets.; Taught in 7.2 · used in1.1 · 2.2 · 3.2 · 3.4 · 4.4 · 5.1 · 5.3 · 5.4 · … On the map
Decision boundary: The border in input space where a classifier's prediction switches from one class to another.; Taught in 1.2 · used in1.1 · 1.2 On the map
Decision record: A document of what was chosen, why, on what evidence, with which limitations.; Taught in 7.7 · used in7.7 · 8.7 On the map
Decision rule: The procedure, fixed in advance, for choosing among candidates.; Taught in 7.7 · used in2.3 · 3.4 · 4.4 · 7.5 · 7.7 On the map
Degenerate repetition: A generator getting stuck repeating words or tokens.; Taught in 7.5 · used in7.5 On the map
Deployment constraint: A practical limit a deployed model must meet, such as latency, memory or cost.; Taught in 7.6 · used in7.6 On the map
Dequantisation: Converting quantised values back to higher precision.; Taught in 8.6 · used in8.6 On the map
Deterministic inference: Exactly the same output for the same input every time.; Taught in 7.6 · used in7.6 On the map
Dev set: Another name for a validation set.; Taught in 7.3 · used in7.3 On the map
Device placement: Putting model weights and inputs on the same device.; Taught in 7.4 · used in7.4 On the map
Distribution shift: When the data a model sees in use differs from the data it was trained and validated on. Covariate shift changes the inputs; label shift changes how often each outcome occurs.; Taught in 4.1 · used in4.1 · 4.4 · 7.3 On the map
Domain shift: A difference between the data a model was built on and the data it meets in use.; Taught in 7.3 · used in7.3 On the map
Double quantisation: Quantising the quantisation scales themselves.; Taught in 8.6 · used in8.6 On the map
Downstream task: The task you actually want to solve with a pretrained model.; Taught in 7.1 · used in5.1 · 7.1 On the map
Dropout: A regulariser that randomly zeroes a fraction of activations during training (only), forcing the network not to rely on any single unit.; Taught in 3.2 · used in1.3 · 2.3 · 2.4 · 3.1 · 3.2 · 3.4 · 4.2 · 6.6 · … On the map
dtype: A tensor's data type, such as float32, float16 or int64 (long). Layers and loss functions expect specific dtypes.; Taught in 2.1 · used in1.3 · 2.1 · 2.2 · 2.3 · 2.4 · 7.6 On the map
Dynamic batching: Grouping requests that arrive close together into one batch.; Taught in 7.6 · used in7.6 On the map

Early stopping: Stopping training (and keeping the best checkpoint) when validation performance stops improving, to avoid overfitting.; Taught in 3.2 · used in1.3 · 2.3 · 2.4 · 3.1 · 3.2 · 3.4 · 4.2 · 4.4 On the map
Embedding: A learned vector of numbers that represents an item (a word, a user, an image) so that similar items get similar vectors.; Taught in 2.2 · used in1.3 · 2.1 · 2.2 · 4.4 · 5.1 · 5.3 · 5.4 · 5.5 · … On the map
Encoder: A model component that transforms an input into a representation used by another component or task.; Taught in 5.1 · used in2.1 · 4.1 · 4.2 · 5.1 · 5.2 · 5.5 · 5.6 · 6.1 · … On the map
Engineered feature: A feature whose formula a person invented and wrote in code, using domain knowledge — for example a ratio or a log transform.; Taught in 1.1 · used in1.1 · 1.3 On the map
Epoch: One complete pass over the whole training set.; Taught in 1.3 · used in1.2 · 1.3 · 2.1 · 2.2 · 2.3 · 2.4 · 3.1 · 3.2 · … On the map
Error analysis: Examining which examples a model gets wrong and why.; Taught in 8.7 · used in2.4 · 4.4 · 8.7 On the map
Error overlap: The number of errors two models share.; Taught in 8.7 · used in8.7 On the map
Eval mode / train mode: Model states that disable / enable training-only behaviour such as dropout.; Taught in 7.4 · used in7.4 On the map
Evaluation leakage: Evaluation data seen during training or model selection, inflating scores.; Taught in 7.7 · used in7.7 On the map
Evaluation protocol: Fixed rules for how every method is measured and compared.; Taught in 8.3 · used in7.2 · 8.3 On the map
Exact / near-duplicate: Identical examples / examples differing only trivially.; Taught in 8.2 · used in8.2 On the map
Expected output (target): What the model should produce for each input.; Taught in 8.2 · used in8.2 On the map
Experiment tracking: Recording each run's settings, data, versions and results.; Taught in 8.7 · used in1.1 · 3.4 · 8.7 On the map
Exposure bias: The mismatch between training on reference histories and generating from model-produced histories, where errors can affect subsequent inputs.; Taught in 6.6 · used in6.6 On the map

F1 score: A combined score of precision and recall for one class.; Taught in 8.2 · used in8.2 On the map
Failure analysis: Examining the inputs a model gets wrong to find patterns.; Taught in 7.7 · used in7.7 On the map
Fast tokeniser: A tokeniser backed by the compiled Tokenizers library.; Taught in 7.2 · used in7.2 On the map
Feature engineering: The human work of inventing and computing input columns that make a problem easier for a model — the core skill of classical ML.; Taught in 1.1 · used in1.1 · 5.1 On the map
Feature extraction: Using a frozen model's outputs as features for a separate small model.; Taught in 8.1 · used in8.1 · 8.3 · 8.5 On the map
Feature map: A spatial grid of responses produced by a filter or layer. It records where particular learned computations respond, without automatically explaining a prediction.; Taught in 5.2 · used in5.2 On the map
Features: The named, typed columns of a dataset.; Taught in 7.2 · used in1.1 · 1.2 · 1.3 · 2.1 · 2.2 · 2.3 · 2.4 · 3.1 · … On the map
Feed-forward network: In a standard Transformer block, a shared nonlinear network applied separately to each token vector. Attention supplies cross-token information.; Taught in 6.4 · used in6.1 · 6.3 · 6.4 · 6.5 · 7.1 · 7.3 · 8.5 On the map
Fine-tuning: Further training of a pretrained model on labelled data for a downstream task.; Taught in 7.1 · used in1.2 · 1.3 · 2.3 · 3.1 · 4.1 · 4.3 · 5.1 · 7.1 · … On the map
Fingerprint: A digest of sorted texts identifies membership under this implementation; a separate ordered text/label digest is needed to check paired prediction alignment.; Taught in 7.1 · used in4.2 · 7.1 · 7.6 · 8.2 · 8.3 · 8.4 · 8.5 · 8.7 On the map
float32 / float16 / bfloat16: 32- and 16-bit floating-point number formats.; Taught in 7.6 · used in7.6 On the map
Forward pass: Running inputs through the network layer by layer to produce a prediction (and then the loss).; Taught in 1.2 · used in1.1 · 1.2 · 1.3 · 2.3 · 3.3 · 3.4 · 4.3 · 6.1 · … On the map
Foundation model: A model pretrained on broad data, designed to be adapted to many tasks.; Taught in 7.1 · used in7.1 On the map
Freezing: Setting requires_grad = False so a parameter is not trained.; Taught in 8.5 · used in4.3 · 8.5 On the map
Frozen: Not updated during training.; Taught in 8.1 · used in3.3 · 4.3 · 4.4 · 8.1 · 8.3 · 8.4 · 8.5 · 8.6 · … On the map
Frozen encoder: A pretrained model whose parameters stay fixed during the current training stage.; Taught in 8.3 · used in8.1 · 8.3 · 8.4 · 8.7 On the map
Full fine-tuning: Fine-tuning that updates every weight.; Taught in 8.1 · used in2.3 · 8.1 · 8.3 · 8.4 · 8.5 · 8.6 · 8.7 On the map

Gated model: A model whose files require signing in and accepting terms before download.; Taught in 7.2 · used in7.2 On the map
GLUE: A collection of English language-understanding benchmarks, including SST-2.; Taught in 7.3 · used in7.3 On the map
GPU: A processor with many parallel arithmetic units, well suited to neural networks.; Taught in 7.6 · used in1.1 · 1.2 · 1.3 · 2.1 · 2.2 · 2.3 · 2.4 · 3.4 · … On the map
Gradient: For each parameter, how much the loss would change if that parameter were nudged slightly. It points in the direction of steepest increase of the loss.; Taught in 1.2 · used in1.1 · 1.2 · 1.3 · 2.1 · 2.2 · 2.3 · 2.4 · 3.1 · … On the map
Gradient boosting: An ensemble of many small decision trees, each trained to correct the errors of the ones before it. A very strong default for tabular data.; Taught in 4.4 · used in1.1 · 1.3 · 2.3 · 4.4 On the map
Gradient clipping: Capping the size of the gradient before the update step, to stop a single huge gradient from wrecking training.; Taught in 3.3 · used in1.2 · 3.3 · 4.3 · 6.6 · 7.6 · 8.4 · 8.5 On the map
Gradient descent: The core learning rule — repeatedly nudge every parameter a small step in the direction that reduces the loss (opposite the gradient).; Taught in 1.2 · used in1.1 · 1.2 · 2.2 · 3.1 On the map
Gradient norm: The overall size of all gradients, as one number.; Taught in 8.4 · used in3.3 · 3.4 · 6.4 · 6.6 · 8.4 On the map
Greedy decoding: Generating by always choosing the most likely next token.; Taught in 7.3 · used in6.6 · 7.3 · 7.5 · 8.6 On the map
Group split: Splitting so that related examples (same author, same item) stay on one side.; Taught in 8.2 · used in4.1 · 8.2 On the map
GRU: Gated recurrent unit: a recurrent architecture using reset and update gates to control a changing state and candidate information.; Taught in 5.4 · used in5.4 On the map

Head: A small task-specific layer on top of the body.; Taught in 7.1 · used in2.3 · 4.3 · 5.1 · 5.2 · 5.3 · 5.5 · 5.6 · 6.1 · … On the map
Head / tail / head+tail truncation: Keeping the start, the end, or both ends of a long text.; Taught in 8.2 · used in8.2 On the map
Hidden layer: A layer between the input and the output. Its units compute intermediate features that are learned rather than written.; Taught in 1.2 · used in1.1 · 1.2 · 1.3 · 3.3 · 3.4 · 5.1 · 5.6 On the map
Hidden state: A vector carrying a learned summary inside a sequence model. It can change at every position and need not preserve all past input information.; Taught in 5.4 · used in2.4 · 5.4 · 6.1 On the map
Hidden states: The vectors produced after each layer of a Transformer.; Taught in 7.1 · used in5.4 · 7.1 On the map
Highly polar: Labels derived from extreme ratings; review prose may still contain mixed opinions.; Taught in 8.2 · used in8.2 On the map
Hosted API: A service that runs a model remotely and returns results over the internet.; Taught in 7.3 · used in7.3 On the map
Hugging Face Hub: A public website where models and datasets are shared.; Taught in 7.1 · used in7.1 · 7.2 On the map
huggingface_hub: The helper library that talks to the Hub: listing, downloading, caching.; Taught in 7.2 · used in7.2 On the map
Hyperparameter: A setting chosen before training rather than learned — learning rate, batch size, number of layers, dropout rate.; Taught in 1.2 · used in1.2 · 2.4 · 3.1 · 3.3 · 3.4 On the map

IMDb: A dataset of movie reviews labelled positive or negative, used throughout Modules 7 and 8.; Taught in 7.1 · used in5.3 · 7.1 · 7.2 · 7.3 · 7.4 · 7.5 · 7.6 · 7.7 · … On the map
Inductive bias: Assumptions built into a model's design, e.g. convolution assumes nearby pixels are related; attention lets any token relate to any other.; Taught in 1.1 · used in1.1 · 5.2 On the map
Inference: Using a model’s current parameters to calculate outputs without the task-training update loop. Evaluation mode and gradient-recording controls have distinct roles.; Taught in 7.4 · used in1.2 · 1.3 · 2.3 · 2.4 · 3.3 · 4.3 · 5.5 · 6.4 · … On the map
Inference service: A wrapper that turns a model into a dependable component with structured results.; Taught in 7.5 · used in7.5 · 7.6 On the map
Inference tensor: A tensor created under inference_mode, excluded from gradient tracking.; Taught in 7.4 · used in7.4 On the map
Instruction formatting: Rewriting a labelled example as an instruction and response for a chat model.; Taught in 8.2 · used in8.2 On the map
Instruction-tuned model: A base model further trained to follow instructions.; Taught in 8.6 · used in8.1 · 8.6 On the map
int8: 8-bit integers, 1 byte each.; Taught in 7.6 · used in7.6 On the map
Intended use: What a model's creators designed it to be used for.; Taught in 7.2 · used in7.2 On the map

Keep-or-reject rule: A criterion, set in advance, for adopting a model.; Taught in 8.7 · used in8.7 On the map
Kernel: A low-level routine performing an operation for specific hardware and number format.; Taught in 7.6 · used in2.4 · 4.2 · 4.3 · 5.2 · 7.6 On the map
KV cache: Stored attention keys and values for past tokens, reused during generation.; Taught in 7.6 · used in6.6 · 7.6 · 8.6 On the map

Label -100: The label value ignored by Hugging Face loss functions.; Taught in 8.6 · used in8.6 On the map
Label scheme: The set of output classes and what each output position means.; Taught in 7.7 · used in7.7 On the map
Labelling rule: How labels were decided.; Taught in 8.2 · used in8.2 On the map
Latency: The time to serve one request.; Taught in 7.6 · used in4.4 · 7.5 · 7.6 · 7.7 · 8.5 On the map
Layer normalisation: Normalisation over configured feature dimensions within each example or token, using its own mean and variance plus optional learned scale and offset. It does not use a training-dataset mean.; Taught in 6.4 · used in3.3 · 4.3 · 6.1 · 6.4 · 7.1 On the map
Leakage: When information that would not be available at prediction time sneaks into training (e.g. call duration known only after the call), making scores look better than reality.; Taught in 2.2 · used in1.1 · 2.2 · 2.4 · 3.4 · 4.1 · 6.5 · 7.7 · 8.2 · … On the map
Learning rate: The step size used when updating parameters. Too small and training crawls; too large and it oscillates or diverges.; Taught in 1.2 · used in1.2 · 1.3 · 2.3 · 2.4 · 3.1 · 3.2 · 3.3 · 3.4 · … On the map
Learning-rate schedule: How the learning rate changes over training.; Taught in 8.4 · used in2.4 · 3.1 · 4.2 · 6.6 · 8.4 On the map
Length bucketing: Grouping inputs of similar length into batches to reduce padding.; Taught in 7.4 · used in7.4 On the map
Licence: The legal terms for using, changing and sharing a model.; Taught in 7.2 · used in7.2 · 7.3 · 7.7 · 8.1 · 8.2 On the map
Linear decay: Lowering the learning rate steadily to zero.; Taught in 8.4 · used in8.4 On the map
Linear head (probe): A single linear layer trained on frozen features.; Taught in 8.3 · used in8.3 On the map
Linear model: A model whose prediction is a weighted sum of its inputs plus a bias. For classification its decision boundary is a straight line (or flat plane).; Taught in 1.1 · used in1.1 · 1.2 · 1.3 · 5.1 · 7.1 On the map
Linear probe: A simple linear classifier fitted on fixed representations to test how accessible task information is for a declared task and evaluation protocol.; Taught in 5.1 · used in5.1 · 5.6 · 8.3 On the map
LOAD REPORT: Hugging Face's table of skipped (UNEXPECTED) and newly created (MISSING) weights when loading.; Taught in 7.1 · used in7.1 · 7.7 · 8.1 · 8.4 On the map
Local cache: The folder where downloaded Hub files are stored for reuse.; Taught in 7.2 · used in7.1 · 7.2 On the map
Logistic regression: A linear classifier that passes a weighted sum of the inputs through a sigmoid to output a probability. Its decision boundary is a straight line.; Taught in 1.1 · used in1.1 · 1.2 · 1.3 · 2.3 · 4.1 · 4.4 · 5.1 · 5.6 · … On the map
Logit: The raw, unbounded output of a classifier before the sigmoid or softmax turns it into a probability.; Taught in 1.2 · used in1.2 · 1.3 · 2.2 · 2.3 · 2.4 · 3.2 · 4.3 · 6.5 · … On the map
Logits: A model's raw, unnormalised output scores.; Taught in 7.4 · used in1.2 · 1.3 · 2.1 · 2.3 · 2.4 · 3.2 · 3.3 · 5.1 · … On the map
LoRA: A PEFT method that adds small trainable low-rank matrices to some layers.; Taught in 8.1 · used in2.3 · 3.1 · 4.3 · 8.1 · 8.5 · 8.6 · 8.7 On the map
LoRA alpha (α): A scaling constant; the update is multiplied by α / r.; Taught in 8.5 · used in8.5 On the map
LoRA dropout: Dropout applied to the LoRA input during training.; Taught in 8.5 · used in8.5 On the map
LoRA rank (r): The inner dimension of B·A.; Taught in 8.5 · used in8.5 On the map
Loss function: A single number measuring how wrong the model's predictions are. Training tries to make it as small as possible.; Taught in 1.2 · used in1.2 · 1.3 · 2.1 · 2.3 · 8.6 On the map
Low-rank: Representable with few directions.; Taught in 8.5 · used in8.1 · 8.5 · 8.6 On the map
LSTM: Long short-term memory: a recurrent architecture with a cell-state pathway and learned gates controlling retention, input updates and visible output. It helps with long dependencies without guaranteeing unlimited memory.; Taught in 5.4 · used in5.4 On the map

Majority-class baseline: Always predicting the most common label.; Taught in 8.2 · used in2.3 · 3.4 · 8.2 On the map
Majority-class floor: A constant-label baseline selected from training data and evaluated under the same protocol; it is not a guaranteed lower bound.; Taught in 8.3 · used in8.3 · 8.4 On the map
Manifest: A record of the model version, settings, data and software behind a result.; Taught in 7.6 · used in7.6 · 7.7 On the map
map / filter: Datasets operations that transform every row / keep rows that pass a test.; Taught in 7.2 · used in7.2 On the map
Masked language modelling: A pretraining task: predict hidden words from the words on both sides.; Taught in 7.1 · used in7.1 On the map
Masked-word loss: Cross-entropy of a masked language model's guesses for hidden words.; Taught in 8.1 · used in8.1 On the map
Mean pooling: Averaging token vectors, ignoring padding, to get one vector per text.; Taught in 7.5 · used in7.5 · 8.1 On the map
Memory mapping: Reading a file on disk as if it were in memory, fetching parts only when used.; Taught in 7.2 · used in7.2 On the map
Merging: Folding a LoRA update into the frozen weights: W + (α / r)·B·A.; Taught in 8.5 · used in6.2 · 8.5 On the map
Mixed precision: Training with some operations in 16-bit floating point (fp16/bf16) to save memory and time, while keeping sensitive parts in 32-bit.; Taught in 4.3 · used in2.4 · 3.4 · 4.3 · 8.4 On the map
MLP: A multilayer perceptron — a feed-forward network of fully connected (Linear) layers with non-linear activations between them.; Taught in 2.3 · used in1.3 · 2.3 · 2.4 · 3.2 · 5.1 · 5.2 · 5.6 · 6.1 · … On the map
Model card: A repository's README.md describing a model, its uses and limitations.; Taught in 7.2 · used in7.2 On the map
Model repository: A Hub folder with a model's configuration, weights, tokeniser files and card, plus history.; Taught in 7.2 · used in7.2 On the map
Momentum: An optimizer trick that keeps a running "velocity" of past gradients, smoothing out zig-zags and speeding progress along consistent directions.; Taught in 3.1 · used in1.3 · 3.1 · 4.3 On the map
Multi-head attention: Several attention computations in learned projected subspaces, whose outputs are joined and projected. Heads divide feature coordinates, not disjoint sentence segments.; Taught in 6.3 · used in6.3 On the map

n-gram: A sequence of n consecutive words.; Taught in 8.3 · used in8.3 On the map
NF4: A 4-bit format with levels matched to bell-shaped weight distributions.; Taught in 8.6 · used in8.6 On the map
nn.Module: The base class for every PyTorch model and layer. It registers parameters and sub-modules and defines forward().; Taught in 1.3 · used in1.3 On the map
Normalisation: Layers that rescale activations to a stable range during training, making deep networks easier to optimise.; Taught in 3.3 · used in1.1 · 1.3 · 2.2 · 2.3 · 3.1 · 3.2 · 3.3 · 3.4 · … On the map
Normaliser: The tokeniser step that cleans text before splitting, such as lowercasing.; Taught in 7.2 · used in7.2 On the map

Off-the-shelf model: A pretrained model used exactly as downloaded, with no further training.; Taught in 8.3 · used in8.1 · 8.3 · 8.7 On the map
Offset mapping: The character positions in the original text that each token came from.; Taught in 7.2 · used in7.2 On the map
One-hot encoding: Turning a categorical column into several 0/1 columns, one per category, so a model can use it numerically.; Taught in 2.2 · used in2.1 · 2.2 · 2.4 · 3.1 · 4.1 · 4.2 · 4.4 · 5.3 On the map
ONNX: A file format for running models outside the framework that trained them.; Taught in 7.2 · used in7.2 On the map
Optimiser state: Extra per-weight numbers kept by an optimiser; AdamW keeps two.; Taught in 8.5 · used in8.1 · 8.5 · 8.6 On the map
Optimizer: The rule that turns gradients into parameter updates — SGD, momentum, Adam, AdamW. Called with optimizer.step() in PyTorch.; Taught in 1.3 · used in1.1 · 1.2 · 1.3 · 2.2 · 2.3 · 2.4 · 3.1 · 3.2 · … On the map
Out-of-distribution (OOD) input: An input unlike the data a model was trained on.; Taught in 7.5 · used in7.5 On the map
Out-of-domain detection: Checking whether an input resembles the data a model was built for.; Taught in 7.5 · used in7.5 On the map
Overfitting: When a model fits the training data (including its noise) so closely that it performs worse on new data. The gap between training and validation scores reveals it.; Taught in 3.2 · used in1.3 · 2.3 · 2.4 · 3.1 · 3.2 · 3.3 · 3.4 · 5.2 · … On the map
Overflow: A value too large for its format, stored as infinity.; Taught in 7.6 · used in4.3 · 7.6 On the map

Padding: Adding filler tokens so sequences in a batch have equal length.; Taught in 7.2 · used in2.2 · 5.2 · 5.3 · 6.5 · 7.2 · 7.4 · 7.5 · 7.6 · … On the map
Padding waste: Computation spent on padding positions.; Taught in 7.4 · used in7.4 On the map
Paged optimiser: An optimiser that moves its state out of GPU memory when needed.; Taught in 8.6 · used in8.6 On the map
Paired comparison: Comparing two methods on the same examples, example by example.; Taught in 8.3 · used in4.4 · 8.2 · 8.3 · 8.4 On the map
Paired sign test: Comparing two models on the same examples by counting where only one is right.; Taught in 8.7 · used in5.2 · 8.7 On the map
Parameter: A number inside the model that training is allowed to change — usually a weight or a bias.; Taught in 1.2 · used in1.1 · 1.2 · 1.3 · 2.1 · 2.2 · 2.3 · 2.4 · 3.1 · … On the map
PEFT: Parameter-efficient fine-tuning: training only a small set of new or selected weights.; Taught in 8.1 · used in8.1 · 8.5 · 8.7 On the map
Per-channel quantisation: A separate scale for each row of a weight matrix.; Taught in 7.6 · used in7.6 On the map
Percentile (p95): The value below which 95% of measurements fall.; Taught in 7.6 · used in7.6 On the map
Permissive licence: A licence allowing broad use, including commercial, with few conditions.; Taught in 7.2 · used in7.2 On the map
Permutation equivariance: Permuting inputs permutes outputs correspondingly. Unmasked self-attention without positional information has this property.; Taught in 6.3 · used in6.3 On the map
Pipeline: A Hugging Face helper that runs tokenising, the model and output decoding in one call.; Taught in 7.1 · used in2.1 · 2.2 · 2.3 · 2.4 · 3.1 · 3.2 · 3.3 · 3.4 · … On the map
Pipeline tag: The Hub's label for the task a model performs, also used as the pipeline task name.; Taught in 7.3 · used in7.3 On the map
Pooling: An operation that summarises a set or neighbourhood, for example by taking its maximum or mean. The rule determines which information is preserved or discarded.; Taught in 5.2 · used in5.2 · 7.5 · 7.7 On the map
Precision: The smallest relative difference a format can represent.; Taught in 7.6 · used in1.2 · 1.3 · 2.1 · 2.2 · 2.3 · 2.4 · 3.4 · 4.1 · … On the map
Preference tuning: Training from judgements of which response is better (e.g. RLHF, DPO).; Taught in 8.6 · used in8.6 On the map
Prefix tuning: Learning extra key/value vectors that tokens can attend to.; Taught in 8.5 · used in8.5 On the map
Pretrained model: A model whose weights were learned by someone else on a large dataset and published for reuse.; Taught in 7.1 · used in1.1 · 7.1 · 7.3 · 7.5 · 8.1 · 8.2 · 8.3 · 8.4 · … On the map
Pretraining: Training a model on a large, general dataset first, so that its learned representation can be reused for other tasks.; Taught in 1.1 · used in1.1 · 5.1 · 5.5 · 5.6 · 6.2 · 6.6 · 7.1 · 7.3 · … On the map
Probing: Testing what a model knows with inputs designed to need specific knowledge.; Taught in 7.1 · used in1.2 · 7.1 On the map
Prompt masking: Excluding prompt tokens from the loss.; Taught in 8.6 · used in8.6 On the map
Prompt template: A sentence pattern used to describe each class, such as "a photo of a {}.".; Taught in 7.5 · used in7.5 On the map
Prompting: Adapting a model only through its input text; no weights change.; Taught in 8.1 · used in8.1 · 8.6 On the map

QLoRA: LoRA on top of a frozen 4-bit (NF4) model.; Taught in 8.6 · used in4.3 · 8.6 On the map
Quantisation: Storing numbers with fewer bits, such as int8 integers plus a scale.; Taught in 7.6 · used in4.3 · 7.3 · 7.6 · 8.6 On the map
Question answering: Finding the span of a passage that answers a question.; Taught in 7.3 · used in7.1 · 7.3 On the map

Range: The largest magnitude a format can store before overflowing.; Taught in 7.6 · used in2.2 · 2.3 · 2.4 · 3.1 · 3.2 · 3.3 · 3.4 · 4.1 · … On the map
Rank: The number of independent directions a matrix uses.; Taught in 8.5 · used in1.3 · 2.1 · 2.3 · 3.2 · 3.4 · 4.1 · 4.4 · 5.3 · … On the map
Rate limit: A cap on how many requests a service accepts in a given time.; Taught in 7.3 · used in7.3 On the map
Recall: The share of a class's real examples labelled correctly.; Taught in 7.5 · used in2.2 · 2.3 · 3.4 · 4.1 · 5.2 · 5.4 · 7.2 · 7.5 · … On the map
Receptive field: The region of the original input that can influence an output location, given the layer kernels, strides and connectivity.; Taught in 5.2 · used in5.2 On the map
Recurrent neural network: A sequence model that repeatedly updates a hidden state using the current input and previous state. The update weights are shared across positions.; Taught in 5.4 · used in3.3 · 5.4 · 6.1 On the map
Regularisation: Any technique that discourages overfitting — smaller models, weight decay, dropout, early stopping, more data.; Taught in 3.2 · used in2.4 · 3.1 · 3.2 · 3.4 · 4.4 · 6.6 · 8.3 · 8.7 On the map
Relative change: The size of a weight change divided by the size of the original weights.; Taught in 8.1 · used in7.1 · 8.1 On the map
ReLU: The activation max(0, z) — passes positive values through and turns negatives into zero. Cheap, and the default in most networks.; Taught in 1.2 · used in1.2 · 1.3 · 2.3 · 2.4 · 3.3 · 3.4 · 4.2 · 4.3 · … On the map
Repository: A Hub storage space for one model, with its files and history.; Taught in 7.1 · used in1.3 · 2.4 · 3.4 · 4.2 · 7.1 · 7.2 · 7.4 · 7.6 · … On the map
Repository name: The owner/name identifier of a Hub repository.; Taught in 7.2 · used in7.2 · 7.4 On the map
Representation: The set of numbers used to describe an example. It can be written by hand (engineered features) or learned by a network.; Taught in 1.1 · used in1.1 · 1.3 · 2.2 · 2.3 · 3.3 · 4.4 · 5.1 · 5.3 · … On the map
Representation learning: Learning the useful description of the data (the features) from the data itself, jointly with the task, instead of specifying it by hand.; Taught in 1.1 · used in1.1 · 1.3 · 4.4 · 5.1 · 5.6 On the map
Requirement: Concrete, checkable criteria a model must meet, written before measuring.; Taught in 7.7 · used in1.2 · 2.2 · 5.4 · 6.5 · 7.7 · 8.1 · 8.6 On the map
Residual connection: An addition of the form input plus learned update. Compatible shapes preserve an identity route through a block and can help gradient flow.; Taught in 6.4 · used in2.3 · 3.3 · 6.1 · 6.4 · 6.5 · 8.5 On the map
Response cache: Stored answers returned directly when the same request repeats.; Taught in 7.6 · used in7.6 On the map
Results file: A saved record of a method's settings, costs, metrics and predictions.; Taught in 8.3 · used in8.3 On the map
ROC-AUC: The probability that the model scores a random positive example above a random negative one. 1.0 = perfect ranking, 0.5 = random. Robust to class imbalance.; Taught in 1.1 · used in1.1 · 2.2 · 2.3 · 2.4 · 3.1 · 3.2 · 3.3 · 3.4 · … On the map
Roles: System, user and assistant parts of a conversation.; Taught in 8.6 · used in8.6 On the map
Run-to-run variation: Differences between runs caused by randomness.; Taught in 8.7 · used in8.7 On the map

Safetensors: A safe file format for weights that cannot contain executable code.; Taught in 7.1 · used in7.1 · 7.2 · 7.3 · 7.7 On the map
Safetensors header: The start of a safetensors file, listing each tensor's name, shape and number type.; Taught in 7.3 · used in7.3 On the map
Scale: The factor converting between original values and quantised integers.; Taught in 7.6 · used in1.1 · 1.3 · 2.1 · 2.2 · 2.3 · 3.1 · 3.2 · 3.3 · … On the map
Screening: Eliminating candidates on non-negotiable criteria before comparing performance.; Taught in 7.7 · used in7.7 On the map
Seed: The starting number for a random number generator. Fixing it makes random choices (splits, initial weights, shuffling) repeatable.; Taught in 4.2 · used in1.1 · 1.3 · 2.2 · 2.3 · 2.4 · 3.1 · 3.2 · 3.4 · … On the map
Selective accuracy: Accuracy on the answered inputs only.; Taught in 7.5 · used in7.5 On the map
Self-attention: An operation in which positions form learned weighted mixtures of value vectors from the same sequence, subject to information-access masks. Queries and keys determine the mixing weights.; Taught in 6.3 · used in6.1 · 6.3 · 6.4 · 6.5 On the map
Self-hosting: Running a model on machines you control.; Taught in 7.6 · used in7.6 On the map
Semantic search: Retrieving documents by similarity of meaning.; Taught in 7.5 · used in7.3 · 7.5 On the map
Sentence embedding: One vector for a whole sentence, placing similar meanings close together.; Taught in 7.3 · used in7.3 On the map
Sentence-embedding model: A model mapping a text to one vector so similar meanings are close.; Taught in 7.5 · used in7.5 On the map
Setting (hyperparameter) tuning: Choosing settings such as max_length by validation results.; Taught in 7.7 · used in7.7 On the map
SFT: Supervised fine-tuning on (instruction, response) examples.; Taught in 8.6 · used in8.6 On the map
SGD: Stochastic gradient descent — gradient descent using the gradient from one mini-batch at a time instead of the full dataset.; Taught in 3.1 · used in1.3 · 2.3 · 3.1 · 3.3 · 4.3 On the map
Shard: One of several files holding a large model's weights.; Taught in 7.4 · used in7.4 On the map
Sigmoid: The S-shaped function σ(z) = 1 / (1 + e^(−z)) that squashes any number into a probability between 0 and 1.; Taught in 1.2 · used in1.2 · 1.3 · 2.1 · 2.3 · 2.4 · 3.2 · 3.3 · 4.2 · … On the map
Sign test: A paired test that counts the examples where exactly one of two models is right.; Taught in 7.1 · used in5.2 · 5.6 · 7.1 · 7.7 · 8.3 On the map
Singular value decomposition: Splitting a matrix into ranked rank-1 pieces.; Taught in 8.5 · used in8.5 On the map
Skip-gram: A word-embedding objective predicting nearby context words from a centre word. Training pairs should respect document boundaries.; Taught in 5.3 · used in5.3 · 6.6 On the map
Softmax: Exponentiate each logit, then divide by the sum, giving probabilities.; Taught in 7.4 · used in1.2 · 1.3 · 2.1 · 2.3 · 5.1 · 5.4 · 5.5 · 6.1 · … On the map
Special tokens: Tokens with reserved meanings, such as turn markers.; Taught in 8.6 · used in5.3 · 7.2 · 7.4 · 8.6 On the map
SST-2: A dataset of short film-review sentences labelled positive or negative.; Taught in 7.1 · used in7.1 · 7.2 · 7.3 · 7.5 · 7.7 · 8.2 · 8.3 · 8.4 · … On the map
Standardisation: Rescaling each feature to mean 0 and standard deviation 1, using statistics computed on the training set only.; Taught in 2.2 · used in1.1 · 2.1 · 2.2 · 5.1 · 5.6 · 6.4 On the map
State dict: A dictionary from each weight's name to its tensor.; Taught in 7.1 · used in1.3 · 7.1 On the map
Step: One update, made from one batch.; Taught in 8.4 · used in1.1 · 1.2 · 1.3 · 2.1 · 2.2 · 2.3 · 2.4 · 3.1 · … On the map
Stride: The step between successive filter starting positions. A larger stride usually produces fewer output positions and lower spatial resolution.; Taught in 5.2 · used in5.2 On the map
Summarisation: Producing a shorter version of a long text.; Taught in 7.3 · used in6.1 · 7.3 On the map
System message: An instruction placed before the conversation that sets the assistant's role.; Taught in 8.1 · used in8.1 · 8.2 · 8.6 On the map

Target modules: The weight matrices that receive LoRA updates.; Taught in 8.5 · used in8.5 On the map
Task head: A model component that maps a representation to task outputs, such as class logits. A new head may need training even when the encoder is pretrained.; Taught in 5.1 · used in7.1 · 8.1 · 8.4 · 8.5 On the map
Teacher forcing: Training with the true preceding tokens as history. A causal Transformer can compute predictions at all known input positions in parallel.; Taught in 6.6 · used in6.6 On the map
Temperature: A divisor applied to logits before softmax, sharpening (below 1) or flattening (above 1) the distribution.; Taught in 7.5 · used in5.4 · 5.5 · 6.6 · 7.5 On the map
Tensor: PyTorch's multi-dimensional array — like a NumPy array, plus a device (CPU/GPU) and the ability to track gradients.; Taught in 1.3 · used in1.2 · 1.3 · 2.1 · 2.2 · 2.3 · 2.4 · 3.1 · 3.2 · … On the map
Test set: Data held back until the very end to give one honest, final estimate of performance. Never used for decisions during development.; Taught in 3.2 · used in2.2 · 2.3 · 2.4 · 3.2 · 4.1 · 4.4 · 5.2 · 6.6 · … On the map
Text classification: Assigning one label to a whole text.; Taught in 7.3 · used in7.2 · 7.3 On the map
Text generation: Continuing a prompt with new text.; Taught in 7.3 · used in6.1 · 7.3 On the map
Text-to-image retrieval: Finding images whose embeddings are closest to a text query's.; Taught in 7.5 · used in7.5 On the map
TF-IDF: A word-count representation that down-weights words common to most documents.; Taught in 7.1 · used in7.1 · 8.1 · 8.3 · 8.4 · 8.5 · 8.7 On the map
Threshold: The cut-off value in a decision rule, set on validation data.; Taught in 7.5 · used in1.1 · 1.2 · 1.3 · 2.3 · 2.4 · 3.2 · 3.3 · 3.4 · … On the map
Throughput: The number of requests served per second.; Taught in 7.6 · used in2.1 · 4.3 · 7.6 On the map
Token budget: The maximum number of tokens per example.; Taught in 8.2 · used in8.2 On the map
Tokenisation: Splitting text into the units (tokens) a language model processes, and mapping each to an integer ID.; Taught in 2.2 · used in1.1 · 2.1 · 2.2 · 5.3 · 5.4 · 6.1 · 6.2 · 6.3 · … On the map
Tokenizers: Hugging Face's fast, compiled library for splitting text into tokens.; Taught in 7.2 · used in7.2 · 7.4 On the map
Tolerance: The allowed difference when comparing floating-point results.; Taught in 7.6 · used in7.6 · 8.5 On the map
Top-k sampling: Sampling from the k most likely tokens.; Taught in 7.5 · used in7.5 On the map
Top-p (nucleus) sampling: Sampling from the smallest set of tokens with total probability at least p.; Taught in 7.5 · used in7.5 On the map
Top-p sampling: Sampling after retaining a highest-probability set whose cumulative probability reaches a threshold, then renormalising.; Taught in 6.6 · used in6.6 · 7.5 On the map
torchao: PyTorch's library for quantisation and related optimisations.; Taught in 7.6 · used in7.6 On the map
Train / validation / test split: Data for fitting / for choosing settings / for one final estimate.; Taught in 8.2 · used in8.2 On the map
Trainable parameters: Parameters that receive gradients and updates.; Taught in 8.5 · used in3.2 · 7.3 · 8.4 · 8.5 · 8.6 · 8.7 On the map
Training from scratch: Training from random weights using only your own data.; Taught in 7.1 · used in3.4 · 7.1 On the map
Training loss: Loss on the training batches.; Taught in 8.7 · used in1.2 · 1.3 · 2.3 · 3.1 · 3.2 · 3.4 · 5.1 · 5.2 · … On the map
Training memory: Weights + gradients + optimiser states + activations.; Taught in 8.5 · used in3.1 · 8.5 · 8.6 · 8.7 On the map
Training records: Files documenting how a model was trained: arguments, logs, data splits.; Taught in 7.7 · used in7.7 On the map
Transfer learning: Reusing a representation learned on one task or dataset as the starting point for another — the reason pretrained models are so useful.; Taught in 1.1 · used in1.1 · 7.1 On the map
Transformer: An architecture that combines attention-based contextual mixing with positionwise transformations, residual routes and normalisation. Its input interface, masks, training objective and output head determine its task behaviour.; Taught in 6.1 · used in3.1 · 3.3 · 4.3 · 4.4 · 5.4 · 5.5 · 5.6 · 6.1 · … On the map
Transformers: Hugging Face's library of model architectures and loading code.; Taught in 7.2 · used in1.2 · 2.3 · 3.1 · 3.3 · 5.3 · 5.6 · 6.1 · 6.3 · … On the map
Truncation: Cutting an input down to a maximum number of tokens.; Taught in 7.3 · used in5.3 · 7.1 · 7.2 · 7.3 · 7.4 · 7.6 · 7.7 · 8.2 · … On the map

Underfitting: When a model is too simple or too constrained to capture the pattern, so it performs poorly even on the training data.; Taught in 3.2 · used in1.3 · 3.2 On the map

Validation loss / accuracy: Loss or accuracy on held-out data, used to monitor generalisation.; Taught in 8.7 · used in8.7 On the map
Validation set: Data held out from training and used to make decisions — choosing hyperparameters, early stopping, comparing models.; Taught in 2.2 · used in1.3 · 2.2 · 2.3 · 2.4 · 3.1 · 3.2 · 3.4 · 4.1 · … On the map
Validation–test gap: The drop from validation to test score, partly due to selecting on validation.; Taught in 8.7 · used in8.7 On the map
Vanishing gradients: When gradients shrink (vanish) or grow (explode) as they pass back through many layers, so early layers learn far too slowly or training blows up.; Taught in 3.3 · used in1.2 · 3.3 · 3.4 · 5.4 On the map
VRAM: The memory on a GPU. Weights, gradients, optimizer state and activations must all fit in it, or training fails with an out-of-memory error.; Taught in 4.3 · used in4.3 On the map

Warmup: Raising the learning rate gradually at the start of training.; Taught in 8.4 · used in3.1 · 6.4 · 6.6 · 8.4 On the map
Weight decay: A regulariser that shrinks every weight slightly towards zero at each step, discouraging overly large weights.; Taught in 3.2 · used in2.3 · 2.4 · 3.1 · 3.2 · 3.4 · 6.6 · 8.4 On the map
Weight memory: Memory needed to hold a model's parameters: parameters × bytes per parameter.; Taught in 7.3 · used in7.3 · 7.6 · 7.7 On the map
Weight tying: Using one weight matrix in two places, such as the input word table and the word-prediction layer.; Taught in 7.1 · used in6.5 · 7.1 On the map
Weight-only quantisation: Quantising stored weights while computing with higher-precision activations.; Taught in 7.6 · used in7.6 On the map
Weights: The learned multipliers on a layer's inputs. Together with biases they are the model's parameters.; Taught in 1.2 · used in1.1 · 1.2 · 1.3 · 2.1 · 2.2 · 2.3 · 2.4 · 3.1 · … On the map
Weights (parameters): The learned numbers inside a model.; Taught in 7.1 · used in7.1 On the map
Wilson interval: A standard confidence interval for a proportion.; Taught in 8.7 · used in8.7 On the map
WordPiece: A subword algorithm that marks continuation pieces with ##; used by BERT and DistilBERT.; Taught in 7.2 · used in7.2 On the map

Zero-shot classification: Classifying into classes described in words, without training on them.; Taught in 7.5 · used in5.5 · 7.5 On the map
Zero-shot image classification: Classifying an image into labels given as words at run time.; Taught in 7.3 · used in7.3 On the map