Fine-Tuning Large Language Models (Conceptual)

A base (or pretrained) language model has learned one thing: to continue text the way text on the internet continues. Ask it a question and it may answer, but it may equally continue with another question, a list or an essay. An instruction-tuned model has been further trained to treat its input as a request and to respond helpfully. The two share the same architecture and almost the same weights; the difference is a stage of fine-tuning.

We can see the difference directly. SmolLM2-135M exists in both versions. We give the base model a plain request and the instruct model the same request in its chat format, both with greedy decoding:

Both models have 134,515,008 parameters and the same architecture. The base model treats the request as the start of a document. Its continuation sounds plausible but immediately repeats itself ("always try to be as objective as possible", twice). The instruct model treats it as a request and starts a helpful answer ("Here are two…"). Same architecture, same pretraining; the instruction tuning made the difference.

Instruction tuning usually happens in stages:

1. Pretraining on huge amounts of text: next-token prediction (Lesson 6.6).
2. Supervised fine-tuning (SFT) on curated examples of instructions and good responses.
3. Often, preference tuning: further training from human (or AI) judgements of which of two responses is better. You'll hear of methods such as RLHF and DPO; they're beyond this lesson.

This lesson is about stage 2, and how it's made affordable.

Figure 8.6.1. This assistant-only SFT setup keeps prompt tokens visible as context while masking their targets from loss. Causal prediction shifts targets by one; LoRA adapts selected parameters.

Supervised fine-tuning trains a pretrained language model on pairs of (instruction, desired response). The training objective is the same next-token prediction the model was pretrained with, so nothing new has to be invented. What changes is the data: instead of arbitrary web text, every example shows the model exactly the behaviour we want, an instruction followed by a good response.

For SFT to teach this behaviour cleanly, two things must be set up carefully, and the next three sections cover them:

• Formatting (Section 3): every example must use the same conversation format, with special markers for who is speaking. This is the format the model will see at inference time.
• Masking (Sections 4–5): the loss should reward the model for writing the response, not for predicting the instruction. The user's words aren't something the model should learn to produce.

The amount of data needed is often surprisingly small compared with pretraining: thousands to hundreds of thousands of examples. That's because SFT mostly teaches a format and a behaviour the model can already almost produce; the knowledge comes from pretraining.

Key terms

• Base model: a language model after pretraining only; it continues text.
• Instruction-tuned (instruct) model: a base model further trained to follow instructions.
• Supervised fine-tuning (SFT): fine-tuning a language model on (instruction, response) examples with the next-token loss.
• Preference tuning: further training from judgements of which response is better (for example RLHF or DPO).

A chat model reads a conversation as one long token sequence, with special tokens marking where each message starts and ends and who wrote it. The exact format is the model's chat template, stored with its tokeniser, and apply_chat_template produces it. Here is a one-turn conversation with an answer, formatted for SmolLM2:

Each message is wrapped in <|im_start|>role … <|im_end|>. The roles are system, user and assistant, and the tokeniser added SmolLM2's default system message by itself. The markers are single special tokens, with IDs 1 and 2, so the model can recognise turn boundaries exactly. The whole conversation is 53 tokens.

Every SFT example must use this exact template, because it's what the model will see at inference time. A model fine-tuned on a slightly different format, say with "User:" and "Bot:" in plain text, learns to expect that format, and behaves worse when served with the official one (and vice versa).

Key terms

• Special tokens: tokens with a reserved meaning, such as <|im_start|> and <|im_end|>, used to structure a conversation.
• Chat template: the model-specific format for turning a list of messages into one token sequence.
• System / user / assistant roles: the instruction-setting message, the person's messages and the model's replies.

In the causal language-modelling loss, every position predicts the next token (Lesson 6.6). But we don't want the model to learn to write the user's message. We want it to learn the response, given the message. So the loss is computed only on the response tokens. In Hugging Face, any label set to -100 is ignored by the loss. We find where the prompt ends (everything up to and including the start of the assistant's turn) and mask everything before it:

Of the 53 tokens, 50 belong to the prompt and are masked with -100; only 3 are trained on: the answer negative, the end-of-turn marker <|im_end|>, and a newline (Ċ is how this tokeniser displays a newline). Training on <|im_end|> matters: it teaches the model to stop after answering, rather than rambling on.

Key terms

• Input (prompt) masking: excluding prompt tokens from the loss so the model learns only the response.
• Label -100: the value Hugging Face's loss functions ignore.

Common mistake: training on the whole conversation, prompt included. The model then spends much of its learning on predicting user messages, and on long prompts with short answers the answer gets only a sliver of the loss. Mask the prompt.

Visual walkthrough: Separate visible context from supervised targets.

1. Format a toy exchange. Input tokens: [USER, hi, ASSISTANT, hello, EOS]. These names stand for token IDs in a simplified assistant-only supervised fine-tuning example.

2. Keep context visible. Attention mask: [1, 1, 1, 1, 1]. All five positions are real tokens. A separate causal attention mask prevents future-token access.

3. Mask prompt targets. Labels: [−100, −100, −100, hello, EOS]. −100 means ignore this target in cross-entropy. It does not hide the prompt from the model.

4. Apply causal shifting. logits at ASSISTANT → hello; logits at hello → EOS. Causal-LM loss aligns logits[:-1] with labels[1:]. Only assistant-response targets contribute here. LoRA changes selected parameters during training.

The app adds a paused animation of this same supplied example.

The causal language-modelling loss is the cross-entropy of predicting each token from all the tokens before it (Lesson 6.6). Position t's output predicts token t + 1, so the logits and the targets are shifted by one before comparing. Let's compute it by hand on our example and check it against the model's own loss:

The model scores a vocabulary of 49,152 tokens at every position, but only 3 positions count, the answer tokens. Our hand computation, 2.9850, matches the model's exactly. Without masking, averaging over every token gives 3.7479, a different number, dominated by how well the model predicts the prompt, which isn't what we want to train.

This is the same loss as pretraining. SFT changes what the examples look like and which positions count, not the objective.

LoRA works on a language model exactly as on DistilBERT in Lesson 8.5: freeze the weights, add trainable low-rank matrices to chosen projections. In LLMs the usual targets are the attention projections (q_proj, k_proj, v_proj, o_proj) and often the feed-forward ones (gate_proj, up_proj, down_proj):

On SmolLM2-135M, rank 8 on the query and value projections trains 460,800 parameters (0.34%); on all seven projection types, 2,442,240 (1.78%). For a large model the fraction is far smaller. We count from Qwen2.5-7B's configuration file:

Qwen2.5-7B has 28 layers, a hidden size of 3584, and smaller key and value projections (512 wide, because many query heads share each key/value head). LoRA rank 8 on the query and value projections trains 2,523,136 parameters, just 0.033% of the 7,615,616,512; on all seven projection types, 20,185,088 (0.265%). The trainable part of a 7B model is smaller than all of DistilBERT. As Lesson 8.5 explained, the frozen weights still have to fit in memory, which Section 7 deals with.

Let's run real SFT, small enough for a CPU. The task is sentiment as a chat task: the instruct model must answer "positive" or "negative" to a question about a review. We use the fixed IMDb split (Lesson 7.1; the fingerprints confirm the same reviews):

First, how does the instruct model do before fine-tuning? As in Lesson 8.1, we score each answer by comparing the model's scores for "positive" and "negative" as the first token of its reply, on the first 100 validation reviews (cut to their first 60 words to keep the run short):

Its accuracy is 0.43, and it answers "positive" for 0.00 of the reviews: it says "negative" to everything. This small model can produce the right format, but it hasn't connected the question to the review's sentiment. (Lesson 8.1 showed that a larger, 360M-parameter model in the same family does far better without any fine-tuning.)

Now we build SFT examples from 256 training reviews, each a conversation ending in the correct one-word answer, with the prompt masked:

Each example is about 125 tokens, of which only 3 are trained on (the answer word, <|im_end|> and the newline). We add LoRA, rank 8 on the query and value projections, and train for one pass over the 256 examples, in batches of 8:

LoRA trains 460,800 of 134,975,808 parameters (0.3414%). Over the 32 steps, the loss on the answer tokens falls from 1.148 to an average of 0.184 over the last 8 steps. Did the behaviour change?

Yes. After SFT on 256 examples, accuracy on the same 100 validation reviews rises from 0.43 to 0.76, and the model now answers "positive" for 0.71 of them instead of never. It has learned to connect the question to the review, though it now leans positive, and 0.76 is well below what the classifiers in Lessons 8.3–8.5 reached. A real SFT run would use far more data, a larger model and several checks on validation. The mechanics, though, are exactly these: chat-formatted examples, prompt masking, the causal LM loss and LoRA.

LoRA shrinks the trainable part to a fraction of a percent, but the frozen weights still have to sit in memory for every forward and backward pass. For a 7B model in bfloat16 that's about 15.23 GB (Section 8), more than most consumer GPUs have. The idea of QLoRA is to store those frozen weights in 4 bits each, a quarter of bfloat16's 16. The LoRA matrices are still trained in higher precision. During the forward pass, each 4-bit weight is converted back (dequantised) when it's needed.

Lesson 7.6 quantised weights to 8-bit integers with one scale per row. Four bits can encode 16 levels; this symmetric teaching example uses 15 of them (-7 to 7), so the scale must fit the numbers more closely. QLoRA-style quantisation uses a separate scale for each block of 64 weights. Here is 8-bit and 4-bit blockwise quantisation of one of SmolLM2's feed-forward matrices:

The matrix takes 3.54 MB in float32. With 8 bits it shrinks to 0.94 MB, with an average error of only 0.6% of a typical weight's size. With 4 bits it's 0.50 MB, about a seventh of float32, but the average error jumps to 11.7%. Fifteen evenly spaced levels are coarse.

QLoRA reduces this error with two further tricks, which we describe rather than implement:

• NF4 ("4-bit NormalFloat"): neural-network weights are roughly bell-shaped, concentrated near zero, so instead of 15 evenly spaced levels, NF4 places its 16 levels where weights are most common. More levels near zero means smaller errors for most weights.
• Double quantisation: the per-block scales are themselves quantised, saving a little more memory.

The 4-bit weights are frozen and never updated. Training only adjusts the LoRA matrices, which learn to work with the quantised model, so the rounding error matters less than in pure inference quantisation. QLoRA also uses paged optimisers, which move optimiser state out of GPU memory when it runs short. In practice QLoRA is run with the bitsandbytes library on NVIDIA GPUs; that's why this lesson describes it rather than running it.

Key terms

• Blockwise quantisation: quantising in small blocks (here 64 numbers), each with its own scale.
• NF4: a 4-bit format whose levels are spaced to match the bell-shaped distribution of neural-network weights.
• QLoRA: LoRA training on top of a frozen model stored in 4-bit (NF4) precision.
• Dequantisation: converting quantised values back to higher precision for computation.

Putting it all together: which method fits on which hardware? The table below estimates memory for the weights and training state only. As a rough rule we leave 20% headroom for activations, the KV cache and framework overhead. Real usage depends on sequence length and batch size, so treat these as first estimates, not guarantees.

For SmolLM2-135M, the illustrated float32 training-state subtotal is 2.15 GB; activations and runtime allocations are additional. For Qwen2.5-7B, the picture is the one that shapes real projects:

• Full fine-tuning needs about 121.85 GB, more than even an 80 GB data-centre GPU. That exceeds the listed 80 GB device under this float32/Adam estimate; other precision, sharding and offload choices change the requirement.
• LoRA with bfloat16 weights needs about 15.23 GB, before adapters, activations and runtime workspaces; actual training fit must be measured.
• QLoRA brings the frozen weights down to about 4.28 GB, before the remaining runtime allocations; this alone does not establish an 8 GB training fit.

That's why QLoRA made fine-tuning large models practical for individuals and small teams.

A practical decision guide, building on Lesson 8.1:

1. Try prompting first (including a few examples in the prompt). If a capable model does the job, there's nothing to train.
2. If you need a behaviour or format the model won't reliably produce, SFT with LoRA on a model that fits your hardware is the default. Use QLoRA if the bfloat16 weights don't fit.
3. Full fine-tuning deserves a clear measured benefit for its cost. Hardware needs depend on model size, precision, sequence length, batch size, sharding and offload; large GPU clusters are not a universal requirement.
4. Whatever you choose, evaluate the adapted model on held-out examples, check for forgetting of the abilities you still need (Lessons 8.1 and 8.5), and compare against the prompting baseline. Lesson 8.7 turns this into a decision process.

Figure 8.6.2. Simplified storage arithmetic, not measured peak training memory: float32 full-model Adam state, bfloat16 base weights, and an illustrative four-bit layout. No bar guarantees a GPU training fit or implements NF4.

Common mistake: assuming the trainable-parameter count decides whether training fits. With LoRA, the trainable part is tiny; it's the frozen weights (and activations) that fill memory. Estimate from the full model size and its storage format.

The 20% margin is illustrative rather than a sufficient bound. Peak memory depends on sequence length, batch size, saved activations, checkpointing, unquantised modules, adapter/optimizer state and implementation workspaces. The hand-written uniform quantiser is not an executed NF4/QLoRA implementation. Full fine-tuning does not universally require a cluster; that statement depends on model size, hardware and the chosen precision/optimizer/offload recipe.

Let's reassemble how large language models are adapted.

A base language model continues text; an instruction-tuned model responds to requests. With the same SmolLM2-135M architecture, the base model rambled and repeated itself while the instruct model began a helpful answer. The difference comes from supervised fine-tuning: next-token training on (instruction, response) examples. Each example is written in the model's chat template, with special tokens marking each role. Masking sets the prompt's labels to -100, so the causal LM loss is computed only on the response. In our example 3 of 53 tokens, including the end-of-turn marker, and the hand-computed loss (2.9850) matched the model's exactly.

LoRA makes SFT affordable. On SmolLM2-135M it trained 460,800 parameters (0.34%); on Qwen2.5-7B the same setting would train only 0.033% of the model. A tiny real SFT run on 256 examples took accuracy on 100 validation reviews from 0.43 to 0.76, though the model then leaned positive. QLoRA stores the frozen weights in 4 bits. Done by hand with even spacing, 4-bit blockwise quantisation had an average error of 11.7% against 0.6% at 8 bits, which is why QLoRA uses NF4. It brings a 7B model's weights from 15.23 GB to about 4.28 GB, as an illustrative storage subtotal, not a guarantee of fitting training on an 8 GB GPU, while full fine-tuning would need about 121.85 GB.