Parameter-Efficient Fine-Tuning (PEFT)

Full fine-tuning changes every weight W of the pretrained model to W + ΔW, where ΔW (delta W) is whatever training adds. PEFT keeps W frozen and learns only a compact replacement for ΔW, or a few small extra pieces of network. The pretrained model is then shared: one copy of the large weights, plus a small, swappable piece for each task.

The rest of this lesson answers four questions:

1. Why is full fine-tuning so expensive? (Section 2)
2. Why can a small number of trained parameters be enough? (Sections 3–4)
3. How does LoRA work, and what are its settings? (Sections 5–7)
4. Does it work as well, here? (Sections 8–9)

Figure 8.5.1. LoRA adds (alpha/r) BA beside a frozen weight matrix. A starts random and B starts at zero, so the initial adapter update is zero; the task head may also train.

The same fixed split, the same 1,000 training reviews (checked by fingerprint), head+tail truncation to 128 tokens, and the same helpers as Lesson 8.4. We load the starting model once and copy it for each experiment below.

During training, memory is needed for more than the weights. Every trainable weight also needs a gradient of the same size, and the AdamW optimiser keeps two optimiser states per weight: a running average of the gradients and a running average of their squares. Let's measure all three after one full fine-tuning step on DistilBERT:

In float32 the weights take 267.8 MB, the gradients another 267.8 MB, and Adam's states 535.6 MB, twice as much. That is 1071.3 MB in total, four times the model's size, before counting the activations a batch needs. The rule of thumb: full fine-tuning with Adam needs about 16 bytes per parameter in float32 (4 for the weight, 4 for its gradient, 8 for the two states).

For a small model that's manageable. For a modern large model it's not:

A 7-billion-parameter model (here Qwen2.5-7B, counted from its file header as in Lesson 7.3) needs 15.2 GB just to run in bfloat16. Fully fine-tuning it in float32 needs about 121.8 GB for weights, gradients and Adam states, far beyond any single consumer GPU. (Mixed-precision training reduces this somewhat, but not by enough to change the conclusion.) And each fine-tuned copy must be stored and served separately.

Key terms

• Optimiser state: extra numbers an optimiser keeps for each trainable weight; AdamW keeps two (running averages of the gradient and of its square).
• Training memory: weights + gradients + optimiser states + activations; about 16 bytes per trainable parameter in float32 with Adam, before activations.

The simplest way to save memory is to freeze parameters: set requires_grad = False, so PyTorch computes no gradient for them and the optimiser keeps no state for them. Here we freeze DistilBERT's whole body and train only the head, the "head only" end of the spectrum:

Only 592,130 of the 66,955,010 parameters are trainable now. After a backward pass, a frozen weight's gradient is None: none was computed, and none is stored. The head's gradient exists, with the shape of the head's weights. Frozen parameters still take memory as weights, and still take part in the forward pass, but they need no gradients and no optimiser state.

Freezing the whole body is very cheap, but it's close to feature extraction (Lesson 8.3), and the body can't adapt at all. PEFT methods keep the body frozen but add a small number of trainable parameters inside it, so the representation itself can adapt.

Key terms

• requires_grad: the flag that decides whether PyTorch computes a gradient for a tensor; False freezes it.
• Trainable parameters: those with requires_grad = True; only these get gradients and optimiser states.

Why should a small number of extra parameters be enough to adapt a model? Because the change that fine-tuning makes seems to be much simpler than the weights themselves.

We can check this with a model we already have. The SST-2 sentiment model is DistilBERT base after full fine-tuning (Lesson 7.1). For one weight matrix, the attention query projection, we take the update ΔW = W(SST-2) − W(base) and ask how many "directions" it really uses. The singular value decomposition splits any matrix into a sum of simple rank-1 pieces, each a direction with a strength, sorted from strongest to weakest. If a few pieces carry most of the update, the update is approximately low-rank.

The fine-tuning updates are strongly concentrated. In the last layer, just 8 of the 768 directions carry 55.8% of the update, and 64 directions carry 85.1%. In the first layer the update is more spread out (25.1% in the top 8), but still far more concentrated than a random matrix of the same size, whose top 8 directions hold only 4.0%. The plot shows the whole curve: the fine-tuning updates rise steeply on the left, while the random matrix needs hundreds of directions.

That's the core idea behind LoRA. If ΔW is approximately low-rank, we can learn it directly in low-rank form, with far fewer numbers than a full matrix, and leave W untouched.

Key terms

• Rank: the number of independent directions a matrix uses; a 768 × 768 matrix can have rank up to 768.
• Low-rank: representable (exactly or approximately) with few directions.
• Singular value decomposition (SVD): splitting a matrix into rank-1 pieces ordered by strength; used here to measure how low-rank an update is.

A rank-r matrix of size 768 × 768 can always be written as the product of a tall, thin matrix B (768 × r) and a short, wide matrix A (r × 768). For r = 8 that's 2 × 8 × 768 = 12,288 numbers instead of 589,824. LoRA uses exactly this: next to a frozen weight matrix W, it adds a trainable update in factorised form, so the layer computes

output = W·x + (α / r) · B·(A·x)

where x is the input vector, r is the rank, and α (alpha) is a fixed scaling number.

Two design details make this work well:

• B starts at zero. So at the start of training B·A = 0, and the model behaves exactly like the pretrained model. Training only ever adds a learned correction.
• A starts small and random, so the gradients for B are non-zero from the first step and learning can begin.

An analogy: W is a large printed map that we're not allowed to redraw. LoRA is a transparent overlay on which we may draw only a few straight strokes (the rank). For many trips, a few strokes are enough to mark the new route.

Here is a complete LoRA layer in a few lines, wrapping an ordinary nn.Linear:

At the start, the LoRA layer's output is exactly the same as the original layer's, because B is zero. The trainable part is 12,288 numbers instead of the matrix's 589,824, 2.1% of it.

The rank r sets the size of the update, and the trade-off between capacity and cost:

Rank 1 needs only 1,536 numbers (0.3% of the matrix); rank 8, 12,288 (2.1%); rank 64, 98,304 (16.7%). At rank 384, half the matrix's width, LoRA already has as many numbers as the full matrix, and there's nothing left to save. In practice, ranks from 4 to 64 are common, and Section 4's measurements suggest why small ranks often suffice.

Alpha scales the update by α / r. Its purpose is practical: when you try a different rank, you can keep α fixed, which changes the explicit scaling factor α/r. The realised update also depends on initialization and optimization, so learning-rate tuning may still be needed. We use r = 8 and α = 16, so the scale is 2.

Which matrices get a LoRA update? A Transformer layer (Lesson 6.4) has the attention projections (query, key, value and output; q_lin, k_lin, v_lin and out_lin in DistilBERT) and the feed-forward network (lin1, lin2). The PEFT library takes a list of names, target_modules, and counts follow directly:

LoRA on only the query and value projections adds 147,456 trainable numbers across DistilBERT's 6 layers; on all four attention projections, 294,912; on everything including the feed-forward layers, 663,552. In each case the new classification head (592,130 parameters) is also trained, because it starts random and has to be learned. The original LoRA work found query and value enough for many tasks, and that is our choice. Targeting more layers gives the update more capacity, at a modest cost.

Key terms

• LoRA rank (r): the inner dimension of the factorised update B·A; it sets how many directions the update can use.
• LoRA alpha (α): a scaling constant; the update is multiplied by α / r.
• Target modules: the weight matrices that receive LoRA updates.

Common mistake: forgetting that the classification head must be trained too. LoRA adapts the frozen body, but a new task's head starts random. The PEFT library trains it automatically when task_type="SEQ_CLS" is set; in a hand-built setup you must make it trainable yourself, to implement the intended trainable-head setup.

Visual walkthrough: See why the initial adapter update is zero.

1. Freeze a 2 × 2 base. W₀ = [[1, 0], [0, 1]] · x = [2, 1]. Use rank r = 1 and α = 1, so α/r = 1. This tiny matrix is an illustration, not a measured trained adapter.

2. Initialize the adapter. A = [[0.2, −0.1]] · B = [[0], [0]]. A starts random; B starts zero. Ax = 0.3, but BAx = [0, 0]. The initial combined adapter update is zero.

3. After an illustrative update. B = [[0.5], [−0.2]] · A held as shown. For illustration, B changes to these supplied values. We are not deriving an optimizer step or running training here.

4. Add the low-rank path. W₀x + BAx = [2, 1] + [0.15, −0.06] = [2.15, 0.94]. The base remains frozen while adapters train. A task head may also train; include it when counting trainable parameters.

The app adds a paused animation of this same supplied example.

LoRA is one member of a family. Two other well-known PEFT methods, for comparison:

• Bottleneck adapters insert small two-layer networks inside each Transformer layer: squeeze the 768-number vector down to a small size m, apply a non-linearity, and expand back, with a residual connection around them. The pretrained weights stay frozen.
• Prefix tuning (and its simpler relative, prompt tuning) leaves every weight untouched and instead learns a few extra vectors that are prepended to the keys and values in each attention layer: "virtual tokens" the real tokens can attend to.

We can compare their sizes for DistilBERT by counting:

With a bottleneck size of 64, two adapters per layer add 1,189,632 parameters; prefix tuning with 20 vectors adds 184,320; LoRA with rank 8 on query and value adds 147,456. All three are around 1–2% of the model or less. They differ in other ways:

• Adapters add extra layers that run at inference time, which adds a little latency.
• Prefix tuning uses up some of the context length.
• LoRA's update can be merged into W after training (Section 9), so the final model has no extra cost at all.

That last property is a large part of why LoRA became the default choice.

Key terms

• Bottleneck adapter: a small down-projection, non-linearity and up-projection inserted into each layer, trained while the rest is frozen.
• Prefix tuning / prompt tuning: learning extra key/value or input vectors while all weights stay frozen.

The PEFT library wraps a model in a few lines: describe LoRA in a LoraConfig, then call get_peft_model. We use rank 8, alpha 16, LoRA dropout 0.1 (dropout applied to the LoRA input during training), and the query and value projections. task_type="SEQ_CLS" tells the library this is sequence classification, so it also makes the new head trainable.

The training recipe is Lesson 8.4's, unchanged except for the learning rate: 1e-3 instead of 5e-5. LoRA trains small new matrices whose combined update starts at zero (A is random; B is zero), not pretrained weights that must be preserved, so it needs larger steps. Rates around 1e-4 to 1e-3 are typical. We use a common default here rather than tuning it, which would mean another training run.

print_trainable_parameters reports 739,586 trainable out of 67,694,596, 1.0925%. Of these, 147,456 are the LoRA matrices and 592,130 are the new head. The total count is a little larger than DistilBERT's 66,955,010 because the wrapped model also contains the LoRA matrices and a spare copy of the original head, kept so the adapter can be removed again.

The loop is identical to Lesson 8.4's, except that the optimiser only receives the trainable parameters, and clipping applies only to them. Checkpoints are checked every 16 steps on 200 validation reviews, and the best one is kept.

Validation accuracy on 200 reviews climbs from 0.790 at step 16 to 0.855 at step 32 and 0.870 at step 48, the best checkpoint, then dips to 0.860. LoRA learns faster at the start than full fine-tuning did in Lesson 8.4: its first checkpoint is already far above chance. That's partly because of its larger learning rate. The chosen checkpoint scores 0.880 on all 1,000 validation reviews. Section 9.4 compares the cost of training with Lesson 8.4's.

Because the pretrained weights never changed, we only need to save what was trained:

The adapter folder holds a small configuration file, a README and the trained weights: 3.0 MB in total, against the 267.8 MB of weights a full fine-tuned copy needs (Section 2). Loading the base model and attaching the saved adapter (PeftModel.from_pretrained) gives identical probabilities. A hundred tasks would need one base model plus a hundred small adapters, rather than a hundred full models.

For deployment we can merge the update into the frozen weights: replace each W with W + (α / r)·B·A. The model then becomes a plain DistilBERT classifier, with no LoRA layers and no extra computation at inference.

After merge_and_unload, the model is an ordinary DistilBertForSequenceClassification with no LoRA modules left, and its probabilities differ from the unmerged model by at most 7.75e-07, only floating-point rounding. So you can choose per deployment: keep the adapter separate when one base model serves many tasks, or merge it when a single task needs maximum speed and simplicity.

All choices are made, so we test once, and compare on the same 1,000 test reviews with the results Lessons 8.3 and 8.4 saved:

LoRA scores 0.834 on test, with recall 0.8289 on negatives and 0.8388 on positives. Against full fine-tuning (Lesson 8.4), the result is a dead heat: each is right on 54 reviews the other gets wrong (p = 1). Against the baselines, the picture is the same as for full fine-tuning:

• no significant difference from TF-IDF or frozen features at 1,000 labels (p = 0.73 and 0.85);
• significantly below the off-the-shelf SST-2 model (43 against 82, p = 0.00062) and TF-IDF with all labels.

This is a single training run (as was Lesson 8.4's), and a different random initialisation of the LoRA matrices would give slightly different numbers. The safe conclusion is not "LoRA equals full fine-tuning to the review", but that on these 1,000 test reviews there's no evidence of a difference, while LoRA trained about 1% of the parameters and saved a 3.0 MB adapter.

Finally, what did each approach cost the pretrained model's other skills?

Interpret the evidence carefully. Equal observed accuracy does not mean the classifiers make identical predictions. In the source reference results, full fine-tuning and LoRA disagree on 108 of 1,000 reviews. A paired p-value of 1 does not establish equivalence; stronger equivalence claims need a declared tolerance and an appropriate analysis.

With LoRA switched off, every pretrained weight is exactly as it was. Forgetting is literally zero, by construction: the original model is always one switch away. Even merged into the weights, LoRA raised the word-guessing loss only from 3.381 to 3.461.

To compare costs, we read Lesson 8.4's saved results next to ours. The training-memory estimate uses Section 2's accounting: weights, plus gradients and two Adam states for the trainable parameters only.

Interpret the evidence carefully. This remains the tied-output retention probe from 8.1. Disabling the adapter restores the frozen base path for this comparison; it does not by itself prove retention of every ability a deployed application might need.

LoRA trained 739,586 parameters against 66,955,010, so its estimated training memory is a fraction of full fine-tuning's, mostly the frozen weights themselves. Training was also quicker, though not a hundred times quicker (Check yourself 3). Validation accuracy was similar: 0.880 for LoRA against 0.860 for full fine-tuning. And full fine-tuning's word-guessing loss was 4.07, against LoRA's 3.461: LoRA's small, low-rank update disturbs the pretrained skill much less.

Everything is saved for Lesson 8.7:

Common mistake: concluding from one run that LoRA is "better" or "worse" than full fine-tuning by a fraction of a point. Single runs on 1,000 reviews differ by several reviews from initialisation and shuffling alone. Compare on the same test examples with a paired test, and treat small differences as noise unless the test says otherwise.

Let's reassemble the PEFT story.

Full fine-tuning is expensive because every trainable weight needs a gradient and two Adam states: DistilBERT's 267.8 MB of weights became 1071.3 MB in training, and a 7B model would need about 121.8 GB. Freezing parameters removes their gradients and optimiser states. Freezing the whole body leaves only 592,130 trainable parameters, but then the body can't adapt.

The core idea of PEFT is that fine-tuning's changes are simpler than the weights. The real update from DistilBERT to its SST-2 version concentrated 55.8% of its strength in 8 of 768 directions in the last layer, against 4.0% for a random matrix. LoRA learns such an update directly in low-rank form: next to each frozen W, two thin matrices B and A, with B starting at zero, so training begins from the pretrained model exactly. Rank 8 on a 768 × 768 matrix trains 12,288 numbers instead of 589,824; alpha scales the update; target modules choose which matrices get one. Adapters and prefix tuning are alternatives of similar size, but only LoRA can be merged away.

In practice, LoRA on the query and value projections trained 739,586 parameters (1.0925%, including the new head). It reached 0.880 on validation and 0.834 on test, equal in observed accuracy to full fine-tuning in this run (p = 1; not proof of equivalence), with a 3.0 MB adapter instead of 267.8 MB of weights. Merging gave a plain model with outputs equal to within 7.75e-07. And it forgot far less: zero when switched off, and a word-guessing loss of 3.461 when merged, against 4.07 for full fine-tuning. Neither approach beat the off-the-shelf SST-2 model with 1,000 labels. That's the question Lesson 8.7 takes up.

Figure 8.5.2. Both source runs achieved 0.834 test accuracy but disagreed on 108 of 1,000 predictions. Equal observed accuracy and p=1 do not prove equivalence. Memory estimates exclude activations.