Supervised Fine-Tuning and PEFT

Hand a freshly pre-trained model the prompt "What is the capital of France?" and it may answer with three more exam questions. It has read enough of the web to know that Paris is the capital, but it has learned to continue text, not to respond to it. There is no turn-taking, no stable persona, no refusal, no fixed output format. The knowledge is present; the behavior is not.

Closing that gap is the first and cheapest step in turning a base model into an assistant. The opening from Chapter 5 is concrete: pre-training buys knowledge under the wrong objective, and we now have to install assistant behavior without paying for a second pre-training run.

That constraint shapes everything. The behavior change must be large in effect but small in data, because high-quality instruction data is written or curated by people and does not exist at pre-training scale. And it must be cheap in compute and memory, because the people who adapt a model are rarely the people who trained it: a downstream team has one or a few accelerators, not a cluster, and wants many task-specific variants, not one. A full fine-tune that copies and updates every weight defeats both goals at once.

One idea runs through that constraint. Adaptation is a small, low-rank, additive, composable change on top of a frozen base. Each word in that sentence pays off as a separate consequence: a few thousand examples reshape behavior because the change is small; a rank-16 adapter approximates a full fine-tune because the change is low-rank and additive; and two fine-tunes can be added together because the change is composable. The sections that follow take the three in turn.

Two of those mechanics are worth pinning down now: a gradient is the per-weight signal, computed from the loss, that says which way to nudge a weight during training, and a frozen weight is one held fixed, receiving no such update and staying exactly as pre-training left it.

Why a few thousand examples reshape behavior

The core method is supervised fine-tuning (SFT), also called instruction tuning, the recipe that InstructGPT made the standard first step of alignment (Ouyang et al. 2022). Continue the same next-token objective from Chapter 5, but on a curated set of (prompt, response) pairs instead of raw web text, and mask the loss so only the response tokens are scored. The model still learns by predicting the next token; what changes is the distribution it predicts over. For a pair with prompt $x$ and response $y$,

$$\mathcal{L}_{\text{SFT}} = - \sum_{t} \log p_\theta(y_t \mid x, y_{<t}),$$

where $x$ is the prompt, $y_t$ is the next response token, and $y_{<t}$ is the response prefix already shown to the model. The sum runs over response positions only. Masking the prompt keeps the model from spending capacity learning to generate user turns, and teaches the asymmetry of the assistant role: read the prompt, produce the answer.

What makes the prompt and response distinguishable is the chat template. A base model sees one undifferentiated stream of tokens. An assistant needs to know where the user's turn ends and its own begins, so SFT wraps each turn in role markers, special tokens like <|user|> and <|assistant|> that the tokenizer reserves and the template inserts. The template is a contract: the same markers must be used at inference, or the model is served off the distribution it was tuned on. This is the smallest example of a recurring truth in this part, that adaptation is as much about format as about weights. Figure 17.2 traces a single example from template to scored loss.

The surprising part is how little data this takes, which is what small means in the unifying claim. The superficial-alignment hypothesis, named by the LIMA paper, holds that a model's knowledge and capability are laid down almost entirely in pre-training, and that SFT mainly teaches it which subdistribution of formats and styles to speak in (Zhou et al. 2023). If that is right, then a few thousand carefully chosen examples should suffice, and LIMA showed exactly that: a 65B model tuned on 1,000 curated examples, without any reinforcement learning, was competitive with far more heavily aligned models (Zhou et al. 2023). The lesson reframes the whole step. SFT is eliciting and shaping latent capability, not adding it, which is why data quality and diversity dominate data quantity, and why a small noisy set can teach a confident wrong style as easily as a right one.

That reframing cuts both ways, and the cut is the trade-off SFT carries. With few examples, every example matters, and a handful of wrong facts or a single overused format teaches a defect the model will repeat confidently. Narrow SFT data bakes in a rigid template; this is the format-overfitting failure that over-refusal and always-bulleted answers are instances of.

Why a rank-16 adapter approximates a full fine-tune

If adaptation is a small, low-dimensional change to behavior rather than a wholesale rewrite of knowledge, then the weight update that encodes it should be small too. This is the inference that makes parameter-efficient fine-tuning (PEFT) plausible. PEFT freezes the pre-trained weights and trains only a tiny set of new or selected parameters. To see why the field arrived at the dominant method, it helps to watch the per-task cost fall step by step.

The efficiency spine in Figure 17.3 runs from full fine-tuning to ever-cheaper approximations of it. The earliest transfer-learning recipe was to fine-tune all the weights of a pre-trained model on a downstream task, the approach that BERT and GPT made standard. It works but it is expensive and wasteful: every task needs a full copy of the model, and the update touches every weight to encode what is often a narrow behavioral change. The first answer was adapters, small bottleneck modules inserted between the frozen layers of a transformer, of which only the inserted parameters train (Houlsby et al. 2019). Adapters proved the principle that a tiny fraction of new parameters could recover most of full fine-tuning's quality, but they added depth, and therefore inference latency, to every forward pass (Houlsby et al. 2019).

low-rank adaptation (LoRA) removed that cost, and it is where low-rank and additive enter the claim. The rank of a matrix is the number of genuinely independent directions its rows or columns span, so a low-rank matrix is one whose content collapses into far fewer directions than its dimensions allow. For a frozen weight matrix $W_0 \in \mathbb{R}^{d \times k}$, LoRA learns a low-rank update (Hu et al. 2021)

$$W = W_0 + \Delta W = W_0 + \frac{\alpha}{r} B A, \quad B \in \mathbb{R}^{d \times r},\; A \in \mathbb{R}^{r \times k},$$

Here $W_0$ is the frozen matrix, $\Delta W$ is the trainable update, $r$ is the rank budget, and $\alpha/r$ is the scaling that keeps update size comparable as the chosen rank changes. With rank $r \ll \min(d, k)$, $A$ and $B$ together hold $r(d+k)$ parameters instead of $dk$. Only $A$ and $B$ train; $W_0$ stays frozen. The hypothesis behind it is that the update a fine-tune wants to apply has low intrinsic rank, so a rank-8 or rank-16 factorization captures most of it (Hu et al. 2021). Because the adaptation is an additive update to existing matrices rather than a new sublayer, at inference the product $BA$ folds back into $W_0$, so a served LoRA model has zero added latency, unlike adapters that bolt on extra layers. That is why LoRA became the default PEFT method. Figure 17.4 shows the two parallel paths and what each holds.

Plug in your own dimensions and ranks to see how few parameters $r(d+k)$ is against the full $dk$, and how the fraction grows linearly with rank.

import numpy as np
d, k = 4096, 4096          # one weight matrix, e.g. an attention projection
full = d * k               # full fine-tune touches every weight
ranks = [4, 8, 16, 32, 64]
print(f"full fine-tune: {full:,} trainable params")
for r in ranks:
    lora = r * (d + k)     # A is r x k, B is d x r
    print(f"  rank {r:3d}: {lora:>9,} params  ({100*lora/full:5.2f}% of full)")
import matplotlib.pyplot as plt
fracs = [100 * r * (d + k) / full for r in ranks]
plt.plot(ranks, fracs, "o-")
plt.xlabel("LoRA rank r"); plt.ylabel("trainable params (% of full)")
plt.title("Cost rises linearly with rank"); plt.grid(True); plt.show()

quantized low-rank adaptation (QLoRA) then attacked the remaining bottleneck by keeping the base model in 4-bit form while training LoRA adapters on top. By then the bottleneck was not the trainable parameters but the memory to hold the frozen base in the optimizer's address space. It quantizes the frozen base weights to 4-bit (an information-theoretically motivated NormalFloat format, NF4) and trains the LoRA adapters in higher precision on top, dequantizing each weight block only when it is needed for the forward and backward pass (Dettmers et al. 2023). Because the base never updates, its quantization error is fixed and tolerable; because the adapters are small, they stay in bf16 (a 16-bit floating-point format). The result is that a 65B model fits the adaptation step into roughly the memory a single 48GB card provides (Dettmers et al. 2023), which is the difference between needing a cluster and needing a workstation. With QLoRA the adaptation of a frontier-scale open model moved from a multi-accelerator job to a single-card one, and that is largely why the open-model fine-tuning ecosystem exists in the form it does.

The low cost is bought with fidelity, and the tradeoffs are worth naming precisely.

• Full fine-tune versus LoRA. A full fine-tune has the most capacity and is the safe choice when the adaptation is large or far from the base distribution. LoRA trades a small quality gap, usually negligible for instruction tuning and larger for tasks that need new knowledge, for a hundredfold cut in trainable parameters and the ability to keep one base with many swappable adapters. Rank is the knob: too low underfits the task, too high spends memory for no gain and starts to resemble a full fine-tune. Later large-scale practice moved the default: applied to every linear layer and run at roughly ten times the full-fine-tune learning rate, LoRA matches full fine-tuning whenever the adapter is not capacity-constrained, and for RL post-training it matches even at very low rank (Schulman and Thinking Machines Lab 2025).
• QLoRA's quantization error. Holding the base in 4-bit introduces a fixed error in every frozen weight. For most instruction tuning this is below the noise floor, because the adapters learn around it, but it sets a capability ceiling a precision-sensitive task can hit. The trade is memory for a small, bounded accuracy risk.

Cost rises linearly with rank, but the quality it buys saturates: drag the slider to watch task quality climb steeply at first, then flatten once the rank exceeds the update's intrinsic dimension, which is why a useful rank range exists.

Why two fine-tunes can be added together

Once adaptation is a small additive change, the changes compose, and that is the last word in the unifying claim. Model merging combines several fine-tuned checkpoints into one set of weights with no further training. The simplest form is weight averaging. Task arithmetic sharpens it: define a task vector $\tau_i = \theta_{\text{ft},i} - \theta_{\text{base}}$ as the difference between a fine-tuned model and its base. Here $\theta_{\text{base}}$ is the starting checkpoint, $\theta_{\text{ft},i}$ is the checkpoint fine-tuned for task $i$, and $\tau_i$ is the displacement caused by that task. These vectors add, subtract, and scale like directions in weight space (Ilharco et al. 2023). Adding two task vectors yields a model good at both tasks; negating one moves the model away from a behavior (Ilharco et al. 2023). The reason this works at all is that fine-tuning moves a model only a short distance across the loss landscape (the surface that maps each setting of the weights to its training loss), and over so small a neighborhood that surface is nearly flat, so the displacement behaves like a vector, and vectors compose.

The merging branch in Figure 17.3 arrived from a different direction than the efficiency spine. Model soups showed that averaging the weights of several models fine-tuned from the same initialization with different hyperparameters often beats picking the single best one, with no inference cost (Wortsman et al. 2022). Task arithmetic generalized averaging into a signed algebra of task vectors (Ilharco et al. 2023). The friction it exposed was interference: when two task vectors disagree on the sign of a parameter, naive addition cancels both contributions. TIES-merging addresses this by trimming small-magnitude changes, electing a sign per parameter, and merging only the agreeing updates (Yadav et al. 2023); DARE drops and rescales a large fraction of the delta before merging (Yu et al. 2024). These are the methods that let a community assemble a capable model by combining specialists instead of running one large alignment pipeline.

Merging trades reliability for near-zero cost, and the trade splits along two lines.

• Merging versus a fresh fine-tune. Merging is nearly free: no training, no data, just arithmetic on checkpoints. It is also approximate. The merged model is a compromise that can underperform a model actually trained on the union of the tasks, and interference between task vectors can erase a capability that each parent had. Merge when the parents are cheap and a good-enough combination is acceptable; train on the union when the combination must be reliable.

Operational form

A LoRA fine-tune is a small change to a standard training loop: wrap the target linear layers with the factored update, freeze everything else, and train only $A$ and $B$. The effective default is to wrap every linear layer, the MLP and MoE blocks included, and to run the adapter at roughly ten times the learning rate a full fine-tune would use; targeting only the attention projections, the early habit, underperforms even at matched parameter count (Schulman and Thinking Machines Lab 2025). A reader can find the canonical reference implementation in LoraLayer, peft/src/peft/tuners/lora/layer.py in the Hugging Face PEFT library, and the 4-bit NF4 base in bitsandbytes. The operational shape is worth holding in mind: one frozen base on disk, a directory of small adapter files, and a serving layer that loads the base once and swaps adapters per request, which is the deployment pattern PEFT exists to enable.

Three failure modes recur, one per consequence, and each has a different fix. The first is an underfit LoRA: the rank is too small for the adaptation, so the model half-learns the task and reverts under pressure. Raise the rank. A broken chat template is harder to spot, because serving with different role markers than training looks like a capable model that ignores instructions, when really it is being prompted off-distribution; the fix is to align the template exactly. Merge interference is the third, surfacing when two combined adapters degrade each other and the merged model ends up worse than either parent. Switch from naive averaging to a sign-aware merge, or train on the union instead.

Further reading