Supervised Fine-Tuning and PEFT

A base language model and an assistant can share the same architecture and most of the same weights. Their behavior differs because they were trained on different conditional distributions. Pre-training rewards likely continuations of text. Supervised fine-tuning (SFT) rewards a specified response after a specified conversation. That change can improve instruction following, teach a new output schema, or specialize the model to a domain. It does not guarantee truthfulness, safety, or knowledge that the examples never support.

Two decisions are often collapsed into the word fine-tuning:

1. What is the training signal? SFT defines target tokens and minimizes their next-token loss.
2. Which parameters may change? Full fine-tuning updates the model, while parameter-efficient fine-tuning (PEFT) updates a smaller set of parameters or adds a small trainable module.

SFT and PEFT therefore answer different questions. SFT is an objective and data pipeline. LoRA is one way to parameterize the update. A model can undergo SFT with every weight trainable, with LoRA, or with another PEFT method.

Turn conversations into supervised tokens

An SFT record may contain a system message, several user and assistant turns, tool calls, and tool results. The model does not receive those fields directly. A chat template serializes them into one token sequence, including the exact role and boundary tokens expected by the tokenizer. Training and serving must use the same template. A mismatch changes the prefix distribution even when the visible conversation looks identical.

Let the serialized sequence be (u_{1:T}=(u_1,ldots,u_T)), and let a binary mask (m_t) select the tokens that should contribute to training. A common assistant-only objective is

$$\mathcal{L}_{\mathrm{SFT}}(\theta) = -\frac{1}{M} \sum_{t=1}^{T} m_t\log p_\theta(u_t\mid u_{<t}), \qquad M=\sum_{t=1}^{T}m_t.$$

Here (T) is the sequence length; (u_t) is the token at position (t); (u_{<t}) is its prefix; (p_\theta) is the model distribution with parameters (\theta); (m_t=1) for a scored target token and (m_t=0) for an ignored token; and (M) is the number of scored tokens. The normalization by (M) makes the loss an average over targets rather than a sum that grows with sequence length.

For assistant-only training, the mask usually selects assistant messages and may include the assistant end-of-turn token. System messages, user messages, padding, and tool results are context but not targets. This is a policy choice, not a definition of SFT. Some recipes score the whole sequence, only the final assistant turn, or tool-call tokens under a separate mask. State the choice because it changes what the model is asked to imitate.

Several implementation details sit inside this apparently simple pipeline:

• End-of-turn tokens matter. Without a learned stopping boundary, a model may continue into a simulated user turn or produce several answers.
• Truncation changes supervision. Left truncation can remove the instruction; right truncation can remove the answer or its stop token. Log the fraction of records and target tokens lost.
• Packing needs boundaries. Concatenating short records improves accelerator utilization, but each record still needs a terminal token and the intended attention policy. Otherwise one example can become context for the next.
• Length changes weighting. Token-averaged loss gives a long answer more influence than a short one. Example-balanced sampling and token-balanced loss solve different problems.

Multi-turn supervision adds another choice. A recipe can score every assistant span in one serialized conversation, or create one example per assistant turn and score only that response. In both cases, later responses are conditioned on the recorded earlier replies during teacher forcing, not on errors the model might make in a live conversation. Full conversational rollouts are therefore a separate evaluation.

InstructGPT used labeler demonstrations for an SFT stage before preference modeling and reinforcement learning (Ouyang et al. 2022). Its final behavior cannot be attributed to SFT alone. Earlier instruction-tuning work also showed that training across many tasks can improve generalization to held-out task types (Wei et al. 2022). These results establish SFT as more than format selection, while leaving its effect dependent on the base model and training set.

Data determines the behavior being copied

SFT supplies positive demonstrations. It says, “produce this continuation in this context.” It does not directly say which alternative was almost acceptable, which defect made another answer worse, or how strongly two requirements should trade off. Preference data in Chapter 18 adds that kind of comparison.

The LIMA experiment is a useful boundary case. A 65-billion-parameter LLaMA model fine-tuned on 1,000 curated examples learned strong response formats and broad conversational behavior in the reported human evaluation (Zhou et al. 2023). That result supports the claim that a strong base model can need little data to adopt a style. It is not a general sample-complexity law. Teaching specialized facts, a new tool protocol, a language underrepresented in pre-training, or a difficult reasoning procedure can require much broader coverage.

A useful dataset audit separates five concerns:

Duplicate prompts can make a random split look better than true generalization. One dominant source can overwhelm a smaller capability even when the record counts appear balanced, because long answers contribute more target tokens. Incorrect demonstrations are especially costly: maximum-likelihood training rewards the model for reproducing them without expressing that the label was uncertain.

Safety examples need the same distributional care. Too many broad refusals can produce over-refusal, meaning that the tuned model rejects benign requests because it learned a boundary wider than the intended policy.

Evaluation should compare the base model, a prompting baseline, and the tuned model on the same held-out tasks. Report task metrics, instruction and format compliance, calibration where relevant, and regression suites for capabilities that should remain unchanged. Tulu 3 illustrates the value of separate development and unseen evaluations plus decontamination rather than choosing a checkpoint on the public test set (Lambert et al. 2025). Training loss alone only shows that the model fits the supervised tokens.

Choose which parameters may move

The update strategy is independent of the SFT dataset. Its main trade-offs are trainable state, optimizer memory, adaptation capacity, checkpoint storage, and serving behavior.

Freezing a parameter means the optimizer does not change it and does not need gradient or optimizer state for it. The frozen parameter is still used in the forward and backward computations. PEFT therefore reduces some important memory terms, but it does not make the base model, activations, temporary buffers, or communication disappear.

A useful memory ledger is

$$M_{\mathrm{peak}} \approx M_{\mathrm{base}} +M_{\mathrm{adapter}} +M_{\mathrm{grad}} +M_{\mathrm{optim}} +M_{\mathrm{act}} +M_{\mathrm{work}}.$$

Here (M_{\mathrm{peak}}) is peak accelerator memory; (M_{\mathrm{base}}) stores frozen base weights; (M_{\mathrm{adapter}}) stores trainable PEFT parameters; (M_{\mathrm{grad}}) stores their gradients; (M_{\mathrm{optim}}) stores optimizer state; (M_{\mathrm{act}}) stores activations needed for backpropagation; and (M_{\mathrm{work}}) covers temporary kernel, communication, and dequantization buffers. The approximation omits allocator details, so measure the actual peak.

LoRA constrains each update to a low-rank factorization

For a frozen linear map (W_0\in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}}), low-rank adaptation (LoRA) replaces the effective weight with

$$W = W_0+sBA, \qquad A\in\mathbb{R}^{r\times d_{\mathrm{in}}}, \qquad B\in\mathbb{R}^{d_{\mathrm{out}}\times r}.$$

Here (d_{\mathrm{in}}) and (d_{\mathrm{out}}) are the input and output dimensions; (r\le\min(d_{\mathrm{in}},d_{\mathrm{out}})) is the adapter rank; (A) and (B) are trainable; and (s) is a fixed scale, commonly (\alpha/r) in the original parameterization for a chosen multiplier (\alpha). The update obeys (\operatorname{rank}(sBA)\le r). For an input (x\in\mathbb{R}^{d_{\mathrm{in}}}), the output is (Wx=W_0x+sB(Ax)). Implementations commonly initialize one factor so that (BA=0), making the adapted model equal to the base model before the first update.

The base matrix contains (d_{\mathrm{out}}d_{\mathrm{in}}) parameters. The LoRA branch contains

$$N_{\mathrm{LoRA}} =r(d_{\mathrm{in}}+d_{\mathrm{out}}), \qquad \rho =\frac{N_{\mathrm{LoRA}}} {d_{\mathrm{out}}d_{\mathrm{in}}}.$$

Here (N_{\mathrm{LoRA}}) is the number of trainable adapter parameters for this matrix and (\rho) is their fraction of the base-matrix parameter count. The total for a model is the sum over every targeted matrix, plus any biases, embeddings, or output heads deliberately left trainable.

The runnable performs the same exact accounting. It describes one matrix, not a complete transformer.

d_in = 4096
d_out = 4096
ranks = [4, 8, 16, 32, 64]

full = d_in * d_out
print(f"base matrix parameters: {full:,}")
for rank in ranks:
    adapter = rank * (d_in + d_out)
    fraction = 100 * adapter / full
    print(f"rank {rank:>2}: {adapter:>7,} parameters ({fraction:>6.3f}%)")

The low-rank form is a capacity constraint, not proof that every useful fine-tuning update has low rank. The original LoRA paper found competitive results in its evaluated settings (Hu et al. 2022). Later experiments on mathematics and code found that standard low-rank LoRA learned less of the new domain than full fine-tuning while retaining more of the base model's behavior (Biderman et al. 2024). Rank, target modules, dataset size, batch size, initialization, scale, and learning rate all affect the comparison.

Target selection matters as much as rank. Attention-only LoRA was common in early work, but later experiments found gains from targeting MLP and, where applicable, MoE projections as well. A 2025 report found high-rank LoRA could match full fine-tuning in its Llama and Qwen sweeps when adapters covered all linear layers and were not capacity-constrained; it also found a different best learning rate and worse behavior at large batch sizes (Schulman and others 2025). Those are empirical results, not universal defaults. Sweep the update strategy on the actual dataset.

QLoRA changes the storage of the frozen base

quantized low-rank adaptation (QLoRA) keeps the frozen base weights in a 4-bit NormalFloat representation and trains LoRA factors in a higher compute precision. During computation, weight blocks are dequantized as needed; gradients flow through the computation to the adapter parameters, while the quantized base remains unchanged. Double quantization compresses quantization constants, and paged optimizers manage temporary memory spikes (Dettmers et al. 2023).

The QLoRA paper demonstrated fine-tuning a 65-billion-parameter model on one 48 GB GPU in its reported configuration. That is an existence result, not a hardware rule. Sequence length, batch size, activation checkpointing, attention kernels, adapter coverage, optimizer, and temporary buffers can move the peak substantially. Four bits per base parameter is also only the ideal payload for the quantized values; scales and other metadata add storage.

QLoRA separates training precision from the deployment format. A QLoRA-trained adapter can be served beside a quantized base, beside a higher-precision base, or after a supported merge and requantization path. These choices need their own quality and latency tests. “Trained with QLoRA” does not by itself mean that the final server executes every operation in four bits.

Merged and switchable adapters have different costs

After training, a LoRA update can be materialized as

$$W_{\mathrm{merged}}=W_0+sBA.$$

Here the merged matrix uses the original linear operation, so the LoRA branch adds no separate matrix multiplications at inference. The cost is operational: the result is another full-size weight set, and that model instance no longer has a small independent adapter to switch off. Keeping (A) and (B) separate preserves modularity but adds adapter computation, memory reads, and routing. The real latency depends on batch shape, kernel fusion, rank, target modules, and whether requests in a batch use the same adapter.

Adapter hot-swapping also has compatibility constraints. A server must pair an adapter with the exact base architecture and parameter names it was trained against, plus the compatible tokenizer and chat template. Current PEFT tooling can replace LoRA weights in place, but target modules and compiled shapes constrain which adapters can share a slot (Hugging Face 2026). “One base, many adapters” is a deployment design, not an automatic property of an adapter file.

Weight merging is a separate approximation

Do not confuse merging one LoRA branch into its own base with combining several independently fine-tuned models. The latter is model merging. It operates on compatible parameter coordinates and tries to preserve several behaviors without another training run.

For fine-tuned parameter vector (\theta_i) derived from the same base (\theta_0), define the task delta and a weighted merge as

$$\tau_i=\theta_i-\theta_0, \qquad \theta_{\mathrm{merge}} =\theta_0+\sum_{i=1}^{n}\lambda_i\tau_i.$$

Here (i\in{1,\ldots,n}) indexes the source fine-tunes; (\tau_i) is the parameter displacement for source (i); (\lambda_i) is its chosen merge coefficient; and (\theta_{\mathrm{merge}}) is the candidate merged model. The operation assumes matching architecture, tensor shapes, tokenizer-related parameters, and parameter alignment. A shared initialization supplies that alignment. Arithmetic on unrelated checkpoints is not justified by the equation.

Model soups found that averaging checkpoints from one fine-tuning sweep often improved the evaluated models (Wortsman et al. 2022). Task arithmetic showed that adding or negating task deltas can steer behavior in tested settings (Ilharco et al. 2023). Neither result says that arbitrary skills add cleanly. TIES trims small deltas, resolves sign conflicts, and merges values that agree with the selected sign (Yadav et al. 2023). DARE randomly drops delta parameters and rescales those retained before a later merge (Yu et al. 2024). These methods manage interference; they do not certify that every parent capability survives.

When the training data can be combined, a joint fine-tune provides direct evidence about the joint objective. A training-free merge is useful when data or compute is unavailable, or as a cheap candidate in a measured merge sweep. Test every constituent task, general capabilities, calibration, and safety after the merge.

Run adaptation as a controlled experiment

A reliable workflow keeps the data, optimizer, parameterization, and deployment artifact distinguishable:

1. Define held-out task and regression suites before inspecting final results.
2. Freeze the base checkpoint, tokenizer, chat template, maximum length, and target-mask policy as one versioned contract.
3. Audit source mixture, duplicates, contamination, truncation, and target-token counts.
4. Measure the base model and a prompt-only baseline.
5. Sweep learning rate and at least one capacity setting, such as LoRA rank or target modules. Do not reuse full-fine-tune hyperparameters without testing.
6. Compare PEFT with a small full-fine-tune baseline when hardware permits.
7. Evaluate the exact artifact to be served: unmerged adapter, merged weights, and quantized deployment can behave differently.

The evidence boundary

SFT makes a model more likely to reproduce demonstrated behavior under a particular serialization. PEFT changes the set or form of parameters allowed to carry that update. Neither choice determines the other, and neither removes the need for held-out evaluation.

Small curated datasets can be enough for style and familiar tasks. LoRA and QLoRA can make those experiments much cheaper. The claims stop there. New knowledge, distant domains, long training runs, precision-sensitive tasks, and combined skills may expose capacity, forgetting, quantization, or interference limits. Choose the smallest update that passes the target and regression suites, then verify the exact deployed artifact.

Further reading