Training and Fine-tuning in Practice

Changing weights is not the first step in customization, and a completed training job is not the end of the work. A fine-tune is justified only when a measured behavior change must live in the weights. It becomes useful only when the resulting artifact can be reproduced, evaluated, served, monitored, and rolled back.

The practical unit is therefore an adaptation release: an adaptation contract, a frozen baseline, a governed dataset, a training run, an evaluation report, and a versioned model artifact. The mechanics of supervised fine-tuning and parameter-efficient methods appear in Chapter 17; distributed training appears in Chapter 10. This chapter connects those mechanisms into an operating procedure for a product team.

Freeze the adaptation contract

Select the training method only after writing down what must change and what must not. The adaptation contract is the reviewable statement of that boundary. It prevents a team from treating a falling loss curve as proof that a product problem was solved.

The target behavior should be observable in examples before training begins. “Know our product” is not a contract. “Return a valid support-category object on at least 98 percent of the frozen test cases, without worsening refusal accuracy or the general-helpfulness retention set beyond its failure budget” is closer. The actual thresholds belong to the application, not to this book.

The frozen baseline includes the complete path that users see. A different chat template, retrieval index, tool schema, or decoder can move a score without any weight update. If those pieces are omitted, the experiment cannot attribute its result to fine-tuning.

Decide whether the behavior belongs in weights

Run the frozen baseline and the cheapest plausible alternatives on the same evaluation set. This comparison is the train-or-not gate.

• Put changing facts and source-backed answers in retrieval, where they can be updated and cited.
• Put exact syntax in a tool schema, grammar, validator, or structured decoder. A weight update cannot guarantee validity.
• Put a short, inspectable policy in the prompt or application code.
• Use tools when the task requires current state or an external effect.
• Fine-tune when a frequent, durable behavior must live in the weights and the measured gain survives the complete serving path.

Do not train merely because examples exist. Training is a poor substitute for a missing database, an authorization check, or a deterministic parser. It is also a poor response to an evaluation failure whose cause has not been diagnosed.

The available supervision then determines the next branch:

SFT learns from demonstrations. DPO learns from paired preferences without an online rollout loop (Rafailov et al. 2023). Classical RLHF first fits behavior from demonstrations, then learns a reward from rankings and optimizes a policy against that reward (Ouyang et al. 2022). These objectives consume different evidence and fail differently. A method name cannot repair labels that do not express the contract.

Build data that can be defended

A training example is not just text. It is a source record, a permission to use that record, a transformation history, a split assignment, a template revision, and a decision about which tokens receive loss.

Create a data provenance record before materializing examples. At minimum it contains a stable source ID, origin, collection time, owner, license or other legal basis, consent constraints where applicable, allowed purpose, retention period, sensitivity class, and deletion mechanism. Remove or explicitly govern personally identifiable information, credentials, customer secrets, and copyrighted material. A model checkpoint does not make deletion obligations disappear. Dataset-aggregator labels are not sufficient evidence: a large audit found frequent missing and misclassified license information and traced the need to retain original-source lineage (Longpre et al. 2024).

Structured data documentation makes those decisions visible. Data Cards record origins, collection and annotation processes, intended uses, and lifecycle maintenance (Pushkarna et al. 2022). The useful internal equivalent is a versioned dataset manifest whose rows point back to source IDs. When a source is withdrawn, lineage identifies every derived example, dataset version, run, and artifact that requires review.

Split by the unit that can leak

Randomly splitting rows is often wrong. Choose a grouping unit that keeps related material together: customer, conversation, document, repository, incident, author, or time window. Assign groups to train, development, and test once, then deduplicate within each split and across splits. Near-duplicates can inflate evaluation and increase memorization; language-model experiments have shown both effects (Lee et al. 2022).

Assign the split before fitting data filters, choosing examples from model scores, generating synthetic variations, or tuning a template against the corpus. Cluster exact and near duplicates before assignment when possible so a duplicate family moves as one group. Compare the resulting training inventory with internal evaluation suites and public benchmarks, including prompts used to generate teacher data, and publish the residual overlap in a contamination report.

The test set enters test quarantine before method selection. Training code must not read it, prompt authors must not tune against it, and synthetic-data generation must not use it as a seed. Store only its digest and access policy in the run manifest. A development set supports iteration; the quarantined test set supports the promotion decision.

Treat formatting as part of the artifact

Apply the exact tokenizer and chat template intended for training and serving. Validate role order, beginning and end tokens, tool-call serialization, truncation, and maximum length. Inspect decoded batches rather than trusting a configuration flag. For prompt-completion or conversational SFT, use a loss mask when the contract calls for assistant-only loss; otherwise the optimizer may spend capacity predicting user text or system scaffolding. Current TRL documentation, for example, distinguishes full-sequence, completion-only, and assistant-only loss behavior (Hugging Face n.d.).

For examples indexed by $i$, an assistant-masked SFT objective can be written as

$$\mathcal{L}_{\mathrm{SFT}}(\theta) = -\frac{1}{Z} \sum_{i=1}^{N} \sum_{t=1}^{T_i} m_{i,t} \log p_{\theta}(y_{i,t}\mid x_i,y_{i,<t}), \qquad Z=\sum_{i=1}^{N}\sum_{t=1}^{T_i}m_{i,t}.$$

where:

• $N$ is the number of examples, and $T_i$ is the token length of example $i$;
• $x_i$ is the prompt and other conditioning input;
• $y_{i,t}$ is output token $t$, while $y_{i,<t}$ is the preceding output prefix;
• $p_{\theta}$ is the model distribution with trainable parameters $\theta$;
• $m_{i,t}\in\{0,1\}$ is the loss mask, normally one on assistant answer tokens and zero on excluded prompt, padding, or scaffolding tokens; and
• $Z$ is the number of included tokens, which normalizes the loss.

Unit-test the formatter with a few fixed examples. The test should assert token IDs, labels, mask positions, truncation behavior, and round-trip decoding. A template change is a data change and a serving-interface change.

Measure memorization rather than assuming privacy

Fine-tuning on a small, repeated, or sensitive corpus can expose rare strings. Before training, scan for secrets and exact duplicates. During evaluation, probe held-out canaries and plausible prefixes, and record whether sensitive spans can be reproduced. Exposure testing was introduced precisely because ordinary loss and accuracy do not reveal unintended memorization (Carlini et al. 2019). These tests do not prove privacy, but they turn a vague concern into a release gate.

Choose the smallest weight change that meets the contract

Method selection has two independent axes: the objective determines what signal the model learns from, while parameter scope determines how much of the model can move.

Let $W\in\mathbb{R}^{d_{\mathrm{out}}\times d_{\mathrm{in}}}$ denote a weight matrix. LoRA freezes that matrix and learns a low-rank update (Hu et al. 2022):

$$W' = W + \Delta W, \qquad \Delta W = BA,$$

where:

• $W'$ is the adapted matrix;
• $A\in\mathbb{R}^{r\times d_{\mathrm{in}}}$ and $B\in\mathbb{R}^{d_{\mathrm{out}}\times r}$ are trainable factors;
• $d_{\mathrm{in}}$ and $d_{\mathrm{out}}$ are the input and output widths; and
• $r$ is the adapter rank, normally much smaller than either width.

For that matrix, each factor contributes to the trainable parameter count:

$$N_{\mathrm{LoRA}} = r(d_{\mathrm{in}} + d_{\mathrm{out}})$$

This count excludes biases and framework-specific scaling. Rank, target modules, scaling, dropout, and which non-adapter modules remain trainable are part of the experiment, not universal defaults.

QLoRA backpropagates through a frozen 4-bit base into higher-precision adapters; its paper combines NF4, double quantization, and paged optimizers (Dettmers et al. 2023). DoRA decomposes magnitude and direction rather than using the same update geometry as LoRA (Liu et al. 2024). Neither paper establishes a universal example-count threshold. Choose among LoRA, QLoRA, DoRA, and full fine-tuning through an empirical comparison on the same contract, beginning with the smallest viable pilot. Hold data, token budget, and evaluation constant when the purpose is to compare parameter scopes.

Size the run before launching it

The checkpoint size is not the training memory requirement. A useful accounting identity is

$$M_{\mathrm{peak}} = M_{\mathrm{params}} + M_{\mathrm{grads}} + M_{\mathrm{opt}} + M_{\mathrm{acts}} + M_{\mathrm{workspace}},$$

where:

• $M_{\mathrm{params}}$ is resident parameter and adapter storage at the chosen precisions;
• $M_{\mathrm{grads}}$ is gradient storage for trainable parameters;
• $M_{\mathrm{opt}}$ is optimizer and any master-weight state;
• $M_{\mathrm{acts}}$ is saved activation memory, which depends on batch, sequence length, model shape, and activation checkpointing; and
• $M_{\mathrm{workspace}}$ covers kernels, communication buffers, temporary tensors, the runtime, and fragmentation.

With $n_{\mathrm{shard}}$ data-parallel workers, a fully sharded method may bring parameter, gradient, and optimizer terms toward

$$M_{\mathrm{local}} \gtrsim \frac{M_{\mathrm{params}}+M_{\mathrm{grads}}+M_{\mathrm{opt}}} {n_{\mathrm{shard}}} + M_{\mathrm{acts}} + M_{\mathrm{workspace}},$$

but this is a planning lower bound, not a promise. Layer all-gathers, unsharded modules, activation placement, communication overlap, and optimizer choices change the peak. ZeRO established the state-sharding decomposition (Rajbhandari et al. 2020); current FSDP2 similarly shards parameters, gradients, and optimizer state while scheduling all-gather and reduce-scatter operations (PyTorch n.d.). Measure the exact model, sequence distribution, batch, precision, checkpointing policy, and backend with a short dry run. Then repeat a checkpoint save, interruption, and resume test before renting a large fleet.

Count the program, not only GPU time

The fixed cost of adaptation is

$$C_{\mathrm{adapt}} = C_{\mathrm{data}} + C_{\mathrm{train}} + C_{\mathrm{eval}} + C_{\mathrm{engineering}} + C_{\mathrm{deploy}},$$

where the five terms cover data preparation and review, training compute, evaluation, engineering time, and deployment work. If two systems meet equivalent quality and operational requirements, their costs after serving $V$ million accepted tokens can be modeled as

$$C_{\mathrm{base}}(V)=F_{\mathrm{base}}+Vc_{\mathrm{base}}, \qquad C_{\mathrm{tuned}}(V)=F_{\mathrm{tuned}}+Vc_{\mathrm{tuned}}.$$

The break-even volume is

$$V^*= \frac{F_{\mathrm{tuned}}-F_{\mathrm{base}}} {c_{\mathrm{base}}-c_{\mathrm{tuned}}},$$

provided $c_{\mathrm{base}}>c_{\mathrm{tuned}}$. Here $F_{\mathrm{tuned}}$ includes $C_{\mathrm{adapt}}$, the $F$ terms are fixed costs in dollars, and the $c$ terms are dollars per million accepted tokens. There is no meaningful crossover if quality, safety, or reliability differs, or if the tuned path is not cheaper per accepted token.

# Illustrative only. All costs are USD; volume is millions of accepted tokens.
base_fixed = 0.0
adaptation_program = 50.0
tuned_fixed = adaptation_program
base_rate = 5.0
tuned_rate = 0.5

if base_rate <= tuned_rate:
    print("no positive break-even volume")
else:
    break_even = (tuned_fixed - base_fixed) / (base_rate - tuned_rate)
    print(f"break-even volume: {break_even:.1f}M accepted tokens")

Select tools by capability

A framework name is not an architecture. Build a capability matrix from the contract, then verify every required cell against the project's official support matrix, a smoke test, and the intended model revision.

High-level post-training toolkits expose SFT, preference, reward-model, and RL trainers; TRL is one current example (Hugging Face n.d.). Adapter libraries such as PEFT implement LoRA-family methods and checkpoint operations (Hugging Face n.d.). Configuration-oriented projects such as Axolotl can be useful when a reviewed file should describe the run (Axolotl AI n.d.). Native distributed engines such as FSDP2 or ZeRO become relevant when model and optimizer state no longer fit the target topology. Managed services replace some infrastructure work, but they do not replace the data contract, evaluation gate, export decision, or rollback plan.

Do not select a tool from star counts, a vendor price snapshot, or a claim that one package is always fastest. Benchmark the smallest representative run. Pin versions and container images. Force a worker failure, restore from a checkpoint, export the artifact, and serve it through the target runtime before scaling up.

Make the run reproducible

The configuration submitted to a trainer is only part of a run manifest. The manifest binds data, code, model, environment, and evaluation into one identity. This is the production counterpart of reproducibility reporting for code, data, configuration, and experimental conditions (Pineau et al. 2021):

The prose names each field as well as the machine key: code revision, container digest, base checkpoint digest, tokenizer revision, dataset manifest, split rule, random seed, optimizer, scheduler, and checkpoint interval.

run_id: support-sft-0042
code_revision: <git-commit>
container_digest: <sha256>
package_lock_digest: <sha256>
driver_and_runtime: <versions>
hardware_topology: <device-and-interconnect-description>
base_checkpoint: <model-id-and-revision>
base_checkpoint_digest: <sha256>
tokenizer_revision: <revision>
prompt_template_revision: <revision>
dataset_manifest: <uri-and-sha256>
split_rule: <grouping-unit-and-assignment-version>
test_set_digest: <sha256-only-no-training-access>
objective: sft
parameter_scope: lora
target_modules: <explicit-list>
adapter_rank: <measured-candidate>
optimizer: <name-and-all-parameters>
scheduler: <name-and-all-parameters>
random_seed: <integer>
determinism_flags: <explicit-values>
precision: <parameter-gradient-optimizer-dtypes>
batching: <microbatch-accumulation-sequence-policy>
checkpoint_interval: <steps-or-tokens>
resume_state: <optimizer-scheduler-scaler-rng-sampler-cursor>
evaluation_plan: <version>

Resolve every placeholder before a run. Record the effective configuration after defaults and environment overrides, not only the file the operator intended to submit. Preserve data-loader order, packing policy, token counts, learning-rate history, gradient norms, skipped updates, throughput, peak memory, and checkpoint events. Resumable state includes the model or adapter, optimizer, scheduler, precision scaler, random-number generators, sampler or data-loader cursor, global token counter, and sharding metadata. Store the last complete checkpoint atomically and test restoration in a clean process.

Bit-for-bit replay is not always available across accelerators and kernels. The operational goal is still strict: the manifest must reproduce the data and objective exactly, restore a run, and produce results within a declared statistical tolerance. When a small dataset or unstable objective makes seed variance material, run more than one random seed and carry that uncertainty into the promotion report.

Evaluate the change, not the loss curve

Training loss is not a release metric. It measures fit to the training objective, including all defects in the labels and template. Promotion compares the candidate with the frozen baseline on evidence that was not used to choose the checkpoint.

Use at least five evaluation families:

1. Target-task quality. Score the named success criteria and every important task slice, including rare but costly cases.
2. Retention. Re-run general capabilities and product behaviors listed in the no-change requirements. Fine-tuning can cause catastrophic forgetting; this has been observed directly in language-model tuning studies (Li et al. 2024). How much is lost also depends on the method: at matched new-task quality, on-policy RL has been observed to retain more than supervised fine-tuning (Chapter 19).
3. Safety and policy. Test refusals, prompt injection, tool boundaries, demographic or language slices, and application-specific harms.
4. Privacy and memorization. Run secret scanning, canary exposure, and extraction probes appropriate to the data threat model.
5. Operational behavior. Measure output length, valid-structure rate, tool call correctness, time to first token, total latency, memory, throughput, and cost through the serving runtime.

Use paired prompts and the same decoder settings for baseline and candidate. Report the paired difference, confidence interval, sample count, and failures, not only an average. If multiple random seeds were trained, separate training variance from evaluation sampling uncertainty. A candidate passes only when its target gain clears the predeclared threshold and every retention, safety, privacy, and operational slice stays inside its failure budget. Chapter 87 develops the harness and statistical discipline behind this gate.

Human review remains necessary when the contract contains qualities that an automatic metric does not capture. Fix the rubric and blind reviewers to model identity where practical. Inspect disagreements and concrete failures before converting production traces into the next training set.

Package and deploy the complete artifact

The release is larger than adapter.safetensors. Package or reference:

• the base model ID, base model digest, and license;
• the adapter digest or full-checkpoint digest and its parameter scope;
• tokenizer files and revision, prompt template, tool schemas, and generation defaults;
• the run manifest, dataset manifest, code and container digests, and training logs;
• the promotion report, intended uses, excluded uses, measured slices, and known limitations; and
• the compatible serving runtime, precision, hardware class, and rollback instructions.

The package enters the serving path described in Chapter 82 only after those identities and compatibility constraints have been checked.

Model Cards provide a useful structure for intended use, evaluation conditions, limitations, and disaggregated results (Mitchell et al. 2019). Sign the manifest and bind it to artifact digests using the same lifecycle discipline as Chapter 89.

An adapter can remain separate or be merged into the base. Separate adapters reduce storage and can support several tasks over one base, but the serving runtime must load the correct base-adapter pair and account for adapter-switching latency and capacity. A merged checkpoint is easier for runtimes without adapter support, but it duplicates the base and needs its own license and digest review. Run merge parity on fixed prompts before release: compare separate-adapter and merged outputs or logits under the same tokenizer, precision, and serving runtime. Do not assume algebraic merge means identical runtime behavior after serialization or quantization.

Deploy first to a canary cohort with explicit rollback triggers. Compare quality proxies, errors, refusals, latency, token use, and task completion with the last-known-good version. Preserve execution provenance so an incident can be mapped back to the exact base, adapter, template, and runtime. A staged rollout does not relax the offline gate; it tests production conditions that offline data cannot reproduce. Operational rollback restores service behavior. It does not remove a withdrawn record's influence from already trained weights; that requires retraining, validated unlearning, or retirement under the deletion policy.

RL adds a serving system inside training

Online RL post-training contains a rollout generator as well as training workers. Generation produces trajectories, a reward path scores them, and the learner updates the policy. Every batch therefore needs a weight version. If rollout workers lag behind training workers, record and bound policy staleness rather than silently treating old trajectories as current-policy data.

A colocated topology places generation and training on the same accelerator pool. It avoids a dedicated rollout pool but may pay for memory contention, sleep or offload transitions, resharding, and phase switches. A disaggregated topology gives rollout and learner roles separate pools, allowing independent scaling but introducing transfer, scheduling, and freshness costs. HybridFlow formalizes one distributed RLHF dataflow and a mechanism for reshaping an actor between generation and training layouts (Sheng et al. 2025). Chapter 37 examines these choices in depth. The important boundary here is artifact identity: rollout data records policy, reward, tokenizer, environment, and sampling versions so the update can be audited.

Operate the adaptation lifecycle

A practical sequence is:

1. Freeze the contract. Name the target behavior, frozen baseline, no-change requirements, budgets, promotion gate, owner, and rollback path.
2. Test the no-training alternatives. Run prompt, retrieval, tool schema, and structured decoder changes against the same evaluation set.
3. Inventory and govern data. Record data provenance, rights, consent, sensitivity, retention, and deletion lineage.
4. Build splits once. Select the grouping unit, deduplicate across splits, and quarantine the test set before experiments.
5. Choose the smallest viable pilot. Match SFT, preference learning, RL, continued pretraining, or distillation to the supervision, then compare the smallest plausible parameter scopes.
6. Size and rehearse. Measure peak memory, throughput, checkpoint time, and cost; inject a failure and pass the resume test.
7. Run from a sealed manifest. Capture effective configuration, data and code digests, telemetry, checkpoints, and random seeds.
8. Write the promotion report. Compare the candidate with the baseline on target-task, retention, safety, memorization, operational, and human-review slices with uncertainty.
9. Package and canary. Verify digests, merge parity, serving compatibility, and rollback, then use a staged rollout with the last-known-good artifact still available.
10. Monitor and requalify. Feed reviewed failures into a new dataset version, never directly into the quarantined test set. Apply retention and deletion jobs to raw records, derived datasets, logs, caches, checkpoints, and model registries through the recorded lineage.

A requalification trigger includes a base-model change, tokenizer or template change, dataset or split-rule change, objective or parameter-scope change, serving-runtime change, precision or merge change, new device class, material traffic shift, policy change, or newly discovered data-rights issue. Each trigger creates a new artifact identity and reruns the affected gates.

The output is an adaptation decision record that links the contract, data lineage, run manifest, candidate artifact, promotion report, serving version, and rollback result. That record is what turns a training experiment into an operable part of the infrastructure.

Lower-layer constraint

Fine-tuning cannot compensate for an artifact that the serving stack cannot load, a tokenizer or template mismatch, insufficient accelerator memory, an unrecoverable checkpoint, or data that the organization has no right to use. The adaptation release therefore inherits constraints from storage, compute, distributed execution, artifact packaging, and serving. Those layers must preserve the identities and limits recorded in the contract; otherwise the evaluated candidate and the deployed system are different systems.

Capacity planning also runs downward. Sequence length and batch policy set activation memory, parameter scope sets optimizer and checkpoint state, and the rollout topology sets communication and freshness costs. A method is viable only when those lower layers meet the recovery, latency, residency, and cost requirements of the release.

Further reading

The primary papers define the adaptation methods and their measured scope; the systems and documentation sources explain how to operate the result.