Training and Fine-tuning in Practice

The theory of how weights move lives in Chapter 17 (supervised fine-tuning and parameter-efficient methods) and Chapter 10 (sharding an optimizer across many accelerators). Practice asks a narrower question: in mid-2026, with a task and a budget, which tools should change a model's weights, how should they wire into the rest of the stack, and should the team be training at all? The answer is opinionated and dated. Lead with the categories and trade-offs, which are durable, and treat every version number, price, and ranking as a snapshot to re-check.

The layer and what it contains

Training and fine-tuning is the layer where you change weights, as opposed to the serving layer where you only run forward passes (Chapter 82) or the agent and retrieval layers where you steer a frozen model with context (Chapter 85, Chapter 86), though that frozen model can itself be trained to act and not only to answer (Chapter 37). In a real 2026 stack it splits into three sub-layers that are easy to conflate and behave very differently.

1. Distributed engines. The heavy systems that shard a model and its optimizer state across many accelerators: PyTorch FSDP2 and TorchTitan, NVIDIA Megatron-Core, Microsoft DeepSpeed (ZeRO), and the JAX/XLA stack (MaxText) on TPUs. This is where 4D and 5D parallelism, FP8 training, and trillion-parameter scale live. Chapter 10 explains the mechanics.
2. Post-training toolkits. Higher-level libraries that wrap those engines for supervised fine-tuning (SFT), preference optimization (DPO, which tunes directly on preferred-versus-rejected answer pairs, and ORPO), and reinforcement learning (GRPO, which scores groups of sampled answers, alongside GSPO and PPO): Hugging Face TRL, Axolotl, Unsloth, LLaMA-Factory. This is what most teams actually touch.
3. Parameter-efficient methods and managed services. low-rank adaptation (LoRA), quantized low-rank adaptation (QLoRA), DoRA, carried by the PEFT library, plus hosted fine-tuning APIs that hide the accelerators entirely.

Two 2026 themes tie this layer to the rest of the book. FP8 training has gone mainstream: Google's Ironwood TPU has native FP8 in its matrix units, Megatron-Core ships FP8 training, and Unsloth does FP8 reinforcement learning (Google Cloud, 2026). More important for an infrastructure book, RL post-training has fused training with inference serving: modern RL toolkits embed an inference engine, almost always vLLM, to generate rollouts (sampled model attempts at a task), so the training box now contains a serving box. That is the wiring story at the end of this chapter.

The first fork: should you train at all?

Before tooling, answer the harder question. In 2026 the realistic default for many tasks is do not train weights. Prompt engineering, few-shot examples, retrieval (Chapter 86), and structured outputs cover much of what fine-tuning used to be reached for. OpenAI, while winding down its self-serve fine-tuning product, points users to prompt caching, smaller base models, and RAG instead (explainx.ai, secondary).

Fine-tune when you need a durable behavior change the context window cannot carry cheaply:

• a consistent output format or house style the prompt keeps drifting from,
• a narrow domain with vocabulary or conventions absent from the base model,
• a latency or cost win from distilling a large model's behavior into a small specialized one (this is the constraint arrow below),
• a capability shaped by reinforcement learning against a verifiable reward, the reasoning and agentic case from Chapter 28.

If none of these holds, spend the week on better prompts and retrieval, not on a training run.

Change the numbers (the one-time fine-tune cost and the two serving prices, all illustrative) and watch where the two cost lines cross: below the break-even volume the big API model is cheaper, above it the fine-tuned small model wins.

import numpy as np
import matplotlib.pyplot as plt

# Illustrative only: shape of the trade-off, not a quote.
finetune_cost = 50.0      # one-time fine-tune run, USD
big_price = 5.0           # large API model, USD per 1M tokens served
small_price = 0.5         # fine-tuned small model, USD per 1M tokens served

vol = np.linspace(0, 40, 200)            # millions of tokens served
big_total = big_price * vol              # no training, pay per token
small_total = finetune_cost + small_price * vol

break_even = finetune_cost / (big_price - small_price)
print(f"break-even at {break_even:.1f}M tokens served")

plt.plot(vol, big_total, label="big API model")
plt.plot(vol, small_total, label="fine-tuned small model")
plt.axvline(break_even, ls="--", color="gray")
plt.xlabel("tokens served (millions)"); plt.ylabel("total cost (USD)")
plt.legend(); plt.title("When does a fine-tune pay for itself?")
plt.show()

Key players (mid-2026)

The tables below are dated and source-attributed.

Distributed engines

Post-training toolkits (what most teams touch)

Managed services (no accelerators of your own)

Parameter-efficient methods

These are techniques, not products; they live inside PEFT, Unsloth, and TRL. See Chapter 17 for the theory of why a low-rank update suffices.

• LoRA. Low-rank adapters; the default. Zero added inference latency once merged into the base weights.
• QLoRA. A 4-bit NF4 (a number format for weights) frozen base plus LoRA; the single-GPU sweet spot for roughly 1K to 50K examples.
• DoRA. Weight-decomposed LoRA (magnitude plus direction); better convergence, worth it at larger data, roughly 50K to 500K examples.
• rsLoRA. Rank-stabilized LoRA for high ranks.
• Newer arXiv variants (RandLoRA, activation-space methods) exist but are not defaults. Treat as frontier, unverified in production (AppScale 2026).

How to choose

Once you have decided you genuinely need to train weights, four questions settle the tool.

Scale of the run. One consumer or single GPU points to Unsloth for its kernels and low VRAM. A few GPUs with config-as-code call for Axolotl or LLaMA-Factory. Multi-node pre-training or continued pre-training is the territory of TorchTitan/FSDP2, Megatron-Core, or MaxText on TPUs.

Ecosystem you live in. NVIDIA GPUs plus a desire for vendor support point to Megatron-Core/NeMo. If you want PyTorch-native and hackable, that is FSDP2/TorchTitan + TRL; on TPUs it is JAX/MaxText. Pure Hugging Face muscle memory needs only TRL directly.

Method. Style or format with small data points to QLoRA. A deeper domain shift, with more data behind it, wants DoRA or high-rank LoRA. For a structural domain change with 500K or more examples, the answer is full fine-tuning. Reasoning or agentic behavior against a reward belongs to GRPO/GSPO: Unsloth on a single GPU, verl for multi-node production runs, TRL or the experimental torchforge on plain PyTorch (Chapter 28).

Self-host versus managed. Managed wins on time-to-first-run, no GPU operations, and elastic cost, with predictable per-training-token pricing. Self-host wins on data residency and IP control, unusual recipes, repeated or large runs where you amortize hardware, and avoiding vendor lock-in or sudden deprecation. Note the convenience tax: Together prices LoRA per training token plus a separate hourly dedicated endpoint to serve the result, so the cheap headline number is not the full bill.

Concrete picks:

• Pick Unsloth when you are on one 24 to 80GB GPU and want the fastest, lowest-VRAM SFT or long-context GRPO.
• Pick Axolotl when you want reproducible YAML-configured runs across a few GPUs and value config-as-code in CI.
• Pick LLaMA-Factory when you need breadth of model support or a low-code UI for non-experts.
• Pick TRL directly when you are writing custom post-training logic and want the canonical trainers.
• Pick TorchTitan/FSDP2 or Megatron-Core when you are doing real multi-node pre-training or continued pre-training.
• Pick MaxText when you are on TPUs.
• Pick a managed service when you have no GPU operations appetite and a standard SFT/LoRA/DPO/RFT workload, but design an export path given 2026's deprecation precedent.

A sensible default (mid-2026)

For a typical product team that has decided it genuinely needs to fine-tune an open-weight model, as of June 2026:

Default: Axolotl (YAML config) on top of TRL + PEFT, doing QLoRA, on rented GPUs (Modal, Runpod, or your own cloud). It is reproducible, config-as-code fits CI and eval (Chapter 87), QLoRA keeps it single- or few-GPU, and you keep your weights and data. A starting recipe is r=16, DoRA on, target_modules="all-linear", and you raise the rank for harder tasks (AppScale 2026).

• Alternative A, maximum single-GPU efficiency or RL: Unsloth, especially for consumer GPUs or long-context GRPO.
• Alternative B, no GPU operations at all: a managed service. Together for cheap open-weight LoRA/SFT, Fireworks if you want tune-and-serve in one place, Bedrock if you are AWS-native, Tinker if you want to keep writing the loop yourself and rent only the distributed GPUs. Build an export and portability path; do not assume a managed FT product is permanent. OpenAI's self-serve wind-down, announced May 2026, is the cautionary tale.

For multi-node pre-training rather than fine-tuning, the default shifts to TorchTitan/FSDP2 or Megatron-Core, or MaxText on TPUs.

Wiring it into the stack

The defining 2026 wiring fact is that RL post-training now embeds the inference layer. A GRPO or PPO loop generates rollouts from the current policy, scores them with a reward, then updates the weights. Generation is the bottleneck, so toolkits delegate it to a fast inference engine, almost always vLLM. Unsloth drives vLLM for rollouts; torchforge runs vLLM for generation and TorchTitan for training, shuttling weights between them through an RDMA tensor store (PyTorch blog); verl, the production multi-node case, schedules vLLM or SGLang rollout workers and FSDP or Megatron trainers under one hybrid controller, the same pattern vLLM's own vime adopted in June 2026. So the training box in your architecture diagram contains a serving box. This is the same vLLM from Chapter 82, now used as a training subcomponent, a clean instance of the book's thesis that these layers are one infrastructure. The architectural patterns that split that loop, colocated versus disaggregated, and the asynchronous-versus-on-policy trade, are the subject of Chapter 37.

After training you merge the adapter, or serve it directly with an adapter-aware engine, hand the weights to the serving layer behind your gateway (Chapter 82), and route an eval harness against it before promotion (Chapter 87). The retrieval and agent layers then consume the served model. Adapter-aware serving, many LoRAs over one base, is where managed adapter platforms historically fit. The whole flow assembles into the end-to-end stack of Chapter 88.

A config and a launch you can adapt

A minimal Axolotl QLoRA config, illustrative and meant to be edited:

# qlora.yml
base_model: Qwen/Qwen3-8B
load_in_4bit: true          # QLoRA: 4-bit NF4 frozen base
adapter: lora
lora_r: 16
lora_alpha: 32
lora_target_modules: all-linear
datasets:
  - path: ./data/sft.jsonl
    type: chat_template
gradient_checkpointing: true
sequence_len: 4096
micro_batch_size: 2
num_epochs: 3
# distributed backend (FSDP2 or DeepSpeed) configured via accelerate

Launch it, then merge the adapter for serving:

# train (single or few GPUs via accelerate)
accelerate launch -m axolotl.cli.train qlora.yml

# merge the LoRA adapter back into the base for adapter-free serving
python -m axolotl.cli.merge_lora qlora.yml --lora_model_dir ./out

The merged weights are then handed to the serving stack. Keep the config in version control so a training run is a reviewable diff, the same discipline you apply to the rest of the infrastructure.

What fine-tuning really buys

Fine-tuning is first a capability lever, the durable behavior change a prompt cannot carry. Increasingly it doubles as an efficiency lever, distilling a large model into a small cheap-to-serve one, which is why the serving-cost constraint arrow points up so often. It also opens a trust surface: a fine-tuned model has absorbed your data, so data residency, reproducible config-as-code, and an eval gate before promotion are not niceties but the controls that let you ship the weights at all. The 2026 default is still to reach for prompting and retrieval first, and to train only when the behavior change has to live in the weights.

Further reading

First-hand sources first.