Training Agents to Act

Training a model to answer and training it to act differ at the boundary where the model meets an environment. An answer may be scored after one completion. An agent chooses an action, receives an observation, and chooses again. The training example therefore includes not only model tokens but also tool results, environment state, termination rules, and the version of the policy that generated the run.

There was no single invention called “agent training.” WebGPT demonstrated in 2021 that a language model could learn to browse through behavior cloning and human-feedback-based selection; its strongest reported system used rejection sampling rather than online policy optimization (Nakano et al. 2021). Later work such as Search-R1 optimized multi-turn search behavior with an outcome reward (Jin et al. 2025). These are different training regimes. Demonstrations remain useful for teaching valid actions and providing a cold start; rejection sampling can select good complete runs; offline objectives can reuse recorded runs; and online reinforcement learning (RL) can collect new runs from the changing policy. This chapter focuses on the additional contracts required when the unit of learning is an interaction trajectory. The planning and runtime architecture used after training belong to Chapter 38 and Chapter 41.

One trajectory has two time scales

A language model always generates tokens sequentially. Agent training adds a second time scale: after one or more model-generated tokens form an action, an external system advances and returns an observation. Calling ordinary language-model RL “single-step” hides the token decisions that are already there. The useful distinction is one environment exchange versus repeated action--observation exchanges.

A partially observed interaction can be written as

$$\begin{aligned} h_t &= (x, o_0, a_0, o_1, \ldots, a_{t-1}, o_t), \\ a_t &\sim \pi_\theta(\,\cdot \mid h_t), \\ s_{t+1} &\sim P_E(\,\cdot \mid s_t, a_t), \\ o_{t+1} &\sim \Omega_E(\,\cdot \mid s_{t+1}), \\ \tau &= (x, o_0, a_0, o_1, \ldots, a_{T-1}, o_T), \\ R(\tau) &= \sum_{t=0}^{T-1} \gamma^t r_t + \gamma^T r_T^{\mathrm{term}}. \end{aligned}$$

Here, $x$ is the task; $t$ indexes environment turns; $s_t$ is the hidden environment state before action $t$; $o_t$ is the observation exposed to the agent; $a_t$ is the action; and $h_t$ is the observation history available to the policy. The policy $\pi_\theta$ has parameters $\theta$. The environment revision $E$ determines the transition distribution $P_E$ and observation distribution $\Omega_E$. The termination condition sets the final turn $T$. The step reward is $r_t$, the terminal reward is $r_T^{\mathrm{term}}$, and the discount $\gamma\in[0,1]$ controls how strongly later rewards count. The full trajectory is $\tau$, and its return is $R(\tau)$. Every symbol in these equations corresponds to a policy, environment, or scoring decision that must be recorded.

An action need not be plain text. It may be a structured function call, a code patch, a browser command, or a multimodal control. Likewise, an observation may be structured or multimodal. Serializing both into one token transcript is a common implementation, not the definition of a trajectory.

Turn the interaction into a valid training record

Suppose a serialized record contains tokens $z=(z_1,\ldots,z_J)$. Here the provenance mask means

$$m_j = \begin{cases} 1, & z_j \text{ is a policy-generated token included in the update},\\ 0, & z_j \text{ comes from the prompt, system, or environment}. \end{cases}$$

A schematic masked policy objective can be read as

$$\mathcal{L}_{\mathrm{policy}}(\theta) = -\frac{1}{\sum_{j=1}^{J} m_j} \sum_{j=1}^{J} m_j\,\widehat{A}_j \log \pi_\theta(z_j\mid z_{<j}).$$

In these expressions, $j$ indexes serialized tokens; $J$ is the record length; $z_{<j}$ is the prefix before token $j$; $m_j$ is the binary provenance mask; and $\widehat{A}_j$ is the training weight or advantage assigned to an included token. Prompt and system tokens have $m_j=0$, as do environment observations. The denominator counts included action tokens and must be nonzero. Algorithm-specific objectives may add probability ratios, clipping, a reference policy, entropy terms, or value losses. The invariant is simpler: an environment observation may condition later actions, but it is not an action sampled by the policy. The mask is an ownership rule, not a claim that observations are unimportant.

Search-R1 applies this rule to retrieved passages: retrieved tokens remain in context but are excluded from the policy loss (Jin et al. 2025). A production record needs the same distinction for system and prompt tokens, tool responses, browser state, compiler output, and injected error messages. It must also state whether private reasoning tokens are retained or trained on. Without explicit roles, a preprocessing change can silently train the model to predict text it did not author.

The record should preserve more than tokens:

Invalid calls require an explicit policy. The environment may reject them and return an error observation, end the episode, or assign a penalty. Dropping them after collection changes the training distribution. Truncated trajectories also need a distinct termination reason; “ran out of budget” is not the same outcome as “task completed.”

Choose the data regime before the optimizer

“Agent training” does not imply online RL. The data source determines what the optimizer can learn:

WebGPT is an example of the first and second rows rather than evidence that online RL is always required (Nakano et al. 2021). In practice, a valid tool-call grammar is often taught before sparse outcome optimization begins. Otherwise the early rollout budget may be spent rediscovering the parser rather than learning the task.

Design the reward as a contract

The environment and the reward are not the same object. The environment changes state and produces observations. A verifier or reward wrapper assigns scores to actions, states, or complete trajectories. Keeping those interfaces separate makes failures diagnosable.

Agent rewards commonly combine several signals:

• Syntax and validity: Did the action parse, name an available tool, and use a valid schema? This provides dense feedback but does not show that the action was useful.
• Intermediate progress: Did a state become closer to the goal? This can improve credit assignment, but a misspecified progress metric can reward a detour or prevent a valid alternative route.
• Terminal task outcome: Did tests pass, was the answer correct, or did the environment reach the goal state? This is closest to the intended result but is often sparse.
• Cost and constraints: How many tokens, tool calls, seconds, dollars, permissions, or safety violations did the run incur? These terms expose trade-offs that a success bit hides.

ToolRL's experiments show why a coarse answer match and a structured tool-use reward are not interchangeable (Qian et al. 2025). SWE-RL uses similarity to a developer patch as a reproducible proxy, not proof that every semantically valid repair received the same score (Wei et al. 2025). Multiple action sequences can solve the same task. Exact action matching may reject a better route, while a weak final checker may reward a superficial one.

Reward hacking is therefore an interface failure as much as an optimization failure. Keep a held-out verifier that is not used for training, add tests that probe known loopholes, and inspect the gap between proxy reward and the intended task metric. Human-created tests and reference artifacts remain human supervision even when scoring is automated.

Trajectory reward is not action-level credit

Group Relative Policy Optimization (GRPO) removes the learned value critic used by PPO and normalizes returns within a group of samples for the same task (Shao et al. 2024). For a group of $G$ trajectories, one simple form is

$$\bar{R}=\frac{1}{G}\sum_{i=1}^{G}R_i, \qquad \sigma_R=\sqrt{\frac{1}{G}\sum_{i=1}^{G}(R_i-\bar{R})^2}, \qquad \widehat{A}_i=\frac{R_i-\bar{R}}{\sigma_R+\varepsilon}.$$

Here, $i$ indexes a trajectory in the group; $G$ is the group size; $R_i$ is trajectory $i$'s return; $\bar{R}$ is the group mean; $\sigma_R$ is the group standard deviation; $\varepsilon>0$ prevents division by zero; and $\widehat{A}_i$ is the normalized trajectory advantage. If every return is the same, this estimator supplies no ranking signal.

With only an undiscounted terminal reward, implementations commonly apply the same trajectory-level advantage to every included action token. That update reinforces or suppresses the whole sampled route. It does not identify which action caused the result. Calling this value “credit assignment” can hide the missing causal resolution.

Several remedies target different parts of the problem:

• A step-level reward changes the feedback timing. It helps only if the intermediate metric is trustworthy.
• A learned value function estimates expected future return from a state or history, enabling per-step advantages but adding a model that can itself be hard to fit under partial observation.
• Replaying alternative actions from the same state or checkpoint creates a local counterfactual comparison. Exact recurrence may be rare in open or stochastic environments.
• GiGPO builds step-level groups from repeated anchor states while retaining a critic-free group estimator (Feng et al. 2025). Its reported gains are scoped to ALFWorld, WebShop, and search-augmented question answering; the mechanism depends on states that can be recognized as equivalent.

Reward design, credit estimation, and exploration are independent choices. A more detailed reward does not by itself create better exploration, and branching more rollouts does not make a flawed verifier correct. RAGEN's reported “Echo Trap” is a useful warning from three stylized environments: falling reward variance and rising gradients accompanied collapse in those experiments (Wang et al. 2025). It is a diagnostic to test, not a universal law of agent RL.

Where the environments come from

Online RL consumes environments the way pre-training consumes text. One run samples many trajectories per task and returns to the task set for many steps, so the supply of executable, checkable tasks becomes the binding constraint. Curated benchmarks do not meet it. They hold a few thousand instances drawn from a handful of repositories, and a collection that is also the evaluation set cannot be spent on training.

Synthesizing tasks is the response, and the published pipelines share one move: begin from something already checkable, then work backwards to a task. SWE-smith installs a Python repository, perturbs the source until an existing test fails, and keeps the broken state as the task with that test as its verifier. Because the verifier is inherited rather than written, the pipeline scales, reporting 50,000 instances from 128 repositories against at most thousands from eleven or fewer repositories in earlier collections, and a 32B student at 40.2% Pass@1 on SWE-bench Verified (Yang et al. 2025). R2E-Gym derives its tasks from commits using test generation and back-translation, reaching more than 8,700 of them (Jain et al. 2025).

Outside code the state has to be built rather than borrowed, because there is no test suite waiting in the repository. Agent World Model backs each of 1,000 environments with a database and ordinary code, on the argument that a language model asked to simulate an environment gives less consistent transitions than a program does (Wang et al. 2026). AgentScaler constructs simulated tool environments and trains in two phases, general function calling first and domain specialization second (Fang et al. 2025). Endless Terminals synthesizes 3,255 terminal tasks over file operations, databases, and scripting; plain PPO on that set moved Qwen2.5-7B from 10.7% to 53.3% on the authors' development set, and the gain transferred to human-curated terminal benchmarks (Gandhi et al. 2026). Huang et al. survey these pipelines as one loop of generation, execution, and feedback (Huang et al. 2025).

A generated task is not automatically a useful one

Generation is the cheap half. The filter decides what the run can learn, and the criterion it should apply is not difficulty.

Take a task the current policy solves with probability $p$ under a binary outcome reward $R\in\{0,1\}$. The variance of that reward across samples is

$$\operatorname{Var}[R]=p(1-p),$$

where $p$ is the current policy's success rate on that one task. The expression peaks at $p=1/2$ and falls to zero at both ends. The group estimator above inherits the shape directly: with $G$ samples of one task, every $\widehat A_i$ is zero whenever the $G$ returns agree. So a task the policy always fails and a task it always solves sit at opposite ends of difficulty and carry the same training value, which is none. This is the reason DAPO resamples until a batch holds prompts with mixed outcomes (Chapter 28). When the task set is generated rather than given, the same argument applies one level earlier, to which tasks are kept at all.

Two practices follow. A solvability filter comes first, because a generator will emit tasks that no policy can complete; sampling each candidate a fixed number of times and discarding those never solved is the usual form. The surviving set then has to track a moving frontier, since $p$ rises as the policy improves and a task that carried signal last week can stop carrying it. GenEnv makes that the generator's objective, paying a curriculum reward for tasks matched to the agent's current ability so the environment supply co-evolves with the policy (Guo et al. 2025).

A warm start is a separate requirement, not an instance of this one. A policy that cannot yet emit a valid tool call fails every generated task for reasons unrelated to the task, which pins $p$ at zero everywhere and returns no signal at any difficulty. The hybrid-curriculum row above is the answer: teach the interface from demonstrations, then let outcome optimization work on a set whose outcomes actually vary.

What synthesis does not settle

Gains on synthesized tasks are in-domain by construction. The quantity that decides whether the pipeline worked is transfer to a held-out benchmark the generator never saw, and that is where reported evidence is thinnest. Contamination is the same problem from the other side: a generator seeded from public repositories can reproduce an evaluation instance without copying it, and exact-match deduplication will not find it. Where breadth stops paying is also open. These pipelines establish that breadth matters, not the point at which another hundred environments stop earning their cost. Thousands of live containers per rollout batch are an infrastructure bill rather than a data-generation one, which is the subject of the next two sections.

Treat the environment as versioned data infrastructure

An executable environment is reusable training infrastructure. SWE-Gym, for example, packages 2,438 real Python repository tasks with runtime environments, unit tests, and natural-language tasks. Its ICML 2025 study uses the collection for supervised agent training and verifier-guided inference-time scaling, and reports up to a 19-percentage-point resolve-rate gain in its evaluated setups (Pan et al. 2025). That result demonstrates the value of a reproducible environment; it should not be retold as an online-RL result.

A training environment needs an operational contract:

• Reset and replay: A reset must restore a known snapshot. Record the seed, environment version, data version, clock policy, and allowed network state. Measure deterministic replay rather than assuming it.
• Isolation: Separate filesystem, process, network, credential, and tenant scope. A rollout must not read another rollout's artifacts or the held-out answer.
• Observation control: Bound output size, label truncation, preserve source roles, and treat tool output as untrusted input.
• Failure semantics: Set a timeout for each action and trajectory. Distinguish parser rejection, tool failure, retry exhaustion, cancellation, and genuine task failure.
• Verifier integrity: Run scoring outside the agent's writable boundary, version its dependencies, and retain enough evidence to audit a score.
• Dataset separation: Keep training tasks, tuning tasks, and final evaluation tasks separate at the repository, template, and dependency levels, not only by issue identifier.

These controls affect learning. A flaky test adds reward noise. A slow browser extends the rollout tail. A leaked solution turns memorization into apparent success. A changed API alters the transition distribution even if the task text is unchanged.

Lower-layer constraint: rollout time shapes optimization

For trajectory $i$, the following terms give a useful wall-clock decomposition:

$$T_{\mathrm{rollout},i} = T_{\mathrm{reset},i} + \sum_{t=0}^{T_i-1} \left(T_{\mathrm{decode},i,t}+T_{\mathrm{env},i,t}\right) + T_{\mathrm{verify},i}.$$

For a synchronous batch of $B$ trajectories, the corresponding approximate critical-path formula is

$$T_{\mathrm{iteration}} \approx \max_{1\le i\le B} T_{\mathrm{rollout},i} + T_{\mathrm{update}} + T_{\mathrm{weight\ sync}}.$$

Here, $T_{\mathrm{reset},i}$ is environment setup time; $T_i$ is trajectory $i$'s number of turns; $T_{\mathrm{decode},i,t}$ is policy generation time for action $t$; $T_{\mathrm{env},i,t}$ is tool or environment time; $T_{\mathrm{verify},i}$ is scoring time; $B$ is batch size; $T_{\mathrm{update}}$ is learner time; and $T_{\mathrm{weight\ sync}}$ is the time to make new weights available to rollout workers. The maximum exposes the straggler: one long tool call can hold the whole synchronous batch. Real systems may overlap some terms, so measure the stage trace rather than treating this sum as a hardware law.

OpenRLHF reports that generation can dominate RLHF wall-clock and uses Ray, vLLM, and DeepSpeed to organize its roles (Hu et al. 2025). In an agent workload, decoding may dominate, or environment and verifier latency may dominate. vLLM is one possible inference backend, not part of the definition. Admission control, cancellation, prefix reuse, and batching still matter, but the unit being scheduled is now a trajectory that can pause between turns.

Placement and freshness are orthogonal

Two systems decisions are often conflated:

These axes are orthogonal. Separate GPU pools can still wait at a synchronous barrier, and a colocated cluster can time-slice roles while allowing queued rollouts from older weights. HybridFlow's 3D-HybridEngine is one colocated design that reshards the actor between generation and training layouts (Sheng et al. 2025). OpenRLHF demonstrates a Ray-based distributed layout (Hu et al. 2025). Neither placement alone determines whether the data is on-policy.

Asynchronous RLHF separates generation and learning and shows the resulting throughput--off-policy trade-off on instruction following and mathematical reasoning (Noukhovitch et al. 2025). AReaL goes further with continuous rollout workers and a staleness-aware learner, reporting up to 2.77 times the training speed of evaluated synchronous systems on its math and code workloads (Fu et al. 2025). Those results establish systems possibilities, not a guarantee for long-horizon tool agents.

Every asynchronous trajectory should carry its behavior-policy version. Here, if trajectory $i$ was generated by behavior policy $\mu_i$ and the learner uses $\pi_\theta$, a token-level importance ratio is

$$\rho_{i,j}(\theta) = \frac{\pi_\theta(z_{i,j}\mid z_{i,<j})} {\mu_i(z_{i,j}\mid z_{i,<j})}, \qquad m_{i,j}=1.$$

Here, $i$ indexes trajectories; $j$ indexes a policy-generated token; $z_{i,<j}$ is that token's recorded context; $m_{i,j}=1$ identifies an included action token; $\mu_i$ is the recorded behavior policy; $\pi_\theta$ is the current learner; and $\rho_{i,j}$ reweights the old sample under the current policy. Clipping ratios can limit variance but cannot repair missing support or arbitrarily stale trajectories. Products of ratios across many tokens and turns can become extreme, so record policy lag, ratio distributions, and clipping fractions rather than labeling a run merely “asynchronous.”

Verify learning, environments, and systems together

A useful evaluation sheet keeps three layers separate:

Compare systems on matched hardware and a matched workload distribution. A throughput improvement obtained by shortening trajectories, dropping slow tasks, or changing the verifier is not a systems-only improvement. Report learning curves against environment interactions, generated action tokens, accelerator time, and wall-clock time; each denominator answers a different question.

Before release, replay held-out tasks with the production parser and tool permissions, perturb tool latency and failures, test budget exhaustion, and inspect high-reward failures manually. Training has succeeded only if the policy improves under an independent evaluation contract without buying the score through excess cost or unsafe action.

The result of this pipeline is a policy that has learned from interaction. The next chapter asks a different question: which runtime loop should expose that policy to planning, memory, tools, and the outside world?

Further reading