Training Agents to Act

The reasoning models of Chapter 28 are trained to produce a single answer: one prompt, one chain, one check. An agent does more. It takes many turns, calls tools, and reads what comes back before it is done. Agentic RL extends the verifiable-reward recipe from rewarding an answer to rewarding a whole trajectory, and that change pulls systems problems into the training loop. A rollout is now the agent's full interaction with an environment, not merely a longer completion; masking the environment's tokens out of the loss is what makes the sequence trainable; the hard part, credit assignment over a long trajectory, does not get any easier; the environment itself becomes a training asset; and inside that training run now sits a serving engine.

Rewarding a trajectory, not an answer

The machinery carries over, GRPO or PPO over sampled rollouts against a verifiable reward (Chapter 28), but a rollout is now an agent's full interaction with an environment rather than one completion. The shift is formal, not just longer. Ordinary language-model RL is a degenerate single-step decision problem, prompt in, one response out, reward, done; agentic RL is a temporally extended partially-observable process over turns, where the agent emits an action, the environment returns an observation, and this repeats (Zhang et al. 2025). This is where the training loop reaches into the agentic loop of Chapter 41: the harness that runs an agent at inference becomes, pointed at a checkable environment, the thing that generates training trajectories.

A rollout is then a single token sequence that interleaves two sources, the model's own action tokens (its reasoning, the tool or search call) and the environment's observation tokens (retrieved passages, tool output, browser state). The decisive mechanical move is loss masking: the policy gradient (a training method that nudges the model's own sampled-token probabilities up when an outcome scored well, down when it scored badly) is computed only over the tokens the model generated, with the observation tokens masked out, so the model is never trained to imitate text it did not author. Search-R1 states this precisely, masking retrieved tokens out of the loss, and reports that without it the gradient flows into copying retrieved text and training destabilizes (Jin et al. 2025). Everything downstream assumes this mask is in place.

The early exemplars are domain-shaped. WebGPT pointed here as far back as 2021, fine-tuning a model to browse and gather references, though its best system used behavior cloning and rejection sampling rather than full RL (Nakano et al. 2021). Search-R1 trains a model to interleave reasoning with live search queries across a multi-turn trajectory, rewarded only by whether the final answer matches (Jin et al. 2025). Moving from search to tool use in general, ToolRL applies GRPO and reports that a coarse answer-matching reward is not enough (Qian et al. 2025). SWE-RL scales the idea to software engineering, rewarding a patch by its similarity to the developer's real fix mined from open-source history (Wei et al. 2025), and ARTIST interleaves tool calls into the reasoning chain under an outcome reward (Singh et al. 2025). In each the reward is verifiable, computed by a checker rather than a learned reward model, which is what lets the loop run without a human-in-the-loop.

The recipe has since left the lab. OpenAI's Deep Research was trained end to end with RL on browsing and Python tool-use tasks, using the same methods behind its reasoning models (OpenAI 2025), and Moonshot's Kimi-Researcher was trained entirely through end-to-end agentic RL, averaging 23 reasoning steps and over 200 URLs per task (Moonshot AI 2025). Multi-turn tool-use RL under outcome rewards is now a standard stage of frontier flagship training rather than an academic exemplar (Zhang et al. 2025), which is why the systems split at the end of this chapter is a production concern, not a research one.

Credit assignment over a trajectory

What does not carry over cleanly is credit assignment, working out which of the agent's many actions across a long trajectory actually earned the reward, and it is the central difficulty. The trick every method here uses is to score an attempt by its advantage, how much better it did than the typical attempt, rather than by its raw reward. GRPO drops the value critic (a learned predictor of expected reward, the usual baseline an advantage is measured against) and estimates each rollout's advantage group-relatively, sampling a group of rollouts for one prompt and normalizing their returns (each rollout's total reward) against the group's mean and spread (Shao et al. 2024). Carried to agents, a group of full trajectories is scored by one outcome reward at the end and normalized across the group, but that single end-of-trajectory scalar is then broadcast to every action token in the episode. A good final answer rewards a wasted early tool call exactly as much as the decisive one. The signal is sparse and delayed, and that is where multi-turn RL gets hard: a value function estimated over many turns is high-variance, and the critic-free group methods give up per-step resolution entirely.

Three granularities of fix have emerged. The first shapes the reward over the action's form: ToolRL decomposes the reward into a format term that checks the tool call's structure and a correctness term that scores tool name, parameters, and values, a finer signal than answer-matching alone (Qian et al. 2025). The second manufactures step-level credit without a process-reward model. GiGPO keeps GRPO's critic-free stability but adds a second level of grouping: alongside the episode-level group it builds step-level groups by finding environment states that recur across different rollouts, an anchor-state grouping, and computes a local relative advantage among all the actions taken from the same recurring state, so each step gets a normalized credit at no extra rollout cost and reports double-digit gains over GRPO on agent benchmarks (Feng et al. 2025). ARPO takes a third route, noticing that token entropy spikes right after a tool call and branching its sampling at exactly those high-uncertainty steps to attribute tool-use credit (Dong et al. 2025).

The instability this provokes has a name. RAGEN, which optimizes the whole state-thinking-action-reward sequence with a method it calls StarPO, identifies an Echo Trap: the agent overfits to a few locally-rewarded patterns, and the collapse announces itself through a drop in reward variance across the group as an early warning, then an entropy drop and a spike in gradient norm after which it is hard to recover (Wang and others 2025). Its stabilized variant filters for the highest-variance prompts and borrows DAPO's asymmetric clipping, but the general lesson is that the multi-turn regime, with its compounding errors and larger correlated action space, pushes a policy to latch onto one template far harder than single-turn reasoning RL does.

Environments as the training substrate

The environment becomes a training asset in its own right, because in verifiable-reward RL the environment is the reward function. SWE-Gym packages roughly twenty-four hundred real repository tasks with executable runtimes and unit tests so an agent's trajectory can be sampled and scored in the loop, the unit test serving as the verifier, and training on it lifts resolve rates on SWE-bench by close to twenty points (Pan et al. 2024); this is the sandbox of Chapter 85 turned from a place to run an agent into a place to train one. The same executable-environment pattern is being mined from existing benchmarks, the app-and-API world of AppWorld and the self-hosted websites of WebArena among them, repurposed from evaluation harnesses into training grounds.

Building one is harder than it looks, and the difficulty is systems-shaped: runtimes must be reproducible, sandboxes isolated at scale, and the environment's execution latency now sits inside the RL loop, where a slow checker starves the trainer. The verifier can also be exploited, so an agent can satisfy an imperfect test specification rather than solve the task, the reward-hacking risk of Chapter 28 moved into the environment. The shift is large enough that environments are now collected and sold as products, a 2025 market of training grounds for agents (Zeff 2025), though the value of any single environment is contested, since a model that masters a skill can generate cheap substitute data for it (Anderson 2025). Both of those last sources are journalism and opinion, not peer-reviewed, and are flagged as such.

The systems split: generation and learning

An agentic RL loop generates rollouts from the current policy, scores them, and updates the weights, and generation is the bottleneck: OpenRLHF measures the rollout phase at around eighty percent of RLHF wall-clock (Hu and others 2024). So modern toolkits embed an inference engine, almost always vLLM, to produce the rollouts (Chapter 84). The training box now contains a serving box, and two architectural patterns split its compute. The colocated pattern keeps the trainer and the rollout engine on the same GPUs and reshards the model between them: HybridFlow, whose open implementation is veRL, reshards the actor between its training and generation layouts with what it calls a 3D-HybridEngine, reusing the training weights for generation so no memory is duplicated, under a hybrid single- and multi-controller programming model (Sheng et al. 2025). The disaggregated pattern puts the actor, reward, reference, and critic models on separate GPU pools and schedules them with Ray: OpenRLHF takes this route, embedding vLLM for the rollout phase that dominates wall-clock (Hu and others 2024). The choice is the familiar one between packing for utilization and disaggregating for scheduling flexibility, now made inside a single training run.

The deeper knob is how fresh the rollouts must be: whether the training data comes from the current policy (on-policy) or from slightly older weights (off-policy). Strict on-policy RL waits for the latest weights before each batch, which leaves the trainer idle while the slower generator runs, an idleness made acute by long agent trajectories where the batch waits on its slowest rollout. Asynchronous RL lets generation run ahead on slightly stale weights and overlaps the two (Noukhovitch et al. 2024), trading a controlled amount of off-policy staleness for throughput, the same staleness-against-utilization trade the pipeline schedules of Chapter 10 make at the gradient level. Pushed to its limit the two sides fully decouple, with rollout workers generating continuously while the trainer updates whenever a batch arrives; AReaL reports up to a 2.77x speedup this way (Fu et al. 2025). The price is that the data is off-policy, corrected by the importance-sampling ratio (a reweighting factor for reusing data the current policy did not generate) that PPO already carries, and how far staleness can be pushed is itself a research question, with methods that constrain the variance of those importance weights rather than just clipping them training stably on data hundreds of updates old (Zheng et al. 2025).

Further reading