Run agent RL across mixed GPUs without wasting a training cluster

Jason Mancuso of Modal explains an agent reinforcement-learning (RL) setup in this AI Engineer talk. Modal provides the infrastructure he describes.

Start with the agent you already run

A production harness, evaluation process and real failure traces can seed tasks and rewards for RL. Mancuso proposes generating training tasks from failures users actually encounter.
Supervised fine-tuning on recorded traces can be brittle when an agent later follows a different path. RL lets it sample alternative paths in the environment; that benefit depends on useful tasks and rewards.
He argues that owning inference and fine-tuning can help teams manage reliability, cost and proprietary tasks. These are his deployment claims, not universal requirements.

In the usual loop, sampler GPUs generate agent rollouts, a trainer updates the model and new weights go back to the samplers.
Running sampling and training in parallel uses GPUs better, but a rollout may come from an older model version. Sampler/trainer hardware and implementations can disagree too.
His recipe combines GRPO, masked importance sampling to handle per-token mismatch, score centering for accumulated drift, and top-p sampling with the sampling mask replayed in training.
He presents an ablation showing more stable training for his selected code environments; the talk does not supply an independently reproduced benchmark.

Modal’s GPUs span several providers and regions, making a large contiguous training cluster harder to schedule. In his example, roughly two inference GPUs are used for every training GPU.
Mancuso says compressing weight-update deltas cuts trainer-to-sampler transfer by a factor of about 1,000 in this run. Samplers can then run outside the training cluster.
Mixing GPU models and precision adds sampling mismatch; he reports stable training under the recipe and 15% lower rollout cost with cheaper GPUs. That is a rollout-cost comparison, not a 15% discount on the entire RL project.
Caveat: stability and cost numbers are Modal’s own selected experiment; the presentation does not show an independent replication or a costed end-to-end production deployment.

“If you have an agent deployment in production, you probably already have the ingredients you need.” — Jason Mancuso