In September, Hugging Face published a write-up of TRL’s new asynchronous GRPO trainer [1]. They trained Qwen2.5-Math-1.5B with a rank-1 LoRA adapter on two H200s, generated rollouts on two more H200s running vLLM, and shipped only the adapter between them after each update. Then they tuned it, and a handful of changes took the run from 3 hours 27 minutes to 53 minutes with the same reward curve.
I wanted to know how much of that carries over to hardware that is nothing like an H200. So I reran the same model, data and script on two NVIDIA DGX Sparks on my desk, one training and one serving, with the adapter moving over an NFS share on the cable between them.
The reward curve reproduced: 0.143 to 0.413 over 500 steps, against their 0.145 to 0.438. The tuning advice did not, or not in the same order. Their biggest lever was worth 22% here, a smaller one was worth most of the gain, and their third does nothing at all. And one of the knobs turned out to change the training objective as well as the speed, which shows up clearly in how long the model’s answers get.
1. What Hugging Face did
Ordinary GRPO [2] alternates: generate a batch of answers, score them, take a gradient step, generate again. While one half works the other waits. TRL’s AsyncGRPOTrainer [3] splits them apart. A rollout worker keeps a vLLM server busy generating against the newest weights it has, finished rollouts go into a queue, and the trainer pulls from the queue as fast as it can. The price is that some rollouts come from a policy a few updates old. The trainer corrects for that with an importance ratio, the probability of each token under the current policy divided by its probability when it was sampled, and drops anything older than max_staleness updates.
With LoRA, the weights that change are tiny. Instead of broadcasting the full model to vLLM over NCCL, the trainer writes the adapter to shared storage every four steps and asks vLLM to load it by name. Hugging Face used a Storage Bucket mounted on every Job; the rest of their post is about making that work across separately scheduled Jobs.
The recipe: Qwen2.5-Math-1.5B, LoRA rank 1 on every linear layer, learning rate 4e-5, the 1,460-problem sail/Sanity-Test-R1D-1.5B math set, eight answers per problem, 128 answers per optimizer step, answers capped at 3,000 tokens, 500 steps. The reward is 1 if the last \boxed{} answer matches the reference and 0 otherwise.
Their baseline run took 22.9 seconds per step. They then read the trainer’s bottleneck metrics and changed three things in turn: token-budget packing, which cut forward and backward time from 21.9 to 5.6 seconds and raised model FLOPs utilisation from 3.9% to 19%; turning off gradient checkpointing, which took it to 4.6 seconds and moved the bottleneck to generation; and adding a third vLLM replica while raising the number of requests in flight, which removed the new bottleneck. That is the 3 h 27 min to 53 min. Their tuned run used TRL v1.14; I used 1.16.
2. The same run on two Sparks
| Hugging Face | This run | |
|---|---|---|
| Trainer | 2× H200, FSDP across both | 1 DGX Spark (GB10), single process |
| Generation | 2 vLLM replicas, 1 H200 each, behind a proxy | 1 DGX Spark, one vLLM server |
| Adapter transport | HF Storage Bucket | NFS over a direct 200 Gb link |
| Memory bandwidth per GPU | 4.8 TB/s | 273 GB/s |
Table 1. The two setups. A Spark has 128 GB of memory shared between CPU and GPU, plenty for a 1.5B model, but roughly a seventeenth of an H200’s memory bandwidth and far less compute.
I used Hugging Face’s own training script [4] with three changes: an import that swaps the attention kernel (below), an output directory on the NFS share, and an optional experiment tracker. Everything else, including every hyperparameter, is theirs. The trainer is single-process, so to keep 128 answers per step it accumulates 128 microbatches of one answer each, where their two ranks did 64 each.
Three things had to be fixed before the first step ran, and each is the kind of problem that will catch anyone running this off the beaten path.
- Version sniffing. TRL picks its vLLM weight-transfer API by version number. The community vLLM build I normally use on the Sparks reports a development version string, which TRL reads as older than 0.28, and it imports an API that no longer exists. The stock
vllm/vllm-openai:v0.31.0image runs on the Spark’s GPU and fixed it. - No FlashAttention 3. Packing (section 4) runs attention over variable-length segments with a FlashAttention 3 kernel fetched from the Hugging Face Hub. There is no build for ARM64 with PyTorch 2.13, and FA3 targets Hopper GPUs anyway. PyTorch’s own
varlen_attntakes the same packed layout, so a short shim swaps it in. Against a dense reference on a packed batch it agreed to about 2×10-3 relative error in the forward pass and 3×10-3 in the gradients, which is bf16 rounding. - A misleading error. The first adapter load failed with TRL reporting that the server “does not expose” the LoRA loading endpoint. It does. The trainer writes the adapter file readable only by its owner, the trainer container ran as root, and NFS maps a remote root to nobody, so vLLM got a 404 for a file it could see but not open. A storage bucket has no Unix file permissions to get wrong, so Hugging Face never hit this. Running both containers as the same ordinary user fixed it.
3. The baseline reproduces
| Hugging Face (r1-dp2) | Two Sparks | |
|---|---|---|
| Reward, first 20 → last 20 steps | 0.145 → 0.438 | 0.143 → 0.413 |
| Importance ratio, range over the run | 0.9993–1.0004 | 0.9976–1.002 |
| Mean staleness (limit 4) | 1.5 | 1.6, none dropped |
| Adapter syncs | 126, median 8.5 s | 126, median 0.31 s, no failures |
| Median step time | 22.9 s | 79.3 s |
| Wall clock, 500 steps | 3 h 27 min | 10 h 54 min |
Table 2. Baseline settings on both. Hugging Face’s 252 adapter loads are 126 syncs to two vLLM replicas. Averaged over the last 50 steps rather than 20, this run ends at 0.429; the difference between 0.413 and 0.438 is within the step-to-step noise of a single run.
Every property the asynchronous design depends on held. The importance ratio stayed within a quarter of a percent of 1, so rollouts generated by a slightly stale policy were nearly on-policy anyway. Staleness averaged 1.6 updates against a limit of 4 and never forced a rollout to be thrown away. Every adapter load succeeded, and over NFS on a direct cable each sync took a third of a second rather than eight and a half.
The run took 3.2 times longer than theirs, on one desk-side machine per role instead of two H200s each. The more interesting number is where that time went. Forward and backward passes were about 90% of every step, and the queue of finished rollouts sat near its cap of 512 almost the whole run. vLLM was waiting on the trainer, not the other way round, which is the same diagnosis Hugging Face started from.
4. The knobs rank differently
I tried their changes in their order, as 100-step runs compared against the baseline’s first 100 steps.
| Baseline | Packing | Packing, no grad checkpointing | |
|---|---|---|---|
| Samples per step | 128 | ~171 | ~170 |
| Tokens through forward + backward, per second | 2,165 | 2,255 (+4%) | 3,151 (+46%) |
| Answer tokens trained, per second | 1,744 | 2,009 (+15%) | 2,739 (+57%) |
| Samples trained per hour | 5,494 | 6,707 (+22%) | 9,010 (+64%) |
| Peak memory | – | 20 GB | 66 GB |
Table 3. Throughput over 100 steps. “Answer tokens” are the completion tokens the loss is computed on; the forward pass also processes the prompts. Packing used a 16,384-token budget and 12 microbatches per step (Hugging Face used 6 across two ranks).
Packing was worth 22%, not 3.9×. Without packing, each forward pass handles one answer of about 1,300 tokens. With it, the trainer lays answers end to end until a pass holds 16,384 tokens, with an attention mask that keeps each answer from seeing the others, so a step is 12 large passes instead of 128 small ones. It goes one step further: the eight answers to a problem share their prompt, so the prompt is stored once and every answer attends to that one copy.
On an H200 a 1,300-token pass through a 1.5B model is a tiny job. The GPU spends most of its time between kernels, which is how Hugging Face measured 3.9% utilisation, and packing filled those gaps. The Spark’s GPU is already busy at one answer per pass. The tokens it pushes through forward and backward each second barely moved, 2,165 to 2,255. What packing did save was the prompts: 11% of the baseline’s forwarded tokens were prompts repeated eight times, and with them stored once that fell to 2%. Nearly all of the gain is that.
Turning off gradient checkpointing was worth another 40%. Checkpointing saves memory by discarding activations in the forward pass and recomputing them during the backward pass, roughly an extra forward pass per step. On a GPU that is already the bottleneck, that recomputation is pure cost, and the Spark had memory to spare: the run peaked at 66 GB of 128. For Hugging Face this was the smaller step.
The third knob does nothing yet. After both changes the trainer was still the bottleneck. The rollout queue still refilled to about 475 of 512, the rollout worker still spent around 20 seconds per step blocked on a full queue, and the trainer never waited on generation. Sending vLLM more concurrent requests can only help when the trainer is starved, and here it never was. On Hugging Face’s hardware, the first two changes made the trainer fast enough to outrun two vLLM replicas. On mine, generation at about 2,800 tokens per second still keeps up with the faster trainer.
5. The full tuned run
With both changes, I ran the full 500 steps again.
| Baseline | Tuned | Hugging Face tuned | |
|---|---|---|---|
| Reward, first 20 → last 20 steps | 0.143 → 0.413 | 0.147 → 0.449 | 0.145 → 0.416 |
| Wall clock, 500 steps | 10 h 54 min | 9 h 19 min | 53 min |
| Samples trained | 64,000 | 96,670 | 84,078 |
| Hours to reach reward 0.30 / 0.35 / 0.40 | 4.8 / 6.1 / 8.2 | 2.7 / 3.8 / 6.7 | – |
| Mean staleness | 1.6 | 1.1 | 2.0 |
| Importance ratio, range | 0.9976–1.002 | 0.9999–1.000 | “very close to 1.000” |
Table 4. The full runs. Times to reach a reward use a 50-step moving average. Over the last 50 steps both of my runs average 0.43; the gap in the 20-step figures is noise.
Both runs end at the same reward, and so does Hugging Face’s. The tuned run got there faster, most clearly in the middle: 1.6 times faster to reach 0.35. Over all 500 steps the saving shrinks to 1.17 times, partly because the baseline’s own steps got quicker in its second half as its answers shortened.
The left panel of Figure 2 shows where the gain came from. Counted in optimizer steps, the tuned run was ahead early: it reached 0.30 at step 142 against the baseline’s 205, and 0.35 at step 204 against 262. Then the lead faded. Both runs reached 0.40 within a few steps of each other, at 359 and 366, and finished level. Each tuned step trained on 165 to 210 answers instead of 128, so it saw 51% more samples for the same final reward, and measured per sample it finished behind: after about 48,000 samples the tuned run was at 0.38 and the baseline at 0.43. The bigger steps helped while there was plenty left to learn, and stopped helping once both runs flattened out near 0.43. Some of the early lead may also come from the change in loss weighting described next, since packing brings both changes at once.
6. Packing also changed the objective
Packing does not change which samples the model sees. Rollouts are taken from the queue in order, nothing below the token budget is dropped, and advantages are computed per group of eight before any batching happens. The attention mask keeps packed answers independent. It looked like a pure speed change.
It is not, because of how the loss is averaged. In TRL’s asynchronous trainer, each microbatch’s loss is the sum of its per-token losses divided by its own token count, and every microbatch then counts equally. In simplified form:
loss = per_token_loss.sum() / tokens_in_this_microbatch / gradient_accumulation_stepsWithout packing, a microbatch is one answer, so this is a per-answer average: every answer carries equal weight however long it is, and a token in a 200-token answer weighs fifteen times as much as a token in a 3,000-token one. That is the original GRPO weighting, and the Dr. GRPO paper [5] pointed out its side effect. A long wrong answer is penalised less per token than a short one, which nudges the model toward long, rambling wrong answers. With packing, every microbatch holds about 16,000 tokens across many answers and the microbatches are nearly all the same size, so the result is close to a per-token average over the step, the weighting DAPO [6] recommends. Switching packing on quietly switched the objective.
The answers show it. Under the per-answer weighting, the baseline’s answers grew from about 1,340 tokens to a peak of 1,650 around step 130 before settling at 1,230, and at the peak nearly 30% of them ran into the 3,000-token cap. Under the per-token weighting, the tuned run’s answers never grew. They shrank steadily from 1,100 to 920 tokens, and by the end only 3% hit the cap against the baseline’s 15%. Both runs reach the same reward, so the tuned model gets the same share of answers right while saying less, and almost never fails by running out of room.
This is one run of each, so I would treat it as a strong hint rather than a result. The direction matches the mechanism, the gap held for all 500 steps, and the same split appeared in the 100-step runs, where both packed configurations stayed shorter than the baseline. Hugging Face’s own tuned run ended a little below their baseline, 0.416 against 0.438. It may be related, but that run also differed in staleness and replica count, and I have no way to separate the causes from their numbers.
7. What I would take from this
- The asynchronous design is robust. The importance ratio, staleness and adapter-only syncing all behaved on hardware with a completely different balance of compute, bandwidth and storage. NFS on a direct cable is a perfectly good adapter transport, and much faster than a bucket.
- Tuning advice is hardware-specific; the method is not. Hugging Face’s real contribution is the procedure: read where the time goes, fix that, read again. Followed on a Spark it produced a different ranking of the same knobs, and stopped one knob earlier.
- Measure speed-ups as wall-clock time to a target reward, not throughput. Packing and no checkpointing raised samples trained per hour by 65%. The extra samples per step helped early on, but the advantage faded, and the tuned run reached each reward level only 1.2 to 1.6 times sooner in wall-clock time, ending level with the baseline.
- Throughput settings can change the objective. If a batching change also changes how the loss is normalised, the speed comparison and the learning comparison are no longer the same experiment. Answer length is a cheap thing to watch.
The obvious next experiment is a second trainer. With two Sparks training and one generating, the trainer would stop being the bottleneck, and Hugging Face’s third knob would finally have something to do.
8. How this was measured
Software: TRL 1.16.0, vLLM 0.31.0, PEFT 0.21.2 and PyTorch 2.13, in containers built from the stock vLLM image on both machines. The training script is Hugging Face’s at commit 29dc7d2 with the three changes in section 2. vLLM ran with their flags and 60% of the serving Spark’s GPU memory, and was restarted before each run so adapter names started fresh.
Every number comes from the per-step metrics the trainer logs. Rewards are training rewards on the rollouts themselves; there is no separate evaluation set, as in the original write-up. Each configuration ran once, so differences of a few hundredths in reward are within noise; the throughput numbers are much more stable, since they average over thousands of passes. TRL has no peak-FLOPs figure for the Spark’s GB10, so I cannot report utilisation the way Hugging Face did.
References
- Async GRPO with LoRA across HF Jobs: a bucket, a proxy, and no NCCL, Hugging Face, September 2026.
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, which introduces GRPO.
- TRL, Hugging Face.
- hfjobs-lora-buckets, the training script from [1].
- Understanding R1-Zero-Like Training: A Critical Perspective, which introduces Dr. GRPO.
- DAPO: An Open-Source LLM Reinforcement Learning System at Scale.