How long does agent RL with Qwen3.8-27B take on two NVIDIA DGX Sparks, with one Spark generating rollouts in vLLM and the other training a LoRA adapter with TRL’s asynchronous GRPO trainer? About 2 to 4.5 days for a modest 200-step agent run, and 0.9 to 1.3 days for single-turn reasoning. Which half sets the pace depends on how much of each trajectory the model writes itself: the trainer runs forward and backward over every token, tool output included, while generation only produces the model’s own tokens.
The main findings:
- Training: skipping PEFT’s fp32 cast of LoRA inputs makes training 11% faster with bit-identical results. Turning off gradient checkpointing does not fit in memory.
- FP8 weights: generation is 45% faster, but the sampler and the trainer disagree about token probabilities 2.6 to 2.9 times as much as with BF16. Whether that hurts learning is not tested here.
- Attention kernel: on the Spark, vLLM 0.31’s default attention kernel (FlashAttention 2) runs the verify step of speculative decoding (MTP) at about half speed. This is why long agent sessions slowed to about 2 tokens per second, and why MTP measured slower than no drafting at 64 trajectories. Adding
--attention-backend FLASHINFERroughly doubles throughput at long context, and MTP is then 1.6 to 1.9 times faster at every load measured, up to 16 sessions. - Custom kernel: a Triton decode kernel written to confirm the cause reads the KV cache at 99% of the measured bandwidth limit. FlashInfer with an FP8 KV cache is within a few percent of that, so better attention kernels have little left to gain.
- Four-bit weights: NVIDIA’s NVFP4 checkpoint of the model is a further 1.1 to 1.7 times faster than FP8 weights. Output quality is not measured.
- Which half is the bottleneck: generation is the slower half when the model wrote a larger share of each trajectory than generation speed divided by training speed. Long, reasoning-heavy agent sessions (57% generated) are generation-bound with FP8 weights, and NVFP4 weights bring generation level with training. Agents that mostly read tool output are training-bound, and lower-precision rollouts gain them nothing. One Spark can train sequences of up to 32K to 40K tokens.
Sections 1 and 2 describe the setup and the baseline, sections 3 and 4 each speed-up, section 5 the run-time estimates, section 6 the long-session measurements and the kernel comparison, and section 7 which half is the bottleneck.
1. The setup
Qwen3.8-27B [1] is a 27.8B-parameter dense model under Apache 2.0, and a hybrid: three of every four layers are Gated DeltaNet linear attention, the fourth is full attention. I had planned to use Gemma 4 12B, but TRL’s asynchronous trainer packs several samples into one sequence and keeps them apart with a FlashAttention variable-length kernel. Those kernels support attention head sizes only up to 256, and Gemma 4’s global attention layers use 512 [2].
| Spark 1 (generation) | Spark 2 (training) | |
|---|---|---|
| Software | vLLM 0.31.0 | TRL 1.18.0 AsyncGRPOTrainer, PEFT 0.21.2 |
| Model | Qwen3.8-27B, BF16, LoRA adapter served | same weights, LoRA rank 32 on every linear layer of the language model |
| Memory | 75% of 128 GB for weights and KV cache | ~73 GB peak with gradient checkpointing |
| Link | adapter written to an NFS share on the direct cable between them, ~11 s per sync | |
Table 1. One Spark generates, one trains. Each has 128 GB of memory shared between CPU and GPU and 273 GB/s of bandwidth.
Two things needed work on the Sparks’ GB10 chip. TRL’s trainer loads FlashAttention 4 from the Hugging Face kernel hub on any GPU of compute capability 10 or above. FlashAttention 4 is written for datacenter Blackwell (B200, capability 10.0); GB10 is capability 12.1, closer to the RTX 50 series, and the hub has no FlashAttention 4 build for it. So I swap in PyTorch’s varlen_attn, as in my earlier AsyncGRPO post [3]. The Gated DeltaNet layers run on TRL’s own Triton kernels. TRL’s packing parity tests, which check that a packed batch gives the same per-sample loss and gradients as running samples one by one, passed 8 of 8 on GB10 with these kernels.
For training I used a random reward, so no group is ever dropped for having no variance and nothing is learned: the point is timing. Each step trains on about 65,000 tokens, 32 samples of 2,048 generated tokens. For generation I wrote a load generator shaped like an agent: a 3,000-token system prompt shared by every trajectory, a unique 600-token task, then four turns of 1,024 generated tokens, each followed by a 400-token tool observation, so each turn resends a longer history.
2. The baseline
Generation. One stream runs at 4.5 tokens per second. That is 92% of the ceiling, since every token has to read all 55.6 GB of weights and 273 / 55.6 is 4.9. Throughput comes from running many trajectories at once: 33 tokens/s at 8, 93 at 32 and 115 at 64, where the KV cache starts evicting shared prefixes and it levels off. Serving the LoRA adapter instead of the base model costs about 4%, and asking for the top-8 log-probabilities that the trainer needs costs nothing measurable.
Training. The trainer does forward and backward at 206 tokens per second, so a 65,000-token step takes 314 seconds, steady to within 1% across steps. The first step of any run takes about 40 minutes, because TRL starts with only a few rollouts in flight and raises the number as it goes.
3. What I tried
| Change | Side | Result |
|---|---|---|
| Skip PEFT’s fp32 cast of LoRA inputs | training | +11%, identical numerics |
| FP8 weights in vLLM | generation | +45% at 32 trajectories, +44 to 57% at 64; 2.6 to 2.9× larger sampler/trainer log-prob gap |
| Speculative decoding (MTP, 2 tokens) | generation | 2.07× at 1 trajectory, 1.35× at 32, 0.93× at 64 (with vLLM’s default attention kernel; 1.6 to 1.9× up to 16 sessions after switching it, section 6) |
| Long prompt shared by a group (tree packing) | training | 2.07× more trained tokens per second |
| Gradient checkpointing off | training | does not fit, even at one sample per micro-batch |
| Half-size micro-batches | training | −2% speed, −9 GB memory |
Allow staler rollouts (max_staleness 4 → 8) | both | no wait for rollouts in later steps; −1% over five steps |
Table 2. Each row is one change against the baseline. Training rows are 5-step runs; generation rows are the load generator at 32 and 64 trajectories in flight.
A free 11% in PEFT
I profiled one training step expecting attention or the Gated DeltaNet kernels to stand out. They did not: attention was about 2 seconds of the step and the DeltaNet kernels 18. Matrix multiplies took 38% of the GPU time, which is fine, but copies took another 22%.

Most of the copies came from one line. PEFT keeps the LoRA matrices in fp32 so the optimizer’s small updates are not rounded away, and before multiplying it casts each layer’s bf16 input to fp32 to match [4]. But TRL trains under bf16 autocast, which casts that input straight back to bf16 for the matrix multiply. That is two activation-sized copies per adapted layer, in the forward pass, again in the checkpoint recompute and again in the backward pass, across 496 adapted layers, on a chip whose limit is memory bandwidth.
PEFT already has a switch for it: setting cast_input_dtype_enabled = False on the LoRA layers. Converting bf16 to fp32 and back is exact, so the outputs and both sets of gradients are bit-identical, which I checked on a single layer before using it. The step went from 314 to 284 seconds. What is left is mostly real work, plus about 13 seconds of LoRA’s scaling multiply, which could be folded into the matrix multiply at the price of slightly different rounding.
FP8 rollouts: fast, and a little off-policy
Generation is limited by reading weights, so halving their size should help, and it does. With FP8 weights the same agent-shaped load runs at 130 tokens per second at 32 trajectories instead of 90, and at 164 instead of 105 to 114 at 64. The smaller weights also leave room for 66% more KV cache, so at 64 trajectories the shared prefixes stop being evicted.

The catch is that the trainer still computes in BF16, so the model that samples a token is no longer exactly the model that is trained on it. TRL measures this. I tried two kinds of FP8: vLLM quantizing the BF16 weights on the fly with one scale per tensor, and Qwen’s own FP8 release [5], which uses one scale per 128×128 block and keeps the output layer, embeddings and a few small projections in BF16. Both are equally fast.
| BF16 | FP8, on the fly | FP8, Qwen checkpoint | |
|---|---|---|---|
| Mean log-prob gap, sampler vs trainer | 0.011 | 0.031 | 0.028 |
| Worst token | 2.1 | 18.7 | 6.4 |
| Tokens dropped from the loss, per step | 0.001% (about 1) | 0.025% (11 to 22) | 0.020% (11 to 16) |
| Time for 5 steps | 87 min | 51 min | 51 min |
Table 3. Averages over five training steps of 65,536 trained tokens each. The ratio is the trainer’s probability of a sampled token divided by the sampler’s; TRL drops tokens whose ratio is outside 0.5 to 5.
TRL corrects for the gap. Its loss uses the sampler’s own log-probabilities as the reference and weights each token by the ratio of the trainer’s probability to the sampler’s, which accounts both for the sampler running a few steps behind the trainer and for its lower precision. Tokens whose ratio falls outside 0.5 to 5 are dropped. With FP8 that dropped 11 to 22 tokens per step, against 0 to 2 with BF16, so almost all of the signal is kept, but the per-token weights are further from 1 and the gradient is noisier. Qwen’s checkpoint removes the worst disagreements; on average both FP8 versions sit at 2.6 to 2.9 times the BF16 gap. Whether that costs anything in learning I cannot tell from a random reward, so it is the first thing to check in a real run.
Speculative decoding, a first look
Qwen3.8 ships with a multi-token-prediction layer that vLLM can use to draft two tokens ahead and verify them in one pass. I measured it on real text, since acceptance depends on what is being written: 67% of drafted tokens were accepted. On one stream that doubles speed, 4.1 to 8.5 tokens per second, but the gain shrank as more trajectories ran at once: 1.35× at 32, and at 64 it was 7% slower than no drafting at all. The cause is vLLM’s default attention kernel on GB10 (section 6). Every MTP step verifies three tokens per sequence, the real one and two drafts, and that kernel runs a three-token step at about half the speed of a one-token step, a penalty that grows with the number of sequences. With FlashInfer’s kernel instead, MTP was 1.6 to 1.9 times faster at every load I re-measured, up to 16 sessions; I have not repeated the 32- and 64-trajectory runs. MTP also took about 70,000 tokens of KV cache and lowered the prefix-cache hit rate by 20 points at every level, which matters for agents that resend their history each turn.
What did not fit, and what packing buys
Gradient checkpointing saves memory by recomputing each layer’s forward pass during backward, and here the recompute is about 29% of a step. Turning it off ran out of memory at a 97 GB cap, even with a single 2,148-token sample per micro-batch: the activations need at least 45 GB on top of 52 GB of weights. Halving the micro-batch size with checkpointing on cost 2% and saved 9 GB, a useful trade if longer trajectories ever need the room.
TRL packs each group of samples as a tree, so a prompt shared by the eight answers to one problem is forwarded once instead of eight times. With a 2,400-token shared prompt and 1,024-token answers, packing reached 2.37× of an ideal 2.59×, the gap being groups split across micro-batches, and trained tokens per second roughly doubled. It also showed that longer sequences cost about 13% more per token. For an agent, only the system prompt and the task are shared across a group, so I expect about 1.3× there.
4. Putting it together

Figure 3 is the most useful picture from the whole exercise. In the baseline the trainer spends more time waiting for rollouts than computing, so making the trainer faster by itself changes nothing: the PEFT fix saved 30 seconds of compute and the trainer waited 30 seconds longer. FP8 shrinks the wait to about 100 seconds, and only then can the trainer’s speed show up. With both changes the five steps took 50 minutes against 87 for the baseline, but only 1 minute less than FP8 alone: during these short runs TRL is still raising the number of rollouts in flight, and the trainer’s 30 seconds went back into waiting. By the fifth step neither FP8 run waited at all, so in a long run the PEFT fix should pay off in full.
5. How long would RL take?
Generation and training overlap, so a step takes about as long as the slower of the two. Generation only costs the tokens the model writes, but training has to run forward and backward over the whole trajectory: prompt, tool observations and answers. The more of each trajectory is prompt and tool output rather than the model’s own tokens, the more the trainer dominates (section 7).
| Run shape (tasks × rollouts × turns × tokens per turn) | Baseline | FP8 + PEFT fix | Bottleneck |
|---|---|---|---|
| Single turn: 4 × 8 × 1 × 2,048 (what I measured) | 1.3 days | 0.9 days | generation |
| Small agent: 8 × 8 × 4 × 384 | 2.4 days | 2.2 days | training |
| Medium agent: 8 × 8 × 6 × 512 | 4.6 days | 4.1 days | training |
| Large agent: 16 × 8 × 8 × 1,024 | 21 days | 18 days | training |
Table 4. Estimated time for 200 steps, from the measured rates: generation 115 tokens/s (BF16) or 164 (FP8) at 64 trajectories, training 206 or 228 tokens/s, 13% less for multi-turn trajectories. Agent rows assume 200 to 400 observation tokens per turn, a 1,500 to 3,600-token prompt and 1.25 to 1.3× packing. Add 40 minutes for the first step.
For single-turn reasoning the two changes save about a third. For these agent shapes they save 10 to 15%, because the bottleneck moves to the trainer, where neither does much; section 7 shows when that reverses. What would help agents is fewer trained tokens: shorter observations, fewer rollouts per task, compacting the history every few turns, and only training on tasks the model sometimes solves, since a group where every rollout gets the same reward teaches nothing. Those are choices about the task, not the hardware.
6. Long agent sessions and the attention kernel
After the benchmark I ran the model as a real agent: Qwen3.8-27B building 3D assets in Blender inside sandboxed virtual machines, served from one Spark with Qwen’s FP8 checkpoint and MTP, 10 to 14 sessions at a time. As the sessions’ contexts grew, each slowed to about 2 tokens per second, and total throughput fell from 85 to 20 tokens per second. The cause is vLLM’s default attention kernel on GB10, which runs the three-token verify step of MTP at about half speed; this section shows the measurements.
The agent’s turns did not look like my load generator. The model thinks for long stretches between tool calls, its context grows to tens of thousands of tokens, and the tools themselves are almost free. Generation at long context is nearly the whole cost.
| Per turn | Median | 90th percentile | Max |
|---|---|---|---|
| Tokens generated | 309 | 6,414 | 32,768 |
| Context length | 20,300 | 56,200 | 82,000 |
Table 5. 234 turns from 23 agent sessions. Tool calls took 0.4 seconds per turn on average, and 78% of prompt tokens came from the prefix cache.
To measure this cleanly I replaced the agent with a benchmark: a number of sessions decoding at once, each with a context of natural text plus a question, with thinking on so that MTP accepts about 60% of its drafts, as it did for the agent. I also replayed the agent sessions’ real mix of context lengths. Figure 4 is the server as I had been running it.

A single session should barely slow down. Only the 16 full-attention layers keep a KV cache, 64 KB per token, so at 64K tokens a step reads about 4 GB of cache on top of 28 GB of weights. Instead single-session speed fell 32% by 64K and halved by 128K. More sessions made it worse much faster than the total context explains: four sessions at 64K gave half the throughput of eight at 32K, with the same number of tokens in the cache. The slowdown grows with the length of each sequence, not with the bytes read.
On GB10, vLLM 0.31 runs the full-attention layers with FlashAttention 2, and because Qwen3.8 is a hybrid it makes their KV cache pages 800 tokens long, to line up with the DeltaNet state. Others had reported nearly flat curves on a Spark using FlashInfer’s XQA decode kernel [8]. Switching to it is one flag, --attention-backend FLASHINFER.

| Total tokens/s | FlashAttention 2 (default) | FlashInfer | FlashInfer, FP8 KV cache |
|---|---|---|---|
| 1 session, 128K context | 7.0 | 10.7 | 11.6 |
| 16 sessions, 16K | 72 | 111 | 126 |
| 16 sessions, 32K | 49 | 91 | 105 |
| 8 sessions, 64K | 33 | 48 | 64 |
| 4 sessions, 128K | 17 | 27 | 32 |
| Agent mix, 8 sessions | 54 | 78 | 79 |
| Agent mix, 12 sessions | 38 | 89 | 97 |
Table 6. Qwen3.8-27B FP8 with MTP on one Spark, vLLM 0.31.0, 512 tokens generated per session.
FlashInfer put single-session speed back on the bandwidth model and roughly doubled throughput with many long sessions: 2.3 times on the agent mix at 12 sessions, where FlashAttention 2 had got slower with every session added past eight. Storing the KV cache in FP8 halves the bytes read per token and adds another 8 to 33% at long context, though it is slightly slower at 2K. What FP8 KV does to output quality I have not measured, and vLLM’s own Spark guide warns about it [9].
Why FlashAttention 2 is slow: one layer, three kernels
To see why FlashAttention 2 is slow, I wrote a split-KV decode kernel in Triton for this exact shape (24 query heads sharing 4 KV heads, head size 256, 800-token pages), which spreads one long sequence across all of the GPU’s cores and merges the pieces at the end. Then I timed one layer’s attention with each kernel on its own, outside vLLM. A plain large read on this chip reaches 242 GB/s of the nominal 273, so that is the ceiling for a kernel whose job is to read the KV cache once.

FlashAttention 2 is slow only on the verify step of speculative decoding. With one new token per sequence it is as fast as anything else (panel b). With three, the real token plus the two drafts that MTP checks on every step, it reads at 95 to 123 GB/s with 16 sequences, about half of what the other kernels manage (panel a). My guess is that its decode-specific optimizations only apply to a single query token; I have not confirmed that in its source. Every step of an MTP server has this shape, which is also why MTP looked so bad at 64 trajectories in section 3.
My kernel came out slightly faster than FlashInfer’s with a BF16 KV cache, 5 to 8% in most cells, and at long context it reads at 240 GB/s, 99% of the ceiling. With an FP8 KV cache the two are tied at about 97%. The setting I would actually serve, FlashInfer with FP8 KV, is already at the memory limit, so a better kernel cannot make it much faster: wiring mine into vLLM would gain about 3% with a BF16 cache and nothing with FP8. Its use here was to locate the cause, not to replace FlashInfer.
MTP, measured again
With FlashInfer I measured MTP against the same server without it once more. It was 1.57 to 1.91 times faster in every cell, from one session to 16 and from 2K to 128K of context, and 1.65 to 1.79 times on the agent mix. One session at 32K runs at 11.7 tokens per second with it and 7.3 without. I have not repeated the 32- and 64-trajectory measurements, but up to 16 sessions MTP is clearly worth keeping.
Four-bit formats
From here the only way to go faster is to read fewer bytes. NVIDIA’s NVFP4 format stores four-bit values with an FP8 scale for every 16 of them, about 0.56 bytes per value against 1 for FP8. vLLM 0.31 supports an NVFP4 KV cache only on datacenter Blackwell GPUs, not on GB10, so I added it to my kernel. It is 1.46 to 1.55 times faster than FlashInfer with an FP8 cache at 16K of context and above; the byte count alone would allow 1.78, and unpacking the four-bit values costs the rest. From the per-layer times, that would make whole decode steps about 9 to 11% faster at long context, once wired into vLLM. NVFP4 weights need no new kernel: NVIDIA publishes an NVFP4 version of Qwen3.8-27B [10] that vLLM serves as is, keeping the MTP layer in BF16. Swapped in for Qwen’s FP8 checkpoint, with FlashInfer, an FP8 KV cache and MTP unchanged, it was 1.1 to 1.7 times faster in every cell. One session ran at 25 tokens per second at 2K and 17 at 128K, against 15.5 and 11.6; the agent mix was 1.23 to 1.42 times faster; 16 sessions at 32K gained least, 116 against 105, because there the KV cache rather than the weights is most of what each step reads. I have not looked at the quality of either format; NVIDIA’s model card reports its own accuracy numbers for the weights.
For RL, this matters most for long agent rollouts like these, which on the default kernel would have generated at about half the speed. The estimates in Table 4 were limited by the trainer and do not change. Their generation rates were measured with FlashAttention 2 on short contexts, so the single-turn row may be somewhat pessimistic. Whether a faster sampler shortens RL steps at all is the subject of section 7.
7. Which half is the bottleneck
With generation and training overlapped, a step takes as long as the slower half. A faster sampler, whether from a lower-precision checkpoint or a better kernel, only shortens a step when generation is the slower half; otherwise the finished rollouts wait in a queue. Generation is the slower half when the share of trajectory tokens the model generated is larger than the generation rate divided by the training rate.
The share depends on the task and on how trajectories are turned into training samples. In the Blender sessions the model’s reasoning stays in the conversation history, so each session can be trained as one sequence, and the model generated 57% of its tokens (22 to 74% per session); 88% of what it generated was reasoning. If earlier reasoning were dropped from the history, each turn would have to be trained as its own sample with the whole history forwarded again, and the share would fall to 7.5%. The synthetic agent shapes in Table 4, with short answers and tool output after every turn, generate 25 to 60%.
Training speed barely depends on sequence length. Forward and backward over a single sequence, with the same LoRA and gradient checkpointing setup as section 3, ran at 222 tokens per second at 4K, 224 at 8K, 222 at 16K and 209 at 32K. Memory is the limit: peak use grew from 56 GB at 4K to 86 GB at 32K, and 40K did not fit under the 97 GB cap I use. 10 of the 23 Blender sessions are longer than 40K tokens; training on those would need the sequence split across both Sparks or the history compacted.
| Generation setting | Tokens/s, agent mix, 12 sessions | Break-even share | Step time at a 57% share, relative to training alone |
|---|---|---|---|
| FP8 weights, FlashAttention 2 | 38 | 18% | 3.10 |
| FP8 weights, FlashInfer | 89 | 42% | 1.35 |
| FP8 weights, FlashInfer, FP8 KV cache | 97 | 46% | 1.23 |
| NVFP4 weights, FlashInfer, FP8 KV cache | 119 | 57% | 1.00 |
Table 7. Generation rates from section 6, all with MTP on one Spark; training 209 tokens/s on one Spark at 32K. Break-even share is generation rate divided by training rate.

For long, reasoning-heavy agents like the Blender sessions, generation is the bottleneck with every FP8 setting, so faster generation shortens steps, and NVFP4 weights bring it level with training; anything faster than that would only make rollouts wait. For agents that write little and read a lot, or whose turns have to be trained separately, training is the bottleneck. At short context FP8 generation runs at 164 tokens per second, a break-even share of 72%, so the Table 4 shapes are training-bound, and FP8 or NVFP4 rollouts buy them no step time while still adding the sampler/trainer gap of section 3. The precision of the rollouts is therefore worth choosing from the generated share of a few real trajectories.
8. Caveats
- Every training run is five steps with a random reward. That measures speed and the sampler/trainer gap, not whether anything learns.
- Run-to-run variation can be large: 17% between two identical baseline runs on different nights, cause unknown. At 64 trajectories, generation drifted 8% between the start and end of a day of runs, while at 32 it stayed within 1%.
- The agent estimates use a synthetic load with forced output lengths. Real agents stop early, call tools of varying cost and produce uneven trajectories.
- Allowing staler rollouts looked like a small win after warm-up, but five steps is too short to say.
- Section 6 uses one Spark and a benchmark with forced output lengths; the agent mix replays real context lengths, not real agents. The kernel comparisons time one layer on its own.
- FP8 and NVFP4 KV caches and NVFP4 weights are measured for speed only, not for output quality.
- Section 7 compares generation at 12 concurrent sessions with training on one sequence at a time. An RL run with more trajectories in flight would generate faster in total, which raises the break-even shares.
- All of this is TRL 1.18 and vLLM 0.31 on GB10; several of the numbers will move with new kernels, as section 6 shows.
9. What’s next
Two to four days per run is workable for one carefully chosen experiment, not for a sweep. The real agent task in sandboxed virtual machines told me how long real trajectories are, and section 6 is what that cost. Still open: whether FP8 rollouts cost learning, and the question all of this was for, whether asynchronous GRPO is worth it on two Sparks or whether running generation and training one after the other is better, which would also let a larger model use both Sparks while generating.
References
- Qwen, Qwen3.8-27B model card.
- Hugging Face TRL, issue #8104: AsyncGRPO tree attention and head dimension 512.
- Stephen Meier, Async GRPO on Two DGX Sparks: Same Curve, Different Bottleneck, October 2026.
- Hugging Face, PEFT, LoRA layer input casting.
- Qwen, Qwen3.8-27B-FP8 model card.
- Hugging Face, TRL v1.18.0,
AsyncGRPOTrainer. - vLLM, v0.31.0.
- Kubesimplify, Qwen3.8-27B on DGX Spark.
- vLLM blog, vLLM on DGX Spark, June 2026.
- NVIDIA, Qwen3.8-27B-NVFP4 model card.