AI Engineering

By Stephen Meier

Reproducing an RL Peak Search with Qwen3.8-27B on Two DGX Sparks

Does a recent paper’s recipe for teaching Qwen3.8-27B to search collider data for new particles (evolve programs, fine-tune on the best, then GRPO) hold up on data its authors never saw? I reran all three stages on two DGX Sparks with my own synthetic data.

Does the recipe in “Reward-Driven Collider Physics Challenges” (Yue, Liu and Gonski, Stanford/SLAC [1]) hold up on data its authors never saw? The paper teaches Qwen3.8-27B to write programs that search a diphoton mass spectrum for narrow peaks, in three stages: evolve programs with the frozen model, fine-tune a LoRA adapter on the best ones, then train that adapter further with GRPO. I reran all three stages on two NVIDIA DGX Sparks with my own synthetic data and my own hidden test sets. The ordering they report held: the base model scored 0.163, the fine-tuned model 0.300 and the fine-tuned model after GRPO 0.567, against the paper’s 0.116, 0.342 and 0.457.

The main findings:

  • The result reproduces: after fine-tuning and GRPO, 29 of 32 single attempts beat the 0.333 a program gets for never claiming a peak, and none of the 384 program runs crashed. A hand-written reference program still scores higher, 0.656.
  • Most of the gain came from finding peaks: on datasets with one true peak the score rose from 0.19 after fine-tuning to 0.64 after GRPO, and on datasets with two peaks from 0.13 to 0.37. In the paper, GRPO mostly improved the datasets with no peak.
  • Fine-tuning on the top programs taught one program: the evolution stage converged on a single behaviour. 296 of the 305 programs with fitness of at least 0.8 made identical claims on every dataset, so following the paper’s rule (fine-tune on the 149 highest-fitness programs) produced a model that wrote the same answer 32 times out of 32. GRPO cannot learn from that, because every attempt in a group gets the same reward. Keeping only the best program for each distinct behaviour (100 programs) fixed it.
  • GRPO straight from the base model learned to claim nothing: it reached 0.278, mostly by writing programs that crash less and report no peaks. The scoring rule pays a guaranteed 0.333 for claiming nothing, while a program that searches loses that on any false alarm and scores zero if it crashes. Starting from programs that already find peaks avoids this.
  • Cost on two Sparks: about 11 hours of evolution on one Spark, 2.2 hours of fine-tuning, and 8.1 hours for the 30 GRPO steps with one Spark generating and the other training.

Section 1 describes the task and the data, section 2 the setup, section 3 the results, section 4 the two failures and their fixes, section 5 the engineering problems with the trainer, and section 6 what this does and does not show.

1. The task and the data

The model is given a description of a dataset of photon pairs and asked for a reusable program, search.py DATASET_DIRECTORY OUTPUT_JSON, that reads the events, finds narrow resonances in the diphoton invariant mass between 118.5 and 132.5 GeV, and writes the masses it believes are real. The program runs in a sandbox with two CPU threads, 60 seconds, no network and a fixed set of libraries (numpy, pandas, scipy, scikit-learn, uproot). Each dataset has zero, one or two true peaks. The score for one dataset is the paper’s soft F1: a claim within about 1 GeV of a true peak earns credit, a claim with no peak nearby costs credit, and a dataset with no peak scores 1 only if the program claims nothing. A program’s score is the mean over datasets, so a program that never claims anything scores exactly 1/3 on a set balanced across zero, one and two peaks.

The paper’s prompts and settings are published, but its code and data are not. I used the system and task prompts from its appendix verbatim, implemented the score as written, and generated the data myself from the numbers the paper gives: about 159,000 background events in the signal region, peak widths of 1.35 to 1.50 GeV, a Higgs-like peak of about 3,100 events in some datasets, other peaks of 380 to 1,370 events, and pairs of peaks 2 GeV apart. The datasets used during training follow one family of smooth background shapes and the hidden test sets another, standing in for the paper’s two separate simulation setups. The prompts for mutation and crossover in the evolution stage are not printed in the paper, so I wrote my own.

DataUsed forSize
Practice setsFitness in the evolution stage6 datasets (2 each with 0, 1 and 2 peaks)
Training bankGRPO reward, one fresh set of three per prompt256 sets of three datasets
Hidden test setsAll results in this post; never used for training or selection12 datasets (4 each with 0, 1 and 2 peaks), different background family

Table 1. Three separate pools of synthetic data. Two sanity checks against the paper: claiming nothing scores 0.333 on the hidden sets (paper 0.333), and my hand-written reference program scores 0.656 (paper 0.689).

2. The setup

The three stages follow the paper’s settings. Evolution: 256 short chains of attempts with the frozen model build a pool, the 32 fittest programs form the first population, and each of 20 generations adds 64 children by mutation or crossover. Fine-tuning: a rank-32 LoRA adapter on every linear layer, learning rate 1e-4, 2 epochs, each program paired with two prompt variants. GRPO: 30 steps of 6 prompts with 8 attempts each, learning rate 1e-5, reward equal to the program’s score on a fresh set of three datasets from the training bank. Sampling everywhere uses Qwen’s defaults with thinking off (temperature 0.7, top-p 0.8, top-k 20).

Spark 1 (generation)Spark 2 (training)Desktop (scoring)
SoftwarevLLM 0.31.0TRL 1.18.0 AsyncGRPOTrainerscoring service, bubblewrap sandbox
ModelQwen3.8-27B, official FP8 checkpoint, LoRA adapter servedQwen3.8-27B in BF16, LoRA rank 32runs each program on three datasets, about 1 s each
Linkadapter shared over NFS on the direct cable between the Sparks after every step; programs sent to the desktop over an SSH tunnel

Table 2. Machines for the GRPO runs. Evolution and all evaluations used Spark 1 only.

Generation uses the FP8 checkpoint because the BF16 model decodes at only 4 to 5 tokens per second per request on the Spark, so a 6,000-token answer takes about 25 minutes; FP8 runs at about 7.7. The cost is a mismatch between the model that samples and the model that trains: their token log-probabilities differ by 0.013 to 0.018 on average from the base model and 0.005 to 0.007 from the fine-tuned adapter. The plan was to retry in BF16 if the reproduction fell short, and it did not. The evaluations in section 3 all use the same FP8 server so the four models are compared on equal terms.

3. Results

Fine-tuning followed by GRPO more than doubled the fine-tuned model’s hidden-set score and beat both the paper’s final number and claiming nothing. Each row is 32 single attempts (8 prompt variants times 4), each run on all 12 hidden datasets.

ModelMeanBest attemptNo peakOne peakTwo peaksAttempts above 0.333Program runs that crashed
Base model0.1630.3410.460.010.022 / 32141 / 384
GRPO from the base model0.2780.3640.760.040.041 / 3220 / 384
Fine-tuned0.3000.5940.580.190.1310 / 3243 / 384
Fine-tuned, then GRPO0.5670.7510.700.640.3729 / 320 / 384
Claim nothing0.333100
Hand-written reference program0.6560.750.770.45
Paper: base / fine-tuned / then GRPO0.116 / 0.342 / 0.457— / — / 0.6830.28 / 0.64 / 0.940.03 / 0.20 / 0.260.04 / 0.19 / 0.18

Table 3. Hidden-set results. “No peak”, “one peak” and “two peaks” are mean scores on the four hidden datasets of each kind.

Grouped bars of hidden-set score for four models (base, GRPO from base, fine-tuned, fine-tuned then GRPO) on datasets with no peak, one peak and two peaks, and overall. The fine-tuned-then-GRPO model scores 0.70, 0.64, 0.37 and 0.57 overall; GRPO from base scores highest on no-peak datasets but near zero on the others.
Figure 1. Hidden-set score by number of true peaks. GRPO from the base model gains only on datasets with no peak; GRPO from the fine-tuned model gains on all three kinds.

The final model did not settle on one answer: its 32 attempts are 32 different programs with 21 distinct scores, between 0.27 and 0.75. Its answers are also shorter than the fine-tuned model’s, 1,850 tokens on average against 2,960. The gain is on the datasets that contain peaks, which is where the paper’s GRPO stage did not move: its final model scored 0.94 on datasets with no peak but 0.26 and 0.18 on those with one and two. My data is easier or harder in ways I cannot measure without theirs, so the comparison of absolute numbers is loose; the ordering of the stages is the result that carries over.

During training the reward on fresh datasets from the training bank rose from about 0.42 to about 0.62 for the run that started from the fine-tuned model, and from 0.15 to 0.26 for the run that started from the base model, which never passed the 0.333 for claiming nothing.

Two lines of GRPO training reward over about 1,600 scored attempts. The run from the fine-tuned model starts near 0.42 and ends near 0.62, above a dashed line at 0.333 for claiming nothing. The run from the base model starts near 0.15 and ends near 0.26, below the line.
Figure 2. Training reward, averaged over the last 48 scored attempts (about one step).

4. Two failures and their fixes

The evolution stage converged on one program

Fine-tuning on the 149 highest-fitness programs, as the paper does, produced a model that answered with the same program every time: 32 of 32 attempts scored exactly 0.611. The evolution run itself went much as the paper describes. The population’s mean hidden-set score rose from 0.387 at generation 0 to 0.594 at generation 20 (paper: 0.382 to 0.509, peaking at 0.626), with the same early dip on datasets with no peak. But the best fitness stopped at 0.811 in generation 3, and the population filled with rewrites of that program: of the 305 programs with fitness of at least 0.8, 296 made identical claims on every practice dataset. The paper’s fitness reached 0.99, which it attributes partly to overfitting, so its top 149 programs probably spanned many different programs; mine were one.

A fine-tuned model that always writes the same program gives GRPO nothing to work with. GRPO compares the attempts within a group of eight, and if all eight score the same there is no signal. So I deduplicated by behaviour: two programs count as the same if they make the same claims on all six practice datasets, and I kept the best program of each behaviour with fitness above 0.5. That left 100 programs (fitness 0.51 to 0.81) from 330 distinct behaviours in the whole pool, each paired with two prompt variants as before. This is a deviation from the paper’s rule. The resulting model scored 0.300, close to the paper’s 0.342, with 18 distinct scores across its 32 attempts.

GRPO from the base model learned to claim nothing

The paper trains GRPO only from the fine-tuned adapter. I also ran it from the base model with the same settings, to see how far reinforcement learning alone gets. It raised the hidden-set score from 0.163 to 0.278, almost entirely in two ways: programs that run (crashes fell from 37% to 5% of program runs) and programs that report nothing (79% of the runs that did not crash claimed no peak). Scores on datasets with peaks stayed near 0.04.

This is not a class imbalance: the training bank and the hidden sets have equal numbers of datasets with zero, one and two peaks, and the peaks in it can be found: the reference program scores 1.00, 0.67 and 0.39 on 64 of its sets with zero, one and two peaks. It comes from the scoring rule. A program that claims nothing collects the full score on every dataset with no peak, a guaranteed third. A program that searches loses that third on any false alarm and scores zero everywhere if it crashes. From a base model whose searching programs mostly fail, the cautious programs win within each group, and GRPO reinforces them. Fine-tuning first puts working search programs into the groups, which is presumably why the paper does it.

5. Problems with the trainer

Six things in TRL 1.18’s asynchronous GRPO trainer had to be fixed or worked around before a run trained as intended. Together they cost most of a day.

  • Request timeout: the trainer gives up on a generation request after 1,800 seconds by default and retries it from scratch. At BF16 speed an 8,192-token answer takes longer than that, and the trainer starts no second group until the first is scored, so the first attempt ran for almost three hours without scoring anything. Setting vllm_server_timeout to 7,200 seconds, capping answers at 6,144 tokens and switching generation to FP8 fixed it.
  • Steps are sized in tokens: a step trains on gradient_accumulation_steps times token_budget tokens, not on a number of samples. With 6 accumulation steps the first step trained on 27 samples instead of the paper’s 48; 11 gave about 48.
  • Tool calls without tools: the trainer parses text that looks like a tool call even when no tools are configured, answers it with an error message and continues the conversation. One test answer ran 8 turns and 8,360 tokens past a 4,096-token cap. Setting max_tool_calling_iterations=0 and scoring only the first answer fixed it.
  • Top-k and top-p sampling: the trainer needs vLLM started with --return-sampling-mask --enable-scale-out to correct log-probabilities for the truncated sampling.
  • Adapter names: each run names its adapters trl-policy-v1, v2 and so on from 1, so adapters left loaded on the vLLM server from the previous run must be unloaded first.
  • Starting from an adapter: the trainer refuses a model that already has a PEFT adapter attached, because its patch for the Gated DeltaNet layers looks up the model’s code from the wrong class. Letting the trainer create a fresh adapter of the same shape and then loading the fine-tuned weights into it works.

The reward function also runs in a separate process that the trainer spawns, so anything set up in the script’s main() is missing there; I passed the output directory through an environment variable.

6. What this does and does not show

The paper’s central claim holds on independently generated data: evolution, then fine-tuning on evolved programs, then GRPO, takes Qwen3.8-27B from well below the claim-nothing score to well above it. The limits of this reproduction:

  • Each stage ran once. GRPO runs of this size vary, and I have no estimate of how much.
  • The fine-tuning set was deduplicated by behaviour, which the paper does not do. Without it, the reproduction would have stopped at the fine-tuning stage.
  • The data is mine, generated from the paper’s description. Absolute scores are not directly comparable with the paper’s, and the difference in where GRPO helped (datasets with peaks here, datasets without in the paper) may come from the data.
  • Generation used the FP8 checkpoint while training used BF16.
  • The hand-written reference program (0.656) still beats the best trained model’s average, though not its best attempt (0.751).

References

  1. Alexander Yue, Qibin Liu, Julia Gonski. Reward-Driven Collider Physics Challenges for Self-Improving Agentic AI via Reinforcement Learning. arXiv 2611.15885, November 2026.
  2. Qwen. Qwen3.8-27B model card.
  3. Hugging Face TRL. TRL documentation (AsyncGRPOTrainer).
  4. My earlier post: Async GRPO on two DGX Sparks.