I wanted to know what each of the standard ways of improving an agent is actually worth. Not in the abstract, which is unanswerable, but in one setting controlled tightly enough to compare them directly: the same 8B model, the same agent loop, the same 240 held-out tasks, the same scoring throughout. Start with the base model, add a carefully written skills prompt, then finetune on demonstrations, then apply reinforcement learning, and measure after every step.
The setting is TextWorld’s cooking games, and the base model fails in ways that are easy to picture. The recipe calls for a sliced carrot, so it chops the carrot, and the game ends. It finds a potato that has already been roasted and roasts it again, which also ends the game. It walks between two rooms for fifty turns looking for a kitchen it has already stood in. Out of 240 games it had never seen, it won 34, killed itself in 90, and exhausted its move budget without scoring a single point in another 62.
By the end the same agent wins 171 of those 240. All three interventions helped, by very different amounts, and the pattern in which one fixed which failure is the part I think carries over to other agent work. It is not the pattern I expected going in.
1. The task
TextWorld [1] is a framework from Microsoft Research that procedurally generates text adventures. A game looks like this:
-= Kitchen =-
You arrive in a kitchen. You make out a closed fridge. You see a
counter, and on it a cookbook and a knife.
> examine cookbook
Recipe #1
Ingredients: red hot pepper
Directions: chop the red hot pepper, roast the red hot pepper,
prepare meal
> take red hot pepper
You take the red hot pepper from the counter.
Your score has just gone up by one point.Code language: PHP (php)The agent has to find the cookbook, gather each ingredient from around a house, process every ingredient exactly as the recipe directs, then prepare and eat the meal. Chopping something the recipe wanted sliced ends the game. So does cooking something that is already cooked. There is no partial credit for a near miss on those.
Three properties make this a good measurement target rather than a toy. The environment scores itself, awarding points per subgoal and declaring wins and losses, so no LLM judge and no human labelling sits in the loop. Difficulty is parameterised, so the benchmark can be a ladder instead of a flat set. And the ingredient pool splits into train, validation, and test partitions, so a finetuned model can be tested on recipes it has genuinely never seen.
I built four tiers of 60 games each, 240 per split, with disjoint seeds so no game appears twice.
| Tier | Difficulty | What it stresses |
|---|---|---|
| tier1 | 1 ingredient, 1 room | Basic procedure |
| tier2 | 2 ingredients, 1 room, cutting and cooking | Multi-step processing |
| tier3 | 3 ingredients, 6 rooms, cutting and cooking | Processing plus navigation |
| tier4 | 3 ingredients, 9 rooms, plus inventory limits | Navigation under constraint |
Table 1. The four benchmark tiers. Scores throughout are the normalised game score, meaning points earned divided by points available, averaged across games. A win means the agent ate a correctly prepared meal.
The agent itself is an ordinary ReAct loop built on deepagents [2] and LangGraph, with tools mirroring the game’s own verb list. The policy model is ibm-granite/granite-4.1-8b [3]. Everything ran on a single NVIDIA DGX Spark.
2. How the untouched agent fails
The baseline scores 0.309 and wins 34 of 240. Reading the transcripts, its losses sort into two piles.
The larger pile is state-tracking failure. An ingredient’s name encodes its history: “chopped roasted red potato” is a potato that has already been chopped and roasted. The recipe directs chopping and roasting. The agent reads the name, reads the direction, and roasts it again. The game ends. This accounts for most of the 90 deaths, and it is not a knowledge problem, since the relevant fact is sitting in the context window in plain English.
The smaller pile is search failure. On the larger maps the agent revisits rooms it has already searched, or asks for an object that is not present and receives a disambiguation prompt (“Which do you mean, the red apple or the red onion?”), then loops on it until the move budget runs out. That is most of the 62 zero-score stalls.
Both are behavioural rather than informational. The agent is not missing facts about cooking. It is failing to act on facts it already has, which is what made the first intervention seem obvious and, in the end, mostly ineffective.
3. A better prompt
The obvious fix is to tell the model what it is doing wrong. I wrote a Claude-style SKILL.md playbook and served it through deepagents’ skills middleware, which advertises skill files to the model and lets it decide when to read them.
It scored below baseline. The reason turned out to be mechanical rather than conceptual: the model called read_file on the skill in 0 of 32 episodes. The body of the playbook never entered the context window even once, so the measurement captured a skill that was never read plus the token cost of advertising it. Progressive disclosure assumes the model recognises when a reference would help. An 8B model that has not been trained for that pattern does not.
The fix was to stop relying on the model’s judgement about when to look things up. The load-bearing rules moved into the 1024-character skill description, which is always in context, and the system prompt now instructs the agent to read the file before its first move. After five versions this reached 0.407, up from 0.309, and lifted wins from 34 to 59.
Then it stopped improving, and how it stopped is the useful part.
Version 4 attacked the state-tracking pile directly. It mandated an inventory check before every cook or cut, so the agent would see an ingredient’s current name before acting on it. The model complied completely: inventory calls tripled. Re-cook deaths went from 23 to 25, which is to say they did not move. The agent would dutifully check its inventory, read “chopped roasted red potato,” and roast it.
Version 4b attacked the search pile by supplying the parser fact behind the disambiguation loops. The targeted loops shrank by roughly a quarter. The score did not move at all.
My reading is that instructions install actions readily and inferences poorly. “Check your inventory” is an action, and the model performed it on command, every time. “Notice that this ingredient’s name already contains the word roasted, and therefore conclude that roasting is done” is an inference, and no phrasing I tried produced it. Version 4b makes the same point from the other side: giving the model the missing fact reduced the surface behaviour without changing the outcome, because knowing why a loop happens is not the same as being able to break it.
That distinction is what sent me to the weights.
4. Supervised finetuning on demonstrations
If instructions cannot install an inference, perhaps examples can. The question is where examples come from.
TextWorld hands you each game’s walkthrough, an oracle command sequence that wins. Training on walkthroughs alone has one disqualifying gap: a walkthrough never triggers a disambiguation prompt. The oracle walks straight to the correct object and never hears “Which do you mean.” So the free, perfect data demonstrates the clean path and leaves the single most common failure entirely undemonstrated.
The alternative is the agent’s own successful runs. A game the policy won at temperature is a game where it actually hit an ambiguity, recovered, hunted through containers, and finished. That is the behaviour worth amplifying, and it arrives in the model’s own distribution rather than an oracle’s.
So I split the data by what each source can actually show. For tiers where the policy sometimes wins, I sampled its own trajectories, eight per game across 60 training games, and kept the wins. For tier4, where the policy had won exactly zero games in every condition ever measured and rejection sampling therefore yields nothing at all, I used walkthroughs. The final set was 250 examples: 190 sampled from 1,920 episodes, plus 60 tier4 walkthroughs. The skills prompt was used to generate the demonstrations and then removed from the training context, so the behaviour would live in the weights rather than the prompt.
Training on that set is ordinary supervised finetuning: 250 examples, three epochs, loss computed on assistant tokens only. The environment’s text is context for the model to condition on rather than something it should learn to produce, and including it would teach the model to imitate the game engine alongside the player.
That last detail cost me a full training run. TRL’s assistant_only_loss option locates assistant spans using {% generation %} markers in the tokenizer’s chat template, and Granite’s stock template contains none. Rather than raising an error, the option silently produced an all-zero loss mask. The run completed, nothing complained, and the weights were unchanged. Patching the template fixed it, and verifying the fix meant confirming that rendered conversations stayed byte-identical while the supervised fraction rose from nothing to between 5.8% and 15% of tokens.
The result, evaluated with no prompt at all: 0.712, and 140 wins.
The failure piles are what convinced me this was real rather than a mean shifting for boring reasons. Deaths fell from 90 to 28. Zero-score stalls fell from 62 to 17. Both happened while the agent played four times as many games through to completion, which normally creates more opportunities to die.
Set that beside version 4 of the prompt, where an explicit order to check inventory tripled inventory calls and left re-cook deaths at 23 against 25. The demonstrations installed the inference that the instruction could not. That is the clearest result in this project and the one I would generalise first.
Tier4 is the sharpest illustration. It sat at exactly 0.000 win rate under every prompt condition ever measured, in both decoding regimes. After finetuning it reached 0.508. Every one of those points traces to the 60 oracle walkthroughs, which is to say to the data source that covers precisely what the agent could never demonstrate for itself.
5. Reinforcement learning
By this point the question had changed. It was no longer whether RL can rescue a weak policy but whether it can improve a competent one.
Finetuning first made RL affordable, in a specific sense worth spelling out. GRPO [4] learns only from reward variance within a group of rollouts on the same problem. If all eight rollouts score identically, that group contributes nothing to the gradient. At the baseline’s 14% win rate most groups came back all-zero, so most of the compute bought nothing. Starting from the finetuned policy, 58 of 60 groups had usable variance, and the two that did not were degenerate from the ceiling: all eight rollouts scored a perfect 1.0 on a game the policy had completely solved.
Reward is TextWorld’s own normalised score, unshaped. Rollouts run through the same agent graph used for evaluation, via TRL’s GRPO trainer [5]. Sixty steps over 20 training games took 59 hours 48 minutes on one machine.
The result: 0.796 and 171 wins, with no prompt. Deaths fell further, from 28 to 23, and stalls from 17 to 14.
The gain concentrates almost entirely in tier3, the tier that combines multi-ingredient processing with a six-room map. Tier1 was already near its ceiling, and tier2 and tier4 moved within noise. I had predicted the opposite spread and was wrong for a mundane reason worth recording: I reasoned from training games where the hardest ones regressed, but the 20 training games are not tier-stratified the way the benchmark is, so “hard training game” was never a proxy for “high tier.”
6. What each rung was worth
| Condition | Score [95% CI] | Wins | Deaths | Stalls |
|---|---|---|---|---|
| Baseline | 0.309 [0.269, 0.352] | 34 | 90 | 62 |
| Best prompt | 0.407 [0.360, 0.456] | 59 | 55 | 58 |
| Demonstrations (SFT) | 0.712 [0.665, 0.758] | 140 | 28 | 17 |
| Reinforcement learning | 0.796 [0.752, 0.838] | 171 | 23 | 14 |
Table 2. All four conditions on the same 240 held-out games, greedy decoding, no prompt for the two trained models. A death is a fatal processing error; a stall is an episode that spent its whole move budget without scoring.
| Condition | tier1 | tier2 | tier3 | tier4 |
|---|---|---|---|---|
| Baseline | 0.494 | 0.362 | 0.247 | 0.133 |
| Best prompt | 0.556 | 0.592 | 0.324 | 0.156 |
| Demonstrations | 0.761 | 0.896 | 0.685 | 0.508 |
| Reinforcement learning | 0.783 | 0.956 | 0.877 | 0.565 |
Table 3. The same runs by tier. Tier4 remains the weakest at 0.565, and its failure mode is systematic exploration of a nine-room map rather than anything about recipes.
Comparing each rung against the one below it, on a per-game paired basis (§11), the prompt was worth +0.098, demonstrations +0.305 on top of that, and reinforcement learning a further +0.083. Every one of those intervals excludes zero.
The ordering is the practical finding. Prompt engineering consumed the most calendar time of the three, across five versions and a good deal of failure analysis, and returned the smallest gain. Demonstrations cost 250 training examples and a few hours, and returned roughly four times as much. Reinforcement learning cost 60 hours of compute and returned a real but modest amount on top. If I were starting a similar agent project tomorrow I would build the demonstration pipeline first and treat prompting as a way to generate those demonstrations rather than as the deliverable.
7. The prompt stopped being worth anything
Since the trained models were evaluated with no prompt, an obvious question follows: would they do better with it? I ran the identical playbook against both ends of the ladder.
| Policy | No prompt | With the playbook | Paired difference |
|---|---|---|---|
| Baseline | 0.309 | 0.407 | +0.098 [+0.051, +0.144] |
| Trained (RL) | 0.796 | 0.813 | +0.017 [−0.011, +0.045] |
Table 4. The same file, applied to the weakest and strongest policies. On the trained model, 201 of 240 games play out identically with and without it.
The playbook that was worth +0.098 on the base model is worth nothing measurable on the trained one, and its interval now spans zero. The scaffold was a stepping stone: useful for generating the demonstrations that made it obsolete, and dead weight in the context window afterwards. Anyone maintaining a long prompt alongside a finetuned model should probably measure whether it is still doing anything.
One exception is worth flagging as a hypothesis rather than a result. On tier4 the playbook is still worth +0.070 [+0.005, +0.136]. The mechanism is plausible, since tier4 is navigation-bound, the playbook has an explicit navigation section, and tier4 is where the demonstration data was thinnest. But that is one positive result across four tiers with no correction for multiple comparisons, and its lower bound sits at +0.005. I would want a dedicated experiment before believing it.
8. Reinforcement learning sharpened rather than broadened
| Policy | Greedy | Sampled, 3 per game | Gain from sampling |
|---|---|---|---|
| Demonstrations (SFT) | 0.712 | 0.731 | +0.019 |
| Reinforcement learning | 0.796 | 0.799 | +0.003 |
Table 5. Temperature sampling at 0.7 with three samples per game, against single greedy decoding.
Sampling three times buys the finetuned policy a small amount and the RL policy almost nothing. In my assessment that is outcome reward doing what it is supposed to do, concentrating probability mass on trajectories that work and leaving less to gain from exploring alternatives at inference. Error counts agree, falling from 30 to 17 across the sampled runs.
The practical consequence is a cost result rather than a quality one: the RL policy reaches the same score at roughly a third of the inference spend, because greedy decoding is as good as sampling three times and averaging.
9. Two-thirds of the RL run bought nothing
| Checkpoint | Dev score | Wins |
|---|---|---|
| Before RL | 0.741 | 150 |
| Step 4 | 0.736 | 150 |
| Step 20 | 0.803 | 175 |
| Step 40 | 0.831 | 184 |
| Step 60 | 0.821 | 179 |
Table 6. Validation scores across the run. Step 40 was selected on this split and then evaluated once on test.
Learning plateaus by step 40. The paired difference between step 40 and step 60 is +0.010 [−0.015, +0.035], and 219 of the 240 games play out identically between them. Roughly 20 of the 60 hours were productive; the remaining 40 changed almost nothing.
I could only see that because I copied checkpoints out from under the trainer’s own rotation. TRL keeps the last few by default, which here would have been steps 52, 56, and 60, all of them inside the plateau. The run would have looked like steady improvement to the finish. A small background script hardlinked selected checkpoints into a separate directory as each save completed, using hardlinks rather than copies because writing 33 GB creates that much dirty page cache, and page-cache pressure immediately after a checkpoint save had already killed two earlier training runs on this machine.
The step-4 row earns its place too. Four steps of RL produce a clean null: 202 of 240 games identical, win count unchanged. That is a useful control, since it shows the evaluation pipeline does not manufacture differences out of serving noise.
10. Things I would do differently
- Reach for demonstrations earlier. Every prompt version that mandated an action got compliance. Every one that required the model to draw a conclusion from state already in its context failed, regardless of phrasing. If a failure looks like the model not noticing something rather than not knowing something, prompting is unlikely to fix it.
- Pick the demonstration source per failure mode, not per convenience. Oracle walkthroughs are free and perfect and structurally cannot demonstrate error recovery, because the oracle never makes errors. Self-sampled wins cover recovery but yield nothing where the policy never wins. Each source covered exactly what the other could not.
- Keep a spread of checkpoints, not the last few. Default rotation preserves the checkpoints that tell you least once learning has flattened. Four preserved checkpoints cost 130 GB and revealed that two-thirds of a 60-hour run was wasted.
- Re-measure the prompt after training. A prompt tuned against a weak policy can become dead weight against a strong one, and nothing will tell you unless you check.
The remaining headroom is legible enough to name. Tier4 sits at 0.565 against tier2’s 0.956, and its failures are about systematically exploring a nine-room map rather than anything to do with recipes. Twenty-three deaths and fourteen stalls survive out of 240 games. Whether those are reachable by more reinforcement learning or need a different kind of demonstration is the next thing I would measure.
11. How this was measured
Two conventions apply to every number above, and they are the reason I trust the ordering in §6.
Every score carries a bootstrap 95% confidence interval from 10,000 percentile resamples. Where two intervals overlap I describe the conditions as indistinguishable rather than reporting the point estimates as a difference.
Comparisons are paired per game, not marginal. Game difficulty varies enormously even within a tier, and that variance swamps the effects being measured. Since every condition plays the identical game set under greedy decoding, the right statistic is the per-game score difference with its own interval. Two overlapping marginal intervals are not evidence of no effect; usually they are difficulty variance that pairing removes.
The cost of skipping that is concrete. Early in the prompt phase, one comparison read +0.19 at 10 games, +0.23 at 32, then −0.006 at 80, before settling at +0.098 at 240. Small samples do not merely widen the interval, they break selection: at three different points I would have shipped a different prompt version, each time on a number that later reversed.
Model selection ran on a separate validation split of 240 games, and the test set was used once, at the end, for the single selected checkpoint. Choosing the best of several near-tied checkpoints on validation inflates that estimate, so the test result landing slightly below the validation result (+0.083 against +0.090) is expected rather than a failure to replicate.
Two measurement mistakes are worth passing on. First, mid-run confidence intervals on the training games cleared zero twice and collapsed twice, because the game loader cycles in fixed order and front-loads the games RL helps, making any partial cycle a biased sample. Only the complete cycle was interpretable. Second, while trying to speed up rollouts, I found a genuine 1.49x improvement from reusing the KV cache across turns and rejected it: it shifted per-token logprobs by 0.11 to 0.56 nats, and GRPO’s importance ratio is an exponential of exactly that difference, so it would have distorted every ratio by 12% to 75% while every downstream signal continued to look healthy. Two successive correctness checks I wrote passed it spuriously before a third caught it, one because it compared only five tokens and one because it derived its own pass threshold from the same broken computation it was checking.
References
- TextWorld, Microsoft Research. Also: TextWorld: A Learning Environment for Text-based Games.
- deepagents, LangChain.
- ibm-granite/granite-4.1-8b, IBM.
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models, which introduces GRPO.
- TRL, Hugging Face.