The one-paragraph version
This paper audits a popular idea in agent training: teach a language model to predict what will happen next after its action, and it should act better. The authors ask whether the task improvement really comes from learning correct world knowledge, or just from side effects of the extra training itself. To test this, they keep the training procedure identical but corrupt the learning signal — either by swapping the true next observation for a real but wrong observation from another episode, or by replacing the reward entirely with a random coin flip. Surprisingly, agents trained on wrong predictions or even random rewards still get much better at solving tasks than untrained models, even though their ability to predict correctly collapses. The benefit seems to come less from accurate world knowledge and more from changed behavior: the trained agents consider more alternative actions, reference what they have seen more often, and get stuck in repetitive loops far less. The authors show the same pattern for text household and science tasks and for visual web tasks, where random-reward training improves coverage without needing a separate vision model to create text targets.
The problem they're solving, and why it matters
Agents that operate tools, browsers, or robots need to answer more than “what word comes next?” They need to answer “if I do this, what will the world look like next?” Standard pretraining on next-word prediction does not directly teach this action-to-outcome link.
Recent post-training methods therefore add an explicit world-model objective. Here world-model post-training means extra training where the model must first reason and then predict the next observation, and gets rewarded or supervised based on that prediction. Variants include reward-based methods that compare predicted and actual next states in embedding space, auxiliary losses that predict states or transitions alongside policy learning, and direct supervision or self-distillation on collected interaction data.
All of these papers share an expectation: better prediction causes better acting, and they report higher task success after training as evidence. The problem this paper points out is that before-versus-after comparisons confound two things: learning correct environment content, and simply doing more optimization, sampling, regularization, and reasoning updates. If training itself makes agents explore more or loop less, task scores could rise even without truer world knowledge. Untangling this matters because building accurate predictors, especially for images or video, is expensive — it often requires an extra vision-to-text teacher — and we should not pay that cost if the gains come from elsewhere.
The key idea, in plain words
If accurate prediction is truly what drives better actions, then breaking the correctness of the prediction target should break both prediction accuracy and task success together.
The authors create controls that preserve everything about training — same prompts, same base model, same optimizer, same number of steps — and change only the informativeness of the target:
A concrete example helps. Suppose the task is “heat an apple in the microwave.” The agent sees an open microwave with an egg inside, holds the apple, and tries heating. The true next observation is “You heat the apple.” A ground-truth trainer rewards predictions close to that. A mismatched trainer instead rewards predictions close to a different real observation, for example “You open drawer 3. It is empty,” taken from another training episode. The task of predicting remains, and the scoring method remains, but the content learned is wrong.
They also test an even stronger control where the reward carries no environment information at all: a coin flip that gives 1 or 0 at random, independent of what the agent predicted or what happened. If coverage still expands under random rewards, then expansion cannot require learning world content.
Alongside task success, they always measure prediction accuracy separately using held-out transitions and an independent language-model judge, so they can see whether prediction and acting move together or split apart.
How it actually works, step by step
The study focuses on two representative implementations: a reinforcement-learning method based on a policy optimizer called Group Relative Policy Optimization, and a distillation method called on-policy self-distillation from a project named COMAP.
First, define the conditions. Base, meaning no post-training, is the untrained reference. Ground-truth, meaning trained with the true next observation as target, is the published world-model method. Mismatched, meaning trained with an in-distribution but wrong observation as target, is the content placebo. Coin-flip, meaning trained with independent random binary rewards, is the zero-information control. In distillation, only ground-truth versus mismatched teacher targets are compared, holding the rest of the pipeline fixed.
Second, define reward construction. For ground-truth and mismatched in the reinforcement-learning case, predicted and target observations are encoded with an embedding model and compared by cosine similarity, which in plain words is a measure of meaning overlap. Similarity above 0.8 gives reward 1, otherwise 0. Mismatched permutes targets across samples so each input gets someone else’s true outcome. Coin-flip skips targets entirely and samples reward from a fair coin. All conditions normalize advantages within each sampled group and keep a penalty that keeps the policy near its starting point.
Third, define environments and evaluation regimes. The main tests use ALFWorld, meaning text household tasks like pick, clean, heat, cool, look, and pick-two, and ScienceWorld, meaning text science experiments. They use a 7-billion-parameter instruction-tuned language model as the base. In the list regime the prompt supplies the allowed actions; in the no-list regime the model must generate a correctly formatted action from scratch, which is harder and more free-form. VisualWebArena, meaning realistic visual web tasks on classifieds, shopping, and forum sites, uses a vision-language base model and conditions on screenshots.
Fourth, define metrics in words before numbers. Prediction accuracy means the share of held-out cases where an independent judge says the predicted effect of the action matches the true effect, ignoring wording differences and not penalizing unknowable details like exactly what is inside a newly opened container. Single-attempt success, called pass at one, means how often one sampled trajectory solves the task. Coverage, called pass at sixty-four, means what fraction of tasks are solved at least once if given sixty-four independent tries. Looping ratio means what fraction of steps are spent repeating an observation-action cycle. They also label reasoning traces for whether the model mentions previously seen objects, revises a proposed action, or compares multiple candidates.
What they showed — results, honestly, with limitations
Prediction and task success do not move together. On ALFWorld with the list of allowed actions, swapping true for mismatched targets lowers prediction accuracy by about 30 percentage points, from 57.0% to 27.3%, but single-attempt success drops only from 37.3% to 34.6% and coverage barely changes, from 83.9% to 83.2%. Across ScienceWorld and distillation, the same split appears: prediction accuracy falls by roughly 15 to 62 points depending on setting, while absolute differences in single-attempt and coverage success stay small, roughly 0.5 to 2.8 points and at most 3.0 points. In other words, learned content changes a lot, task coverage changes little.
Random rewards also expand coverage. In ALFWorld with allowed actions, coin-flip training raises coverage from 56.2% for base to 87.2%, reaching the level of ground-truth and mismatched training, even though its reward contains no environment information. The curves show ground-truth mainly helps single-attempt reliability, while the gap narrows as more attempts are allowed.
More accurate prediction does not guarantee executable free-form actions. Without an allowed-action list, ground-truth actually solves the fewest tasks among the three reinforcement-learning conditions in both text environments. On ALFWorld no-list, coverage is 52.9% for ground-truth versus 70.8% for mismatched and 74.1% for coin-flip, and ground-truth has the highest rate of actions that fail to parse or execute, 65.3% versus 49.6% for mismatched.
Behavior changes without requiring correct content. At shared decision points, ground-truth favors the true next observation over a wrong one much more strongly than mismatched does, by about 9 log-probability units, so the content intervention worked. Yet after full step-by-step reasoning, the final action distributions of ground-truth, mismatched, and coin-flip become more similar to each other and jointly diverge from base. All trained variants mention observed entities more often, revise actions more often, compare candidates more often, use slightly longer reasoning, and loop far less. For example, list-regime looping falls from 24.7% for base to 1.9% for coin-flip.
The same simplification transfers to vision. On VisualWebArena, keeping the policy training but replacing observation-matching with coin-flip rewards raises overall coverage from 34.8% to 39.8%, about a 14% relative gain, with lower looping, fewer turns, and more autonomous stops. The paper is unclear on exact compute-matched cost and reports largely single runs with bootstrap intervals rather than multiple training seeds, so seed variance is not fully shown. The prediction judge and reasoning labels rely on another language model, and the authors’ own check finds only 40 of 60 action-revision labels confirmed on re-review, so those behavioral numbers are noisy. The mechanism is also unsupported beyond correlation: we know search broadens and looping shrinks, but not precisely why random or mismatched optimization does that.
Why this might matter to me
For spatial reasoning in language and vision models, text-to-CAD and parametric geometry, and 3D generation and scene representation, the caution is direct: higher task completion after prediction training is not proof of better spatial or physical understanding.
If you train a CAD agent to predict the next sketch, extrusion, or constraint state and then see more models solve, this paper suggests the gain could come from broader candidate search and less fixation — trying a different plane, reordering operations, avoiding repeated invalid commands — rather than from a truer internal geometry engine. That distinction matters enormously for parametric CAD, where a nearly right action can still produce non-manifold, under-constrained, or unmanufacturable geometry. Coverage, meaning solved at least once in many tries, can hide poor single-shot precision.
For world models and 3D scene representations, the result pushes toward separate evaluation: measure next-state prediction fidelity, executability, geometric validity, and diversity of attempts independently, rather than inferring one from the other. It also suggests a practical shortcut the authors demonstrate on the web domain: when screenshots, renderings, or view-dependent states are hard to turn into clean text targets, lightweight training without matched targets may still improve exploration and reduce loops. That could be useful for bootstrapping a CAD or scene agent before investing in an expensive teacher that captions images or diffs boundary representations. Conversely, if your goal is true forward simulation — predicting occlusion, contact, support, or exact parameter updates — you will likely need stronger grounding than task reward alone, because this audit shows task reward alone can improve without it.
Terms worth knowing
World-model post-training: extra training after pretraining where the agent must predict how the environment will change after its action, in addition to or as part of learning to act.
Next-observation prediction: given task, history, current observation, and action just taken, generating the resulting observation. Example: after “open microwave,” predicting “microwave is open with an egg inside.”
Ground-truth, mismatched, and coin-flip conditions: ground-truth trains toward the actual outcome; mismatched trains toward a real outcome from a different episode, so the target is plausible but wrong for this input; coin-flip gives random 1 or 0 rewards independent of prediction and environment.
GRPO: Group Relative Policy Optimization, a reinforcement-learning optimizer used here that samples several responses per prompt, scores them, and updates toward relatively better ones within each group while staying near the starting policy.
Self-distillation with a privileged teacher: a student learns to mimic the token distribution of a teacher that additionally sees the true or mismatched next state. Privileged means the teacher sees information the student will not have at test time.
Pass at one versus pass at sixty-four: single-attempt reliability versus coverage given many tries. High coverage with low single-attempt success means the model can sometimes solve it, but not reliably.
List versus no-list regime: whether the prompt supplies the allowed actions to choose from, versus requiring the model to invent a valid formatted action on its own.
Loop ratio: share of interaction steps spent repeating a cycle of same observation and action, a measure of getting stuck.
ALFWorld, ScienceWorld, VisualWebArena: benchmarks used here — text household tasks, text science-experiment tasks, and screenshot-based web navigation tasks, respectively.