All summaries · 2026-09-03

Towards a Belief-Based World Model for LLM Agents

Shubham Kumar, Harshit Kumar, Narendra Ahuja, et al.

arXiv:2609.00455 · abstract · pdf · summarized by meta/muse-spark-1.3-contributor on 2026-09-03

The one-paragraph version

This paper asks whether large language model agents — language models used to choose actions step by step — should get more than just future simulation from their world models. A world model here means a separate component that tracks how an environment works and predicts what will happen. Today that component mostly answers: if you do this action, what happens next? The authors argue that is not enough when the current situation is uncertain, because the agent also needs to know what is known and unknown right now. They propose Belief-Based World Models, which keep an explicit belief — meaning a representation of uncertainty over the current hidden state, such as where an object might be — and let the language model ask about it in plain English, in addition to simulating actions. To test whether that idea helps at all, they hand-build simple, correct beliefs for two text household-science games and show that giving the agent access to those beliefs improves success and efficiency, especially for smaller models, and combines well with simulation.

The problem they're solving, and why it matters

The setting is decision-making under partial observability. Partial observability means the agent does not see the full true state of the world; some important facts are hidden until it looks. An example is a kitchen robot that does not know which cabinet holds a mug until it opens cabinets.

Large language models are increasingly used as policies. A policy means the decision-maker that picks the next action given what has been seen so far. They are good at reasoning, but the paper notes they struggle to keep track of uncertainty over long tasks with hidden information.

World models are meant to help, both in training and at test time. At test time means while the agent is actually solving a new task, not during earlier learning. The current test-time use is a simulation interface. A simulation interface means the agent proposes a candidate action, and the world model predicts the future that would follow.

The authors say simulation fits one kind of action well but not another. They borrow a distinction between pragmatic actions and epistemic actions. A pragmatic action moves you toward the goal, like picking up a mug to drink. Simulation helps there because you want to know consequences. An epistemic action gathers information to reduce uncertainty about the present, like checking a blind spot before changing lanes, or opening a fridge to see if the egg is inside.

The problem: variation across simulated futures mixes together two different things — uncertainty about the present and randomness about the future — and it can take many simulations to recover what is actually uncertain now. The driving example in the paper makes this concrete: a simulator may sample a future where the blind spot is empty and say lane-change is safe, when in reality the model simply did not know what was in the blind spot. What the driver really needed was a direct answer: the blind spot is 90% clear and 10% occupied, so first check.

The key idea, in plain words

Keep a running picture of what might be true right now, update it as new observations arrive, carry it forward when actions are taken, and let the language model ask about it directly in words.

In other words, split the job. Let the world model do the bookkeeping of memory and uncertainty, so the language model can focus on reasoning and choosing actions. Give the agent two complementary tools: ask about the present when you need to decide whether to go gather information, and simulate the future when you need to decide whether a goal-directed move will work.

A concrete example from the paper: instead of wandering through every cabinet, the agent can issue a query like “where is the egg?” and get back a ranked list of still-unsearched places that could contain it. After it searches the fridge and finds no egg, it can ask again and get an updated list that excludes the fridge. That list reflects the model’s prior knowledge about where that game ever puts eggs, plus what has been ruled out so far.

How it actually works, step by step

The authors deliberately do not try to learn a general belief model yet. Learning here would mean training a system to estimate uncertainty automatically. They first ask the more basic question: if the belief were correct, would exposing it even help? So they hand-specify the belief in simple text environments where the hidden part is almost entirely object location.

They use two benchmarks. ALFWorld is a text household game where the agent must find objects among receptacles like cabinets and drawers, then clean, heat, cool, or move them. ScienceWorld is a text elementary-science game in a fixed ten-room house, with tasks like boiling a substance or growing a plant. In both, key objects start in random allowed places.

Each hand-built model has two parts to its internal state. The deterministic part, which the paper also calls memory, tracks facts already seen: where the agent is, what it holds, which receptacles or rooms have been searched, and what was seen inside. The probabilistic part, the belief proper, is a distribution over possible locations for the task-relevant object. A distribution here just means a list of possibilities with weights that sum to one. It starts as a uniform prior, meaning equal weight on each allowed location given by the game engine — for example, all receptacle types that the ALFWorld engine ever uses for eggs. The update rule is simple presence-or-absence renormalization: renormalization means re-dividing the remaining weight after ruling something out. If the agent finds the object, the belief collapses to that place. If it searches a place and does not find it, that place gets zero weight and the rest share the weight equally.

The agent interacts through two interfaces. The simulation interface is borrowed from prior work called WALL-E in ALFWorld, and an oracle version in ScienceWorld. An oracle here means they use the true game to give a perfect valid-or-invalid answer. Importantly, this simulator is incomplete: it does not predict the full next state, it only checks whether a proposed action is feasible and gives feedback if not. The new belief-query interface is a free query action that does not advance the environment or cost a step. Examples are asking where an object is most likely to be, what was seen in a place, whether a place was already searched, and for a summary of current location, inventory, and searched places. Responses are in natural language, meaning ordinary ranked lists and summaries, so a frozen pretrained language model can use them without retraining.

They test three language models as policies — a smaller open model, a mid-size open model, and a large proprietary model — each paired with two agent styles that interleave thought and action. Those styles are ReAct, which alternates reasoning and acting, and ReflAct, which adds explicit reflection on state relative to the goal.

What they showed — results, honestly, with limitations

On the unseen test splits of both games, adding the combined belief-plus-simulation system, which they call BB-WM, helped the base agent in nearly all cases on both success and efficiency. Success in ALFWorld is success rate on the first attempt, shortened to SR@1, with efficiency measured as average steps on solved tasks. Success in ScienceWorld is average reward from 0 to 100, shortened to AR, where partial credit is given for milestones, with efficiency measured as average reward gained per step.

The gains can be large for smaller models and the two kinds of help often stack. For example, with the smallest model tested paired with ReAct in ALFWorld, base success was about 29.9%, belief alone raised it to about 43.3%, simulation alone to about 43.8%, and the combination to about 62.9%. With the mid-size model and ReAct, base success about 76.1% rose to about 90.3% with belief alone and about 91.0% with the combination. Simulation alone sometimes even hurt — for instance that mid-size model with ReflAct dropped from about 76.1% to about 72.4% with simulation alone — but the combination recovered to about 90.3%, suggesting the two interfaces address different failures. ScienceWorld showed a similar pattern: the smallest model with ReAct went from about 53.0 average reward to about 66.0 with the combination, and reward per step from about 4.6 to about 6.2. The largest model largely saturated both benchmarks, meaning it was already near ceiling, so the paper reports only minor success gains and modest efficiency gains there, for example about 10.4 to about 11.4 reward per step in ScienceWorld. The authors suggest large models can implicitly track state and avoid invalid actions on easy tasks, but the paper does not directly prove that explanation.

A memory-only ablation, meaning a version that remembers seen facts but drops the distribution over unseen locations, performed worse than full belief and sometimes no better than no model. For instance, with the mid-size model and ReAct in ALFWorld, memory alone stayed at about 76.1% while full belief reached about 90.3%. That supports the claim that uncertainty itself, not just remembering, matters.

Qualitative trajectories help explain why. In one ALFWorld task to heat an egg, the egg starts inside the garbage can — an unusual spot the large model apparently does not expect. Without belief it exhausts its budget checking normal spots like fridge and shelves. With belief, the prior list includes the garbage can as allowed, so after ruling out normal spots the agent re-queries, checks the garbage can, and succeeds. In a ScienceWorld plant-growing task, without belief the agent plants the seed in a bathroom bathtub and fails; with belief it asks where a flower pot is, is told the greenhouse with full weight, teleports there, plants correctly, and succeeds.

Limitations are important and stated plainly. These are hand-crafted, benchmark-specific beliefs, not a scalable learned method — the paper is explicitly a “does this interface help if correct?” study. Uncertainty is narrowly limited to object or room location. The simulator is weak and only judges action validity. The largest model was run only once, versus three runs with standard deviations for the others, so those comparisons are less firm. The work does not show how to maintain accurate beliefs from pixels, language history, or open-ended physics, nor how often agents should query versus simulate, which related work suggests is itself a failure point.

Why this might matter to me

Given interests in spatial reasoning in language and vision models, world models, text-to-CAD and parametric geometry, and 3D generation and scene representation, the useful move here is separating “what might the scene be now?” from “what would happen if I edit it?”

Much work in 3D and CAD focuses on forward prediction: generate a shape, render a view, or simulate a toolpath. This paper suggests a complementary query layer that would be valuable for ambiguous geometry: which constraints are fixed versus uncertain, which parts are occluded versus absent, which receptacles or volumes are still viable for a target part. For text-to-CAD, an analogue of “where is the egg?” could be “which sketches or extrusions could host this hole given the parametric history?” with ranked candidates that update as the user reveals or rules out features. For scene representation, it points toward exposing an explicit occupancy or layout belief — free versus unknown versus occupied — rather than forcing a planner to infer uncertainty from many sampled renders or rollouts. The blind-spot moral carries over: a generator that samples one plausible completion can look confident while hiding that the input was fundamentally under-constrained, whereas a belief query makes the need for an information-gathering move explicit, like requesting another view, a section cut, or a dimension.

Terms worth knowing