The one-paragraph version
This paper asks a new question: not can a language model generate a believable world, but can it keep an already-delivered world running correctly over time. The authors call this job the world agent task and build a test for it called WorldAgent-Benchmark, with 37 authored cases in two tracks. In the maintenance track, the model reads a story told round by round — for example, a courier carries a card while a guard walks to a reader — and must turn each event into explicit updates to a stored world state using tools, respecting cause, timing, and what happens at the same time. In the deduction track, the model must predict how a deterministic simulated world will evolve, sometimes with full rules and sometimes with only vague descriptions, sometimes with no feedback and sometimes while acting. Across eight models, getting individual events right is relatively easy, but getting their organization right is hard: scores fall as pre-built help is removed, and checking causal ordering is the weakest part for every model.
The problem they're solving, and why it matters
Most current tests stop too early. Video-style world models are judged on whether a few seconds look good, even if objects or states drift when you look away and come back. Code-generated games are judged at the moment they are delivered, with no check on what happens when a player does something unexpected later. Single-step simulators are judged on one transition at a time.
The paper argues this misses the core failure players actually feel: loss of persistent, organized state. Their running example makes it concrete. A courier brings a card to an east gate. Mid-trip, a guard reaches a reader, then opens the gate with the delivered card. A generative model might later show the gate open even though the card never arrived, because nothing maintained the absence. A one-shot coded game might freeze with no state change if the player tries to climb the gate, because that interaction was never anticipated at delivery time.
In both cases local details can look right — both characters are at the gate, the gate is open — while the causal flow is wrong: the handover was skipped, a dependency was reversed, or an event happened outside its required window. The authors want to measure that difference between correct local outcomes and a valid causal event flow.
The key idea, in plain words
Move the test from delivery to continued operation. Do not ask the model to invent a world. Hand it a world that already exists with explicit records — where places, objects, and abilities are stored as data — and hold it responsible for operating it.
That breaks into two complementary jobs. Maintenance means reactively grounding what happens: when the story says the guard scans the card and the gate opens, you must create the needed objects and abilities if they are missing, trigger the behaviors in the right order, and make sure consequences carry forward to constrain later rounds. Deduction means predictively reasoning: given a starting state plus full or partial rules, forecast what fields will change, what background processes will fire, and what would have happened under a different action.
Crucially, everything is checked against ground truth frozen before any model runs, with logs of what the model actually did. A matching final state is not enough; the paper requires evidence of the right triggers with the right effects in the right interval.
How it actually works, step by step
Think of the benchmark as a shared workshop with the same tools for every world, and two different assignments that use it.
For maintenance, each case has an authored story, a map, and hidden per-round contracts. A contract, meaning the list of requirements for that round, contains required events, meaning things that must happen either stated outright or implied as necessary middles steps like a handover; structure requirements, meaning the types, entities, and capabilities needed; probes, meaning prohibitions like a forbidden route that must not be used while a condition holds; and relations, meaning causal, temporal, and concurrency ordering between events.
Difficulty is varied by removing pre-built structure. In the easy tier everything is supplied: the kinds of things, the specific things, and what they can do. In the hard tier kinds and things are supplied but the model must build the abilities. In the extreme tier only the map is supplied and the model must build kinds, things, and abilities.
The model works through a tool interface. In plain terms: it can ask for help, fetch the story so far and its budget, read the current state and log, build or edit structure, submit one or more behavior triggers together as an atomic batch, and close the round. Only the engine writes authoritative state, and it checks format and structure but does not supply missing story events or enforce story logic. That logic is the model’s job. If it bundles independent events into one generic macro-trigger, it fails even if the end state looks right.
Scoring is two-layered. An isolated language-model judge binds claimed events and structures to actual logged triggers and frozen snapshots, item by item. Programs then validate those bindings, derive each event’s interval from the steps containing its triggers, and check relations. This yields two separate scores per round: completion, meaning share of events, assigned structures, and probes passed; and relation, meaning share of causal, temporal, and concurrency relations satisfied. Relation failures do not lower completion, to separate did it happen from was it organized correctly.
For deduction, six worlds like an outbreak town, a leaking station, or a spy network run on a separate deterministic engine with fixed dynamics. The design crosses information with feedback. Open gives detailed machine-readable rules; Naming gives only brief prose and leaves numbers unspecified. M1 is pure prediction of autonomous evolution with no true intermediate feedback; M2 lets the model act toward a goal and predicts all changes until the next decision, then gets feedback. There is also lookahead, meaning forecast 48 time units ahead under a plan, and counterfactual, meaning what field values would differ at the same time if the last action had been different. Scoring is purely programmatic field-level comparison of predicted changed fields and values versus the engine’s independently computed reference.
What they showed — results, honestly, with limitations
On a common nine-case maintenance subset run by all models, completion clusters tightly from about 0.69 to 0.81, while relation spreads more. Qwen-3.8-Max-0902 leads clearly on both, around 0.813 completion and 0.738 relation, significantly ahead of others. The rest group around 0.69 to 0.75 completion. Swapping the harness for DeepSeek v4.1 Flash leaves scores essentially unchanged, which the authors read as evidence the gap is model capability, not scaffolding.
Two patterns reproduce across full and subset grids. First, a difficulty and time gradient: scores fall steadily from easy to extreme and from early to late rounds within each tier. The paper reports relative completion drops from first round of easy to last round of extreme of about 34.8% for DeepSeek v4.1 Flash, 32.7% for Gemini 3.8 Flash, 22.8% for GPT 5.6 Luna, and 18.7% even for the leader. Second, causal checking is the weakest component for essentially every model in easy and for all eight in extreme, falling from a cross-model mean around 0.683 to 0.356, the steepest drop. In hard, where capabilities must be authored, construction failures dominate for most models, except GLM 5.3 Flash which builds best but pays an ordering cost.
On deduction, pure prediction without feedback is much worse than closed-loop prediction. Under Naming without rules, field-value prediction collapses to around 0.03 to 0.15, even though models still guess which rules fire fairly well — they know something will happen but not what value results. Feedback recovers much of the gap but not all. Lookahead 48 units ahead is a shared failure below 0.25 for all models. Counterfactual ability dissociates: Kimi K3 scores highest around 0.593 while GLM 5.3 Flash is low around 0.323 despite decent interval scores.
Limitations the paper itself notes: the maintenance judge is a language model and can mis-match or miss violations; scores are recomputable from saved judgments but rerunning the judge may differ. Some strong readings like Kimi K3 and Qwen 3.8 Max deduction are single-fold and described as preliminary. The nine-case subset is deliberately discriminative and slightly biased below the full-grid easy mean. The authors also say crash-atomic persistence, sandboxing against a malicious judge, and sensitivity of horizons and tolerances are not established.
Why this might matter to me
For spatial reasoning in LLMs and VLMs, this reframes the question from can you describe a space to can you maintain it. The maintenance state stores regions, tiles, connections, and holder chains where being held by something on a tile counts as being there. Errors like placing an event outside its required during-window or mis-ordering a handover are exactly the kind of containment, adjacency, and interval mistakes that break embodied plans, and here they are scored separately from surface fluency.
For world models, the courier-gate example is a diagnostic you can borrow: test what persists during long unobserved intervals and under unexpected actions, not just next-frame plausibility. The finding that relation scores separate models while completion saturates suggests adding causal and concurrency checks to your own rollouts.
For text-to-CAD and parametric geometry, the easy-hard-extreme progression mirrors your workflow: from editing with all sketches, parts, and constraints present, to authoring constraints and features from scratch, to building types and instances from only a canvas. The paper’s distinction between building structure correctly and using it in the right order, plus its generation-tracking where delete-and-recreate starts a new generation, maps directly to maintaining design intent across edits.
For 3D generation and scene representation, the separation of rendering from scored state is useful: visualization is explicitly not scored, only the underlying explicit records and logs. That points toward scene graphs and operation logs as the source of truth to evaluate persistence, with images as views, and toward field-level prediction metrics for forecasting scene changes under actions.
Terms worth knowing
World agent: the tested role in this paper — not the player or the renderer, but the separate operator responsible for keeping explicit world state correct through tools.
Maintenance track: the reactive part of the benchmark where the model turns a streamed narrative into state updates, building missing structure and triggering events.
Deduction track: the predictive part where the model forecasts evolution under fixed engine dynamics, with or without acting and feedback.
Contract: the frozen per-round ground truth for maintenance, listing required events, needed structures, prohibitions, and ordering relations.
Completion versus relation: completion measures whether required events, assigned structures, and prohibitions were satisfied; relation measures whether their causal, temporal, and concurrency ordering was satisfied. Both can be high or low independently.
Causal, temporal, concurrency relations: causal means each authored cause ended before its effect began; temporal means one event preceded another, sometimes immediately after; concurrency means intervals started together, overlapped, or ended together.
Open versus Naming and M1 versus M2: Open gives detailed rules while Naming gives only brief prose; M1 predicts without state feedback while M2 acts, predicts the interval until the next decision, then receives feedback.