All summaries · 2026-09-11

Code World Model: Coding Agent as World Brain

Yiwen Chen, Guosheng Lin, Chi Zhang

arXiv:2608.25927 · abstract · pdf · summarized by meta/muse-spark-1.3-contributor on 2026-09-11

The one-paragraph version

This paper proposes a new way to build open-ended world models by splitting the job in two. A coding agent — a large language model that can write and edit programs — acts as the “world brain.” It reasons about events, rules, and long-term consequences, and it writes executable code that keeps track of a persistent world state: who is where, what changed, what rules are now active. A video model acts as the visual renderer. It does not have to remember all that hidden logic; it just turns the current state into high-fidelity video. To connect them, the authors introduce a proxy: a coarse, code-built 3D sketch of positions, trajectories, layout, and camera motion, rasterized into a low-resolution proxy video. That proxy video plus text conditions a fine-tuned video model, MiniMax-H3, to generate the final observations. Trained on only about five hours of paired game data, the system qualitatively follows proxies made by the coding agent while inventing rich appearance and fine motion itself.

The problem they're solving, and why it matters

A world model, in this paper’s sense, is a system that represents how a world stands now and how it changes when actions and events happen. The motivating example is a fantasy game: if a player assassinates a city’s ruler and leaves, succession, trade, patrols, faction attitudes, and many characters’ plans should keep evolving off-screen, and still matter when the player returns much later.

Current video-based world models, the authors argue, are bad at this. A video shows only observable outcomes — people moving, camera shifting — not the hidden knowledge, rules, relationships, and history that caused them. Important changes happen out of view or over timescales far longer than a model’s context window, which the authors note is typically under a minute of video versus days or years of world time. Scaling up video data gives more outcomes but still no direct record of the underlying mechanisms, like game code that was discarded after rendering. The result is worlds that look good moment-to-moment but cannot maintain persistent consequences or reason coherently about what should happen next. That limitation matters for games, autonomous driving, embodied intelligence, and for training and evaluating agents that need a world that keeps evolving beyond a scripted trajectory.

The key idea, in plain words

Decide what happens with code, show how it looks with video.

The authors argue those require different kinds of learning. Broad world knowledge, commonsense, long-horizon planning, and rule-following have already been learned well from text and code by language models. Rich appearance, natural motion, and physical interaction have been learned well from video by generative video models.

So they do not ask one model to do both. The coding agent handles infrequent but hard reasoning — interpreting a new event, deciding a city should enter an emergency state, adding a patrol or succession mechanism — and turns that decision into reusable code. That code then handles frequent, repetitive, deterministic updates — positions, collisions, cooldowns, damage, counts — without calling the big language model every step. The video model is then freed from having to simulate hidden logic, and only has to realize the current state visually.

The bridge is the proxy. Think of it as a programmer-controlled storyboard made of very simple shapes: capsules for people, wireframes for cars, boxes for complex objects, plus depth and identity colors. It says where things are and how the camera moves, frame by frame, but deliberately says nothing about texture, lighting, materials, or fine motion. Text says what things are and what they are doing. The video model fills in the rest.

How it actually works, step by step

Interaction flows in a loop. A player or environment action changes the situation. The coding agent reads the maintained world state and either calls existing code or edits the world program. Code executes and advances many variables. The new state is compiled into proxy plus text, which condition video generation. The generated video and code execution feedback return for the next step.

To make that precise, the paper first describes a conventional world model in words: current world state plus current action determines next state, which determines the next observation. The authors then split that idea of state into two parts. The first part, which they call executable state, is everything code can directly operate on: the world program itself, entity attributes, rules, relations, and event history. The second part, which they call visual state, is appearance and motion that must stay consistent across frames. In symbols only after that idea: complete state S_t is the pair of executable state S_exe_t and visual state S_vis_t. Code plus agent advance the executable part, written as S_exe_{t+1} = T_AC(S_exe_t, A_t), where T_AC means the joint agent-code transition and A_t is the action. The video model G then generates the new visual part from the old visuals plus the new executable state.

The crucial practical problem is how to feed that executable state to the video model. The authors first tried the obvious route: structured text listing identities, positions, orientations, and intent. They report that even recent large video models with a dedicated camera pathway failed to follow precise camera trajectories and entity motion from text alone. Language could in theory describe every pixel, but it would be token-inefficient, slow, and unreliable.

Hence the proxy. The coding agent programmatically combines a small library of simple reusable primitives with addressable parameters for position, scale, pose, layout, camera, and state markers. A lightweight deterministic compiler rasterizes them into a proxy video at one-quarter resolution per spatial dimension — so one-sixteenth the pixels and, the authors say, negligible extra token cost. Text and proxy come from the same state but use complementary channels: text for semantics and appearance, proxy video for frame-wise spatial constraints.

To teach a model to follow proxies, they build aligned pairs. For game data, they record gameplay video and, synchronously from the same run, only the state needed for the proxy: camera, entity positions and orientations, approximate scale, layout, and interaction state. No full meshes, textures, or materials. This can be recompiled into different proxy granularities without re-recording. Each proxy pixel also carries a ground-truth instance identity, so text descriptions can be bound to the right region. For real data, they show a proof-of-concept on KITTI-360 without game-engine access: rectified RGB as target, calibrated poses plus accumulated 3D reconstruction and 3D object labels used only offline to project depth, normals, and the same coarse primitives through a shared depth buffer to preserve occlusion and alignment. No action labels are needed because action effects already appear as motion in the proxy.

What they showed — results, honestly, with limitations

The evaluation is entirely qualitative, with no quantitative metrics, benchmarks, or user studies reported in the paper. That is important to state plainly.

For training, they fine-tune the MiniMax-H3 Ref2V A backbone with rank-128 LoRA, about 596M trainable parameters, on 9,420 five-second clips sampled from 157 takes totaling about 5.6 hours of GTA V-like gameplay. Targets are 124 frames at 1344x768 and 24 FPS; proxies are aligned 124 frames at 336x192 combining depth and semantic-ID maps. Inference uses a GPT Image 2 first-frame anchor that is instructed to respect proxy layout and framing, then generates 124-frame windows with 34-frame overlap for longer videos.

At test time, an agent the authors call GPT-5.6 Sol builds simple interactive worlds by reusing and editing existing game-engine code for controls, collisions, and update loops, not by writing AAA-scale systems from scratch. The authors show examples — a wand dance, a lagoon battle, a manta-skiff flight, a meadow run, a monastery run, plus appendix prompts like a stationary rainkeeper with arcing camera, a courier walking then falling, and a stationary bird with idle motion — where the model preserves layout, entity count, trajectories, and camera motion while generalizing appearance far beyond the training game.

They claim finer, more responsive control than systems that condition only on low-dimensional action or camera signals, but the direct motion comparisons are said to be on a project webpage, not evaluated in the paper itself. Limitations the authors acknowledge: small compute and data scale, no autoregressive real-time streaming implementation, and coding agents still cannot reliably invent highly complex game mechanisms from scratch. The real-world KITTI-360 pipeline is only a construction demonstration, not used to train the reported video model, so its benefit for realism remains unsupported so far.

Why this might matter to me

Given your interests, the value here is less the demo quality than the interface pattern.

For spatial reasoning in LLMs/VLMs: the paper is an admission that dense text alone does not give current video models precise frame-wise grounding of position, occlusion, trajectory, and viewpoint. The proxy is essentially visual prompting — control in the output modality itself — while keeping every signal traceable to code. That suggests a research direction where language models reason symbolically but delegate spatial precision to an inspectable intermediate sketch rather than hoping attention will resolve it from words.

For world models: the executable-versus-visual split offers a concrete answer to persistence and off-screen causality. Memory is not another history buffer of frames; it is variables plus revisable rules that keep running when unobserved. The agent-code frequency decoupling — slow reasoning to write rules, fast code to run them — is directly relevant to long-horizon simulation.

For text-to-CAD and parametric geometry: the proxy vocabulary is deliberately like coarse parametric CAD — primitives with position, scale, pose, and extents, plus a deterministic compiler. It shows a minimal sufficient geometry that a coding agent can actually maintain, with a fallback to bounding box plus text when discriminability fails. That trade-off they call condition bandwidth, between constructability by the agent and grounding strength for the generator, maps closely to level-of-detail decisions in parametric modeling.

For 3D generation and scene representation: the proxy is explicitly not a final world. It uses depth, semantic IDs, and shared depth-buffering for occlusion, but leaves materials, lighting, and fine dynamics to learned priors. That points toward hybrid scene representations where explicit structure guarantees layout and viewpoint consistency while generative models supply open-ended detail — a middle path between full 3D reconstruction and pure video synthesis.

Terms worth knowing