The one-paragraph version
This paper argues that today's robot policies that map camera images plus language straight to motion are missing something essential: an inspectable memory of what is true, what is done, and what remains. The authors propose Physical Coding, where a task is carried out by two linked programs — one that writes down the world state as code, and one that writes down the procedure as code — checked by an independent verifier. They build an agent system called HexaAnything that uses this setup to call tools, including existing vision-language-action models, verify each step from fresh camera and sensor evidence, and recover when something fails. Verified runs are then saved as reusable data and training examples. In tests on simulated kitchen tasks, simulated physics labs, and a real two-arm robot, this harness improves success over the underlying action model alone, and traces it collects can train a slightly better planner.
The problem they're solving, and why it matters
The starting point is a failure mode in vision-language-action models, meaning models that take an image and an instruction and output a chunk of robot motion, and in world-action models, meaning models that also try to predict future video frames.
The paper shows these models often ignore the instruction and memorize scene-to-motion patterns. As one example, a policy trained on the LIBERO benchmark with the instruction completely masked still succeeds in 92.3% of trials, compared to 96.2% with the instruction present. The paper also cites prior work finding that small changes in viewpoint, robot starting pose, or object layout can cut success by more than half or drive it near zero.
The authors diagnose this as a representation problem, not just a need for bigger models or more data. An action chunk plus a stop signal has nowhere to record unfinished subgoals, conditions for the next step, or what to do after a mistake. A concrete example runs through the paper: a robot puts bread in an oven and closes the door while a kebab is still on the counter, making the task unrecoverable. Nothing in the action output itself says "both ingredients must be inside before closing." Closed-loop re-prompting helps for one episode, the authors say, but without a checkable, persistent record, the lesson does not become a reusable, independently verifiable artifact.
The key idea, in plain words
Borrow the working method of software coding agents. A software agent succeeds because everything lives in one shared, editable medium: the repository is the state, an edit or tool call is the action, and tests decide completion independently of the agent's own claim. Fixes and tools persist as versioned code.
Physical Coding tries to give the physical world that same medium, using two coupled programs:
Code as World is an executable notebook of what is true. It lists relevant objects, spatial relations, measurements, constraints, and progress checks, each with its source. Example: "bread inside oven = false, kebab inside oven = true, therefore both-inside = false," tied to a specific camera image and tool output.
Code as Policy is an executable recipe for what to do next. It breaks the task into observable subgoals, calls tools, branches, loops, interrupts a stuck tool, verifies the result against Code as World, and recovers or stops. Example: "only allow close-door if both-inside holds on fresh evidence; otherwise place the remaining item."
Because both are code, failures can be localized to a missing predicate, a bad workflow branch, a broken tool, or a weak check, and successes can be kept as data, memory, or training signals. The authors call the larger long-term vision self-evolution: improving the harness, meaning the tools, workflows, verifiers and recovery logic around the model, plus data and environments, model weights, and eventually hardware, all gated by independent evaluation.
How it actually works, step by step
HexaAnything is the concrete system. A general language model writes and updates the world and policy programs, and a piece of software called the Harness runs them and connects them to the robot.
The loop is observe, update world, act, verify, recover or stop:
First, observe. The system gathers language goals, camera images and depth, robot joint and gripper state, and outputs from detectors and instruments. It does not read hidden simulator state; it must build state from what a real robot could see.
Next, update the world program. Scene structure is written as code from images, measurements come from tools such as a ruler reader or optical timer, and execution state comes from the robot. Entries are updated one at a time with provenance, meaning what call produced them and when, so contradictions stay visible. If evidence is too weak to decide a question, the verdict is insufficient evidence, not a guess.
Then, act through typed tools. Available tools include perception, motion planning, grasping, placement, plus learned action models like XR-1 called with a local subgoal, plus file and computation tools for analysis. The policy program selects the next tool call.
Then, verify independently. A verifier checks the relevant world predicate against fresh evidence and returns a typed verdict such as pass, fail, insufficient evidence, blocked, or safety stop. The model's own claim of "done" and a low-level "finished" flag do not count as success. In simulation, the benchmark's own success check assigns the final label.
Finally, recover or persist. Depending on the verdict, the Harness continues, re-observes, retries, replans, or stops fail-closed rather than promoting ambiguity to success. Every step leaves a record of what was read, what ran, what was observed, and what was decided. Failures become targeted edits; verified traces can train the next model, called HexaModel.
A seven-step evolution protocol governs larger changes: freeze tasks and seeds, execute, verify independently, diagnose the bottleneck, propose a bounded edit, test with regression and held-out checks, then commit or roll back.
What they showed — results, honestly, with limitations
The evidence is preliminary and component-specific, which the authors state explicitly.
With the action model held fixed on RoboCasa365, a kitchen benchmark with Atomic-Seen, Composite-Seen, and Composite-Unseen splits: using XR-1 natively versus calling the same XR-1 as a tool inside HexaAnything driven by a model called GPT-5.6-Sol, overall success rises from 56.6% to 61.1%, with Composite-Seen from 54.8% to 61.5% and Composite-Unseen from 34.3% to 38.3%. A general coding agent called Codex with the same driver reaches 59.5% overall, beating HexaAnything on Atomic-Seen but trailing on composites. On three hard unseen tasks with 100 seeds each, gains are larger: LoadKebabSandwich 17% to 48%, PortionHotDogs 22% to 37%, WaffleReheat 53% to 70%. Doubling the native action budget does not explain this; on PortionHotDogs it stays at 21 of 100 versus 22 normally.
For model evolution, HexaModel v0.1 fine-tunes Qwen3.8-27B on 1,010 agent traces plus state-judgment examples and general data, with harness-returned data making up 38.8% of 178.6M training tokens. Back in the same harness, it beats its base on every split and edges GPT-5.6-Sol overall, 61.7% versus 61.1%, with its largest gain on Composite-Unseen. The absolute margins are small, 0.1 to 2.2 points per split.
For tool evolution on three RoboDojo simulation tasks with fixed tasks and evaluator, two rounds of tool edits raise Fold cloth, Pour vase, and Press by number from 0%, 40%, 0% to 80%, 100%, 100%, non-monotonically. The detailed Fold cloth history is on only five seeds, the operator chose pixels using diagnostic keypoints for some seeds, and re-evaluation without privileged information gives 3 of 5 on development seeds and 4 of 5 on held-out seeds.
Beyond manipulation, a new simulated lab called PhyBench asks the agent to design experiments to estimate spring stiffness, gravity, and coupled-oscillator frequencies, with success thresholds of 15%, 3%, and 10% relative error. With strong drivers, HexaAnything gets valid runs in nearly all 10 trials per task with mean errors below 5%, for example 1.3% on Hooke's law with one model. A weaker driver often fails to produce valid runs. One Hooke's-law trace is shown fitting stiffness as 24.82 against a hidden reference of 25.16.
On a real AgileX dual-arm robot over seven tabletop tasks with three trials each, five tasks succeed 3 of 3, while tossing blocks and folding clothes partially fail on perception and reach. Speed comparisons to published results on other hardware favor HexaAnything but are not same-robot controls, and three trials are few. Longer-horizon kitchen tasks are designed but, the paper says, not yet evaluated.
Why this might matter to me
If you care about spatial reasoning in language-vision models, this paper is a useful caution and a pattern: do not ask a vision-language model for a yes-or-no state judgment when counting, containment, and relations are its weakest skills. Ask it to write checkable spatial code — masks, slots, keypoints, predicates like "each plate has one bread and one sausage" — then verify that code against fresh images, depth, and tool returns. That matches your interest in making spatial competence explicit rather than hidden in embeddings.
For world models, the contrast is sharp. Video prediction rewards plausible frames, not correct physics. Here the "world model" is a symbolic-executable world program with provenance, residuals, and fit code, as in the spring, pendulum, and oscillator labs. That suggests a hybrid direction: neural perception proposes geometry, code-based state and physics fits dispose.
For text-to-CAD and parametric geometry and for 3D generation and scene representation, Code as World looks like a task-scoped scene graph plus constraints and progress predicates, updated incrementally instead of regenerated each query. It is not a CAD kernel or neural field, and the paper does not claim that, but the interface — objects, relations, affordances, tolerances, and a verifier before acting — is close to how you would want language to drive parametric edits or 3D assets safely: generate an editable representation, check it against measurements, keep the verified trace as reusable data rather than keeping only pixels or tokens.
Terms worth knowing
- Vision-language-action model, or VLA: a model that maps images plus language to short chunks of robot motion. XR-1 and Pi 0.5 style models are examples here.
- World-action model, or WAM: a variant that also predicts future video or world state alongside actions. The paper argues predicted frames are not themselves checkable state.
- Physical Coding: the paper's paradigm of representing physical state, procedure, and feedback all as executable code so experience can be inspected, revised, and reused.
- Code as World: the program recording task-relevant objects, relations, measurements, constraints, and progress predicates, with the observation behind each entry.
- Code as Policy: the program organizing decomposition, tool calls, verification, branching, recovery, and stopping, conditioned on Code as World.
- Harness: the software boundary around the model that provides tools, observations, verification, recovery, memory, budgets, and logging. HexaAnything is the paper's harness plus agent.
- Verifier: an evaluator independent of the acting model that checks predicates against fresh evidence and returns pass, fail, insufficient evidence, blocked, or safety stop.
- HexaModel: a planner model fine-tuned on traces returned by the harness, tested back inside the same harness.
- PhyBench: the paper's simulated laboratory benchmark for estimating spring constant, gravity, and coupled-oscillator frequencies from robot-collected measurements.
- Typed trace and provenance: the saved record of task, world and policy versions, observations, tool calls, outcomes, and verdicts, including what produced each observation, used for diagnosis and training.