The one-paragraph version
HexaShell asks a simple question: what happens if you let an ordinary software coding agent drive a two-armed robot, with no special robot brain added? The team gives the agent a deliberately thin remote control — one command called robo, three color pictures and a one-page manual — that only turns commands into joint motion and never decides anything itself. Work is split between two copies of the same model, Codex powered by GPT-6-Astra: an acting agent tries a task, and after a failure an optimizer agent reads the full log with every command, picture and message, diagnoses the bug, and writes or repairs a small reusable program called a tool, complete with tests. Learning stays as code, not as neural-network weights, and no demonstrations or training are needed. Because the tools locate objects only with geometry from calibrated color cameras — where rays from two views meet, or where a ray meets a measured flat surface like a table — they work in simulation and on a different real arm. On RoboDojo’s official hidden test, 50 episodes on each of 42 tasks, this reaches 46.9% success and 53.2 progress score, the top of the leaderboard, up from 22.5% for the same model without self-made tools. The limit is touch, not thinking: memory tasks hit 81.0%, precision tasks only 32.0%.
The problem they're solving, and why it matters
Robot leaderboards have been dominated by learned policies. A vision-language-action model, meaning a big model trained on thousands of human demonstrations to output arm motions directly, or a world-action model, meaning a model that first imagines future video and then steers toward it, holds the top spots. Separately, vision-language-model agents — chat-style models that can see pictures and use tools — have started to compete by picking gripper targets through a fixed harness written by the benchmark makers.
The authors ask whether there is a third path, borrowed from software: the coding agent that reads code, runs it, reads the error, fixes it, and keeps the fix. On a computer that loop is powerful because everything is exact text. On a robot it is hard because what the agent sees is pixels, but what it must command is a position in the world to about a centimeter. Vision-language models are known to be poor at judging such distances from a single image, and the standard observation here has no depth image. If a coding agent could overcome that by writing its own measuring tools, we would get reusable, inspectable skills without collecting demonstrations. If it cannot, we learn exactly where the boundary is.
The key idea, in plain words
Leave all decisions to the agent, and make what it learns persist as tested code.
Think of cooking in someone else’s kitchen with only a walkie-talkie. The harness is the walkie-talkie: “move right 5 cm, close hand.” It does not chop, season, or decide. The recipe cards are the tools: little programs like “find the top of the T-block” or “pick on a moving belt at the right second” that the agent itself writes after burning a dish.
A concrete example is pushing a T-shaped block onto a T-shaped pad within 7 millimeters. At first the height-measuring tool refuses to answer because its check for “do these two cameras see the same area” counts grid cells, and a far camera naturally fills fewer cells than a near one. The optimizer reads that refusal message, not just pictures, replaces the count with the true footprint of each pixel on the plane, adds a test, and next time the height comes back as 0.780 meters instead of a guessed table height. The same placement that previously looked 12.5 mm off now measures 1.6 mm off and succeeds. That fix stays for all later layouts.
How it actually works, step by step
The sandbox and remote. The acting agent is a stock coding-agent program in a container with no simulator, no scoring code and no internet except the model. It sees a folder refreshed on demand with three 640 by 480 color images — one fixed head view and two wrist views — plus for each camera its internal lens data and its position in the world, plus the poses of the two gripper tips, called tool center points, and joint angles. A roughly 60-line manual defines base moves: translate a tip up to 0.20 meters, rotate up to 90 degrees, choose a gripper orientation, open or close, go home, wait, observe, check status, finish. Every move returns a report with whether planning worked, what pose was reached, and how much of the task’s action budget remains — typically 200 to 1900 steps at 25 actions per second. Thinking and measuring cost nothing; moving costs steps.
Behind the scenes a server checks each command, draws a straight path spaced every 2 cm, solves joint angles with a solver called cuRobo, times it within official speed limits, and sends it through the official RoboDojo control call. The same server can drive the simulator directly for development or act as a policy for the official client that has no reset.
Turning pixels into positions. A pixel plus calibration gives only a ray outward from the camera, like pointing a laser without knowing how far to go along it. The tools add one more constraint, using plain textbook geometry. First, the shared table tool finds table height once per episode for free by trying candidate heights and keeping the one where images warped through that flat surface line up best — a technique called plane sweep. Second, any other known flat height, like a block top or bowl rim, turns a pixel on it into a point. Third, if two cameras see the same point, the point between the two nearly meeting rays is triangulated. The model picks which pixel, the code computes the world point. If the head view is occluded or too coarse, the agent moves a wrist camera for a second view. That is active perception, and it spends precious action steps.
Tools and the fix-it loop. A tool is a folder with code and a short interface description that gets appended to the manual. It can only use joint angles, tip poses, motion primitives, and the latest pictures plus calibration — never hidden simulator poses, never hard-coded sizes from task files. Free measuring calls are called often; moving calls return the same planning verdict as base moves.
Development runs layout by layout over the first ten layouts of seed 0. On success the optimizer writes a short playbook note. On failure it reads logs, frames, transcript and code, writes a diagnosis, edits one tool with tests, and the same layout is retried, up to five edits. After ten layouts the set is frozen and retested once with no optimizer. The eight Open evaluation-only tasks share one generic setup with no task-specific tools: a scene finder that separates table color from objects and masks arms, plus eight shared skills for points, outlines, rims, guarded grasps and timed belt picks.
What they showed — results, honestly, with limitations
In RoboDojo’s own official run on a hidden seed, HexaShell leads: 46.9% success, 53.2 progress, versus 39.4% and 46.6 for the best learned-agent system WorldScape-R2SI, 32.3% and 39.3 for VPP2, 31.4% and 36.3 for PhysicalRSI, and 22.5% for the same GPT-6-Astra model through the benchmark’s fixed harness. By type: Generalization 34.5%, Precision 32.0%, Long-Horizon 31.5%, Memory 81.0%, Open 55.3%. It leads where reading and remembering decide; learned policies still lead where fine contact decides, with VPP2 at 35.2% on Precision and 37.0% on Long-Horizon. Two variants pass all episodes, 11 stay at zero.
Three findings clarify why. First, clutter hurts learned policies far more: HexaShell keeps 82% of its standard success in cluttered scenes with 13 to 18 extra objects, versus a median 20% for 31 learned policies, because the agent points at the target pixel and the tool measures only that object. Fold-clothes is the exception, dropping 56% to 24%, because its color-connectivity finder breaks when clutter touches the shirt.
Second, geometry beats a depth network. With the same harness and model, triangulation scores 8.7 points higher than Depth Anything V2 on the leaderboard metric, turning 57 failures into successes while losing 23. The reason is accuracy: even after fixing scale with the table, that network’s median error was 41 mm at the head and about 100 mm at wrists, while grasps need about 10 mm. True rendered depth, not normally available, is still 10.6 points higher, mostly in tight insertion.
Third, looking costs little and pays. Viewing moves take 3.6% of all steps versus 0.7% with depth, appear in 43% of episodes, yet fewer episodes hit the step limit. On layouts where viewing was used, triangulation wins 39 to 22.
Limits are sharp. By physical need, pushing hits 92.5% and rigid pick-place 50.0%, but articulated use 7.5% and tight insertion 8.3% with 7 of 12 variants at zero. Most failures, 57%, exhaust the budget while the agent sees the error and retries — an execution gap — while memory failures are often false declarations of success, a perception gap. Insertion needs the hole and part in one frame to millimeters; a head pixel already covers 2-3 mm. Honest caveats: each task had up to 12 hours of optimizer time with a median 6.3 minutes per fix round, retests take a median 739 seconds mostly in model waiting, 25 tasks gave the optimizer a short note about checker rules the actor never sees, and the real-robot proof is only two qualitative demos on PiPER X arms after a small arm-model change.
Why this might matter to me
For spatial reasoning in LLMs and VLMs, this is a clean separation experiment: let the model do what it does relatively well — pick a pixel on a top face, rim or mallet head — and let deterministic code do what it does poorly — turn rays and planes into millimeters. The Xylophone case, where one view errs by 23 mm and two views by 2.7 mm, and the Swap-T case, where mixing side and top faces of the same hue corrupts rotation by 3 degrees, show that many “reasoning” failures are really ungrounded measurement.
For world models, it is a counterpoint to imagining futures in pixels. WorldScape-R2SI and VPP2 learn dynamics from data; HexaShell measures the present with calibration and pays in motion steps instead of training data. The fact that explicit two-view geometry beats monocular depth networks here suggests hybrid futures: use generative world models for contact prediction, but keep calibrated triangulation for free-space localization, especially under clutter.
For text-to-CAD and parametric geometry, the tools read like tiny parametric fitters: footprint and turn on a plane, rim circles in two views, heights from plane sweeps. Language names the feature, code fits parameters with tests. That pattern — instruction to outline to plane-constrained solid — is directly reusable for picking CAD constraints from images.
For 3D generation and scene representation, the lesson is about what representation survives sim-to-real. Because tools use only color images, calibration and arm state, they transfer to a different arm with little change. No neural depth, no hidden poses, no memorized textures. If you build scene models, consider exposing the same contract: rays, planes and two-view agreement as first-class outputs, not just novel views, so an agent can both see and measure.
Terms worth knowing
VLM agent / coding agent: a vision-and-language model wrapped in a loop that can read files, run shell commands, and edit code. Here an unmodified Codex CLI.
Thin harness: the robo command line that only translates agent commands into joint motion and returns feedback. It makes no task decisions.
Tool: a small persistent program the agent writes, like surface_pair or timed_pick, with its own interface text and tests. Task-specific except for shared pick and table_plane.
Acting vs optimizer agent: two roles of the same model. The actor runs episodes; the optimizer diagnoses failures and edits tools between episodes.
Triangulation: locating a 3D point where rays from two calibrated cameras nearly meet. Needs the same point visible twice.
Plane sweep / ray-plane intersection: trying candidate flat-surface heights, warping one camera’s image into another through each height, and keeping the best match; then a pixel’s ray meeting that plane gives a point.
Active perception / viewing move: moving a wrist camera to get the needed second view. It costs action steps from a fixed budget.
Step limit and progress score: RoboDojo ends episodes after a fixed number of actions or at first checker success. Progress is partial credit toward success; the leaderboard averages five task families with Generalization counting standard and cluttered equally.