The one-paragraph version
This paper asks whether a robot agent driven by a large multimodal model needs to look at camera images at every single decision step. The authors argue no — for routine manipulation, most of what matters can be carried in words and symbols instead. They build a go-between layer, called a geometric abstraction, that watches calibrated color-plus-depth cameras locally, converts what it sees into a small list of task-relevant 3D parts like centers, corners, edges, and faces, and then hands that list to the language model as text. The language model then writes programs or picks actions using stable names like red.bottom_face and blue.top_face, while the local layer keeps rebinding those names to fresh measured coordinates after each move. Across simulated tabletop building tasks, this text-first approach completed more multi-step assemblies than an agent that reasoned directly from pixels, and it did so with far fewer image queries and less waiting on the model.
The problem they're solving, and why it matters
Frontier multimodal models — meaning language models that can also take images as input — are getting good enough to control robots directly, without training a separate policy from scratch. The usual way to do this is closed-loop and image-heavy: take a picture, send it to the model, ask what to do, act, take another picture, repeat.
The paper argues this is wasteful and brittle for two reasons. First, what a stacking or placement decision actually depends on is often compact: where is this face, where is that face, how far apart are they, are they aligned. Recovering those precise distances and orientations from raw pixels over and over invites error. Prior evaluations cited by the authors find that vision-language models still struggle with exact metric and structural 3D relations from images alone, but reason more reliably when those relations are spelled out explicitly in text.
Second, images are token-heavy. Sending full pictures at every step costs time and money, even when most pixels have not changed in a relevant way.
The alternative extremes are also unsatisfying. Keep everything as raw images and you pay repeated interpretation cost. Reduce everything to crude perception outputs like a single object center or bounding box and you lose the part-level detail you need for precise mating, like putting a beam across two blocks or threading a ring onto a peg. The authors want a middle path: keep rich 3D structure, but expose it in a form a language model can reuse without re-seeing.
The key idea, in plain words
Separate what stays the same from what changes, and only refresh what changes.
What stays the same is the symbolic plan structure: the idea that to stack, you align the bottom face of the moving block with the top face of the target block, check support, place, then check the leftover error. That logic does not depend on exact coordinates.
What changes is the metric grounding: exactly where those faces are right now in meters, measured in the shared world coordinate frame, and how trustworthy that measurement is.
Concretely, imagine building a tower from red, green, and blue cubes. Instead of showing the model a new photo after every nudge and asking “are we there yet?”, the local perception system quietly re-measures the scene and sends a short text note: the center of green is at this position, the gap to its goal is this vector in meters, the bottom face was directly seen but the back face is only guessed from the known cube shape. The language model reasons over that note plus reusable names for parts. It only asks for a fresh image when the text is not enough — for example, when something failed in a way geometry alone cannot explain, or when it needs richer context.
A useful image from the paper is Figure 1: without abstraction, the loop is observe-image, reason-in-multimodal-model, act. With abstraction, the loop is refresh-geometry-locally, reason-in-text-plus-symbols, act, with images on demand.
How it actually works, step by step
The system has three linked pieces: local grounding from cameras to 3D, symbolic names plus small geometric tools, and short text updates for reasoning.
1. From camera pixels to measured 3D parts. At each time, the robot has color and depth images from a side camera and a wrist camera, plus its own joint readings. Color segmentation says which pixels belong to which object. Depth says how far along the camera ray each pixel is. Using the known camera lens properties and camera-to-world transform, each valid pixel is projected out into a 3D point in world coordinates — in words, un-project the ray through the pixel and push it out to its measured depth.
Those point clouds are then fit to simple shape models. Because objects are regular — cuboids and cylinders in the tests — the system starts from an initial size estimate and a prior that objects sit on a table, masks out the robot arm itself using known arm geometry, and fits pose and visible supporting surfaces. It records not just a center point, but typed parts when supported: center, orientation, size, and where visible, vertices meaning corner points, edges meaning line segments between corners, and oriented faces meaning flat surfaces with a center position and an outward-pointing direction.
Crucially, every piece carries evidence status and a timestamp: directly observed, inferred from the shape prior, or unavailable because of occlusion. A standing rule is that shape size may persist, but pose must be refreshed — after an object moves, you cannot just copy its old pose or assume your command worked; you must re-measure it or mark it unavailable.
2. Symbolic references and geometric tools. A policy written by the language model does not hardcode numbers. It uses a reference that names an object and a geometric role, for example, the bottom face of the red block. Evaluated against the current 3D state, that reference either resolves to a value plus its type, evidence, and time, or fails explicitly if the geometry is not currently supported.
Around those references the authors provide a tiny operator set, summarized in their Table 1:
- reference: look up a part and report its evidence.
- relation: test a spatial predicate like support — is A above B with enough planar overlap and compatible face orientation — and return true, false, or unknown plus the numeric leftover error.
- align: compute the rigid motion to mate two faces, meaning the rotation plus translation that would bring the source face center onto the target face center and flip its normal opposite to the target normal. Since a point plus a normal still leaves spin around that normal free, the tool returns the smallest-angle rotation and reports the remaining twist.
- checkpoint: package the relevant geometry for reasoning.
This lets the same program generalize: align bottom to top works whether the tower is on the left or right side of the table, because the names rebind to new numbers.
3. Text checkpoints instead of re-sending images. After each action, the system returns a compact structured text summary for the active object and its goal. In words, it says: here is the current position of the chosen reference, here is the vector from current to goal — what the paper calls the metric residual or goal residual — and here is whether that estimate was seen or guessed. If the object is occluded, it keeps the goal but explicitly marks the residual unavailable rather than hallucinating from a stale pose.
4. Two ways to use the same interface. In policy synthesis, the language model revises a Python program between batches of rollouts. During a rollout, geometry and control run locally with no model calls. Between rollouts, outcomes and observations inform the next revision. In online execution, the model stays in the loop of a single continuing episode with no reset after failures: it reads the fresh text checkpoint plus history and prior skills, picks the next action, then the local layer re-grounds and reports again.
What they showed — results, honestly, with limitations
All tests are in MuJoCo simulation with a simulated SO-100 arm and calibrated side plus wrist color-depth cameras. The baseline, called Pixel Reasoning, reasons directly from visual observations and writes its own perception code. The proposed agent gets the same task, sensors, actuators, and starting skills, plus the grounded part references and text checkpoints. Neither side sees simulator ground-truth poses.
In single-program learning, with five training layouts, three revisions labeled R1 to R3, and 100 separate held-out layouts per task: on Bridge-plus-cylinder and Three-cube stacking, the geometric agent jumps ahead early — roughly 45% versus 21% on the bridge task at first revision, and roughly 57% versus 17% on stacking — and by the second revision already beats the baseline’s final third-revision score, using about half the cumulative tokens at that point. Final scores remain higher with lower variation across runs. On Ring-on-peg, which needs fine alignment of a hexagonal ring onto a vertical peg, both start at 0%, but the geometric agent reaches about 38% by R3 while the image-only agent stays at 0% within the budget. The paper is clear these gains combine better synthesis plus local grounding during execution; they do not isolate text formatting alone.
In continuous online execution under a 200-second wall-clock budget per episode, with 50 paired layouts per task and no reset until success or timeout: the geometric agent solves Bridge-tower on 28 of 50 versus 12 of 50, and Tower relocation on 42 of 50 versus 15 of 50. On Ring-on-peg, where both almost always succeed after a shared grasp-and-lift prefix, it finishes about one-third faster and needs far fewer model calls — roughly 6.9 versus 11.9 requests, 2.7 versus 10.9 attached images, and about half the tokens and model waiting time. Tower relocation shows a similar halving of model time and tokens with similar physical execution time. Bridge-tower is the exception on token count: it more than doubles completion with slightly less model time but about 15% more tokens, suggesting the benefit there comes from better corrections, not just shorter prompts.
Limitations are substantial and stated openly. Everything is simulated tabletop with regular cuboids, cylinders, and rings, using lightweight CPU estimators — no real-world sensor noise, heavy occlusion, calibration drift, articulated or deformable objects, or learned 3D perception. Contact and stability checks are geometric candidates, not physical certificates, and some evaluation contact checks are noted as incomplete. Nine online pairs needed audited repeats due to runtime timing or hardware faults. This is evidence that explicit geometric state helps in these controlled settings, not proof it will transfer unchanged to the wild.
Why this might matter to me
If you care about spatial reasoning in language and vision-language models, this is a concrete case for “think in text about explicitly measured space” rather than “think in pixels.” The checkpoint idea suggests a benchmark direction: give different text-only models the same standardized geometric state and residuals, and see which ones can actually do multi-step spatial repair. It also reframes world models: instead of predicting next raw observation, predict next compact, evidence-tagged geometric state, with images consulted only when uncertainty is high.
For text-to-CAD and parametric geometry, the parallel is direct. CAD already separates stable topology and constraints from numeric parameters — mate this face coincident, keep this extrusion length. The paper’s reusable symbolic references plus rebound metric values are the robotics analogue of parametric sketches and assembly mates, including the honest handling of under-constrained degrees of freedom like twist around a normal. Techniques for constraint solving, mate inference, and tolerance reporting could transfer both ways.
For 3D generation and scene representation, the lesson is that a persistent, part-typed, evidence-aware scene graph — centers, edges, oriented faces with observed versus inferred tags and timestamps — can be a better substrate for an agent than either raw images or a single centroid per object. That points toward hybrid representations: neural fields or meshes for rich shape on demand, distilled into a small symbolic-text layer for fast language-model planning, refreshed locally rather than re-inferred globally at every step.
Terms worth knowing
- Agentic policy: a robot controller where a language model actively synthesizes code, picks skills, diagnoses failures, and repairs behavior, rather than just mapping one observation to one action.
- Multimodal / vision-language model: a language model that also accepts images, and sometimes depth or other modalities, as input alongside text.
- RGB-D: color images plus per-pixel depth, allowing back-projection into 3D points when camera calibration is known.
- Grounding: binding a word or symbol to a concrete measured value in the current scene, for example resolving “bottom face” to a specific 3D position and orientation right now.
- Symbolic reference: a stable name for a geometric role, such as the top face of the blue block, that can be rebound to new coordinates as the scene changes.
- Textual checkpoint: a short structured text report after an action summarizing relevant geometry, goal residuals, and evidence status for the next reasoning step.
- Metric residual: the numeric vector from where a part is to where it should be, for example 2 cm left and 4 cm down from the goal.
- Evidence status: a tag saying whether a geometric estimate was directly observed, inferred from a shape prior, or unavailable — so the reasoner knows when not to trust a number.
- Rigid transform / alignment: a rotation plus translation that moves one part onto another without deforming it; here used to mate source and target faces.
- Policy synthesis vs. online execution: learning a reusable program between episodes versus making decisions step-by-step inside one unreset episode, recovering from failures as they happen.