The one-paragraph version
Simify is a training-free system that lets a robot solve new multi-object rearrangement puzzles by thinking them through in simulation first. From just one color-plus-depth photo of a tabletop, it segments each object, paints in the hidden parts, and uses an image-to-3D generative model to make a complete 3D mesh for each thing it sees. It drops those meshes into a massively parallel physics simulator, then runs an evolutionary search — meaning it tries thousands of random placement orders and poses, keeps the winners, mutates them, and repeats — to maximize a given task score like tower height or overhang length. The best, most noise-robust plan is then executed on a real Franka arm with grasp sampling and collision-free motion planning. On three real-world tasks with unseen household objects — stacking, cantilever balancing, and domino-style toppling — it beat both a large-language-model planner and a version without generative completion, mainly because complete geometry lets simulation predict stability and contact correctly.
The problem they're solving, and why it matters
Spatial reasoning here means figuring out not just where objects are, but where they could go to achieve a physical goal, and whether that arrangement will actually stay up, balance, or knock something over once you let go.
That is hard for current approaches. Vision-Language Models, meaning large models trained on images and text, are good at high-level semantics like "put the big box at the bottom," but they often miss the millimeter-level pose and contact physics needed to stack a wobbly bottle on tape or balance a heavy bottle on a thin edge. Vision-Language-Action policies, meaning models that directly output robot motions after training on many demonstrations, tend to copy what they saw in training and struggle to invent a new stacking order when a new shape or starting pose changes what is possible.
The authors focus on a different regime: novel object rearrangement at test time with no demonstrations, no known object models, and no dataset of good arrangements. The robot gets a single view and a score function — for example, height of the tallest point for stacking — and must discover both the discrete choice of which object goes first, second, third, and the continuous choice of exactly what 6D pose to place each one in. This matters because real homes are open-world: you cannot pre-train for every bottle, box, and clutter pile.
The key idea, in plain words
Instead of trying to learn all geometry and physics inside one neural network, give the robot external thinking tools: a 3D world builder plus a physics playground.
Think of how a person solves "build the tallest tower from these three odd objects." You do not just guess blindly — you mentally rotate the bottle on its side, imagine if the tape roll would make a wider base, and discard ideas that look tippy. Simify does that literally: rebuild the scene as simulatable 3D objects, then brute-force imagine thousands of futures in parallel physics and keep the one that scores highest and survives wobble.
The two enablers are, first, modern generative 3D priors that can hallucinate the unseen back, bottom, and occluded sides from a single photo, and second, GPU-parallel simulators that can settle thousands of stacks or topples in seconds. Completeness turns out to be critical: a mesh missing its bottom cannot stack or roll correctly in simulation, so the plan will not transfer to reality.
How it actually works, step by step
The pipeline has three stages: make the simulation, search in simulation, build it for real.
1. From one photo to simulation-ready meshes. The wrist camera takes one color-plus-depth image. A bottom-up segmenter called SAM-2, meaning Segment Anything Model version 2 that proposes masks for anything that looks like a separate region, over-segments the scene. A vision-language model, GPT-4o in their tests, looks at the numbered segments and groups them into manipulable objects, so no fixed object list is required.
Because image-to-3D models are trained mostly on clean, unoccluded views, each object crop is first completed with a 2D amodal inpainting model, meaning a diffusion model that fills in the parts hidden behind other objects, guided by a text description from the vision-language model. That clean image goes into Hunyuan3D-2, an image-to-3D diffusion model that generates a full textured mesh.
That mesh has no real-world size, so the system aligns it to the observed depth points using ICP, meaning Iterative Closest Point, an algorithm that scales and shifts one point cloud to best overlap another. It then estimates the starting pose with Foundation Pose, a pose estimator that matches the mesh to the RGB-D observation. Everything is loaded into Isaac Lab, a GPU-accelerated parallel physics simulator.
2. Evolve arrangements in simulation. The goal is an ordered list of desired goal poses, where order defines placement sequence and each pose defines position plus orientation in the world. Real execution is noisy: after the robot places an object, it settles to a slightly different final pose due to contacts or toppling. The authors call this place-then-settle transition S, and they want the sequence that maximizes expected task reward after settling.
They use an evolutionary algorithm because the space is mixed — discrete order plus continuous poses — and multimodal with many local optima. In plain terms: start with L random solutions, here 1024 parallel environments by default. For each, reset simulation, add small Gaussian placement noise, place objects one by one, let physics settle after each, then score. Keep the top 5 percent unchanged as elites, pick the rest with probability weighted toward higher scores, and mutate both poses and order. Repeat for a fixed number of generations.
Then a validation stage re-tests finalists over many noisy rollouts and picks the highest average score. More noise favors safe, robust solutions; less noise allows precise but risky ones. A concrete example from dominoes: without validation the system liked balancing a green bottle on top of a can to tip it — occasionally brilliant but fragile. With validation it preferred rolling the can down a book ramp — lower peak but far more reliable.
3. Execute on the real robot. Re-observe, re-estimate object poses against the generated meshes, then sample grasps from the complete mesh geometry by picking random surface points weighted by triangle area and orienting the gripper along the surface normal, discarding grasps in collision. Motion planning uses cuRobo, a CUDA-parallel collision-free planner. For precise stacking, the last centimeters use force feedback: move down until contact force spikes, then release, to avoid dropping early due to pose error.
Task scores are given, not learned from language in this paper. Stacking fixes horizontal position and optimizes order, height, and tilt to maximize tallest point. Cantilever fixes one axis and optimizes to maximize length extended from a weighed-down support box without touching the table — touching gives zero. Dominoes lets the robot place a tall bottle and heavy can anywhere in a 70 by 10 by 70 centimeter volume to maximize how far a bottom-heavy target bottle moves, allowing use of an immovable book pile as a ramp.
What they showed — results, honestly, with limitations
The evaluation asks five questions about effectiveness versus language-model reasoning, importance of 3D priors, real-world transfer, design choices, and mesh quality.
For spatial reasoning in simulation, they ran 10 fresh single-view captures per task per method and evaluated each proposed plan with 2048 noisy simulated placements, using the same meshes the method used to plan. Baselines were LLM-GROP, a zero-shot language-model task-and-motion planner given the reward function, constraints, annotated image, and object names, masses, and bounding boxes using GPT-5, and TSDF-EA, which uses only visible-surface reconstruction via Truncated Signed Distance Field fusion plus the same evolutionary search.
Simify scored higher on all three tasks. Mean task scores reported were 82.2 percent for stacking, 85.9 percent for cantilever, 94.3 percent for dominoes, for 87.5 percent mean. The language-model baseline did reasonably on stacking but failed reliably on cantilever and dominoes where precise continuous interaction matters. Partial TSDF geometry failed especially where unseen surfaces matter, like the bottom of a tin can that must roll down a ramp. This supports the claim that physics simulation as a reasoning engine beats pure semantic guessing for contact-rich tasks, though only on three tasks and 10 repeats each.
For real-robot execution over 10 trials per task, Simify made substantially more task progress than TSDF-EA. The failure-stage analysis is telling: all 10 TSDF-EA stacking trials failed at the planning stage because unstable plans were rejected by a simulation success threshold, while Simify reached later placement stages more often. A breakdown over 30 Simify trials shows remaining failures spread across planning, grasping, and placement — the paper does not claim perfect execution.
Ablations show search matters as much as geometry. Random sampling with good meshes scored only 10.3 percent mean, with zero on cantilever. Removing validation dropped mean from 87.5 to 64.0 percent. More parallel environments helped monotonically, especially on precise tasks, with 1024 environments taking about 73 seconds on average — practical but still over a minute of thinking per scene.
Mesh quality on YCB objects with ground truth showed generated meshes were slightly less accurate on visible surfaces than depth-based TSDF, but far more complete, with much better completeness distance, F1-score, and normal consistency, and more robust under high occlusion.
Limitations are stated clearly: mass, friction, and center of mass are assumed given with uniform density, and cantilever is sensitive to those guesses; objects are placed directly at goal poses in simulation without a simulated robot arm, so kinematic feasibility is only heuristically filtered; single-view image-to-3D inherits precision limits; the viewpoint is given rather than actively chosen; and reward functions are hand-specified.
Why this might matter to me
For spatial reasoning in LLMs and VLMs, this is a useful counterpoint: language models grasp task semantics but are weak at continuous 6D pose search under physics. Simify suggests a hybrid pattern you may see more often — let the VLM do open-vocabulary segmentation, object naming, and inpainting prompts, then offload precise spatial deliberation to simulation. Future work they propose, where a VLM sets constraints and initial guesses, maps directly onto agentic VLM planners.
For world models, this is an explicit, physics-grounded alternative to learned latent dynamics. Instead of predicting next pixels or latents, the world model here is a rebuilt mesh world plus a real rigid-body simulator with noisy rollouts for robustness. It is less general than learned models — rigid objects only, known friction — but it is interpretable, parallelizable, and zero-shot. The validation-for-robustness trick, optimizing expected reward under placement noise, is a principle you could carry into any world-model planner.
For text-to-CAD and parametric geometry, the lesson is about completeness versus accuracy. Hunyuan3D-2 meshes are not CAD-precise — TSDF actually wins on visible surfaces — yet watertight, plausible hidden surfaces enable stacking and rolling where partial scans fail. If you care about functional geometry, this argues for completing to solid, simulatable models and then refining critical faces, or swapping in multi-view or parametric CAD generators as they improve, which their modular pipeline explicitly allows.
For 3D generation and scene representation, the pipeline is a template for generative real-to-sim from minimal input: segment, inpaint, generate, scale with ICP, register with a pose estimator, simulate. The occlusion study and resolution trade-off — they use octree resolution 64 to balance speed and accuracy — show what matters downstream is not perceptual score but physical utility: normal consistency, bottom-surface closure, and stability under perturbation.
Terms worth knowing
Spatial reasoning: inferring object shapes, relations, and feasible placements in 3D to achieve a goal, here including whether the result stays stable or causes desired collisions. Real-to-sim / sim-to-real: building a simulator from real observations, then transferring plans found in simulation back to reality. Image-to-3D generative model: a diffusion-based model like Hunyuan3D-2 that hallucinates a full 3D mesh from one or few images, including unseen sides. Vision-Language Model and Large Language Model: foundation models that connect images and text or reason over text; used here for segmentation grouping, inpainting descriptions, and the LLM-GROP baseline. Evolutionary search: optimization by maintaining a population of candidates, keeping high-scoring elites, and mutating the rest over generations; suited to mixed discrete order plus continuous pose spaces. TSDF: Truncated Signed Distance Field, a volumetric fusion of depth data that reconstructs only observed surfaces, leaving occluded parts missing. ICP: Iterative Closest Point, alignment of two point clouds by iteratively minimizing distance; used here to recover metric scale. Foundation Pose / Isaac Lab / cuRobo: off-the-shelf pose estimator, GPU-parallel physics simulator, and parallel motion planner used to instantiate, test, and execute plans.