All summaries · 2026-10-16

DeSpatial: The devil of Agentic Spatial Reasoning Lies in 3D Itself

Yifei Liu, Xiaolong Zhou, Ziyang Gong, et al.

arXiv:2610.20990 · abstract · pdf · summarized by meta/muse-spark-1.3-contributor on 2026-10-16

The one-paragraph version

DeSpatial is a system for video-based 3D question answering that argues the hard part of agentic spatial reasoning is not clever tool use, but getting clean 3D objects in the first place. The authors show this with an oracle test: when they replace noisy perceived objects with ground-truth 3D boxes, the same small vision-language model and the same measurement tools jump from about 50% to about 98% on five object-focused question types in VSI-Bench. To close that gap without ground truth, they build one shared 3D perception tool that turns messy 2D masks and 3D reconstructions into a single metric scene where up is up and each physical object has one stable identity. That tool first cleans each frame by merging duplicate masks and cutting away wrongly projected background points, then links observations across frames by 3D voxel overlap while forbidding merges of two things seen as separate in the same frame, plus a visual check for broken-apart pieces of the same object. A separate reasoning agent then just measures and compares those already-clean objects to count, size, distance, direct, and plan. With Qwen3-VL-8B as the brain, this scores best-in-class among compared agents on VSI-Bench, MindCube-Tiny, ViewSpatial, and long-video VSI-SUPER.

The problem they're solving, and why it matters

Spatial understanding here means answering concrete questions from video like how many chairs there are, how tall a table is, how far the sofa is from the lamp, or whether the door is to the left of someone facing the TV. Videos make this hard because the same chair looks different from different angles, gets split by occluders like a table hiding a chair leg, and reappears many times.

Recent spatial agents — meaning a vision-language model, or VLM, that can call outside tools — already have access to powerful tools: segmentation for cutting out objects in 2D, meaning labeling which pixels belong to which thing; 3D reconstruction for estimating camera motion and dense depth and shape; and detection for finding object classes. The problem is those tools disagree. A 2D mask for a chair back may include sky or floor pixels because depth was estimated independently, so when the system lifts pixels into 3D by back-projection — meaning shooting each pixel out along its camera ray to its estimated distance — floor points get glued onto the chair and inflate its box. Across frames, the front and back of one table may be treated as two tables, or two nearby chairs may be merged into one. Once the object list is wrong, even perfect counting or measuring code gives the wrong answer.

This matters because many agent papers add more elaborate planning, programs, or map memories without fixing the underlying evidence. If the bottleneck really is perception consistency, more reasoning cannot help.

The key idea, in plain words

The key idea is to split the job in two and fix perception first: one shared perception tool owns all the reconciliation, and reasoning tools are only allowed to work on its finished product.

Think of it like a construction crew. Today, every electrician and plumber brings their own tape measure and argues about wall locations. DeSpatial hires one surveyor to produce one agreed floor plan — where every object has a persistent ID, a semantic class like chair or table, a 3D center, and an oriented bounding box, meaning a box that can rotate to fit the object with center, side lengths and axes, placed in real-world meters with the vertical axis aligned to gravity. The rest of the crew just reads that plan.

That surveyor does two cleanups. Inside each single image, it merges masks that are really the same thing and deletes 3D points that do not really belong. Across images, it links views of the same thing using shared 3D volume, but it refuses to link two things that were seen as distinct at the same moment, and it asks a visual judge to reunite far-apart fragments of one object when views never overlapped.

How it actually works, step by step

First, the system builds metric, gravity-aligned geometry — meaning 3D shape in meters where the z-axis points straight up. It runs reconstruction models to get cameras and dense points, rescales to approximate meters using Depth Anything 3, then estimates which way is up in each camera view with a tool called GeoCalib. In words, it converts each camera-local up guess into a common world direction, keeps the direction that agrees with the most other frames within 20 degrees, then jointly refines those agreeing views to one shared up vector. It then rotates and scales the whole scene so that up becomes +z.

Second, inside each frame, it does joint segmentation-depth co-refinement. For consolidation, take the example of two overlapping sofa masks from an open-vocabulary segmenter like SAM 3. If their overlap is more than half of the smaller mask, unite them. For occlusion, take the example of a chair split into two visible blobs by a table in front. The system checks whether their horizontal footprints overlap in 3D and then shows a VLM judge the image with parts labeled A and B and asks: same physical object considering structure, occluder, appearance and context? Only a yes merges them and records that this mask has, say, two confirmed parts.

Then it filters bad geometry. For each consolidated mask, it keeps only pixels that pass three ideas expressed before any formula: the reconstructor is confident about that point, the point is not on a sharp depth edge where foreground and background mix, and the point is not a global statistical outlier in 3D. Even after that, a coherent patch of floor seen through chair slats can survive. So it keeps only connected lumps in 3D — meaning points touching each other in space — retaining only the largest lump, or the largest k lumps if k parts were confirmed in the previous step.

Third, across frames, it does cross-view instance fusion. Each cleaned observation becomes a node with its class, source frame, and points. To decide if two views show the same chair, it voxelizes space into small cubes, here 0.1 meter cubes, and measures shared occupancy. In words, the score is the number of cubes occupied by both divided by the number in the smaller of the two, which tolerates one view seeing only part. Edges above 0.1 are considered in strongest-first order. Critically, it enforces same-frame exclusion: if frame 5 already saw A and B as separate chairs, A and B can never end up in the same global object even via a chain through frame 8. This protects precision.

For truly disjoint fragments — for example only the tabletop seen early and only a leg seen much later with no shared cubes — it retrieves nearby same-class candidates by 3D surface distance and by horizontal footprint distance, then asks a VLM identity judge with up to three masked views per fragment plus top-down and oblique 3D renderings. The judge must answer same, different, or unclear based on compatible position and orientation plus matching materials and details, without moving the fragments to force a fit. Merging two groups requires an explicit same decision for every cross-pair.

Finally, reasoning happens on that finished scene. The Qwen3-VL-8B agent calls structured tools to count instances, measure box sizes, compute distances from box geometry, get left-right-front-back in an observer frame, plan routes, and find appearance order from source frames. If a question needs facing direction or an observer pose, that is estimated and passed in.

What they showed — results, honestly, with limitations

The oracle diagnostic is striking. On the same 4,030 VSI-Bench questions across object counting, absolute distance, object size, relative distance, and relative direction, raw perception built by lifting SAM 3 masks with Pi3 geometry averages 50.38%, while ground-truth oriented boxes with the same reasoning tools average 98.23%, with each category above 96%. The authors themselves caution this should not be over-interpreted because VSI-Bench annotations are imperfect and match the answers by construction, so it measures compatibility with annotations, not perfect physics. Still, it strongly points to perception as the bottleneck for those five categories.

With real perception, DeSpatial with Qwen3-VL-8B averages 69.22 across all eight VSI-Bench categories, including counting 65.43, absolute distance 55.08, size 66.86, room size 73.58, relative distance 67.93, relative direction 82.53, route planning 64.43 and appearance order 77.90. That is above the listed training-free and agentic baselines in the paper. On MindCube-Tiny it reaches 87.19 overall, and on ViewSpatial 68.35, again leading the compared table entries. On long-video VSI-SUPER it holds recall around 86-90% from 10 to 240 minutes and improves counting at longer horizons over S-Agent, though absolute counting scores remain modest at 33.4 to 53.4.

Ablations on a fixed 504-question subset show each cleaning step helps: geometry filtering adds about 5.3 points over raw lifting, keeping the largest component adds about 2.3 more, and mask consolidation plus part-aware keeping adds another 3.6 to reach 68.83. Voxel overlap at 64.30 to 68.83 beats bounding-box overlap at 59.90 and density clustering at 61.37, same-frame exclusion adds 2.52 points, and visual fragment reunion adds 2.01. For gravity, human judgment of projected up arrows on 156 scenes rates GeoCalib joint calibration correct on 156 of 156 versus 68.59% for a plane-fitting baseline.

Limitations are stated clearly: the whole pipeline depends on detections, reconstruction, metric scale and gravity; it cannot recover surfaces never seen or objects never detected; wrong part counts or wrong visual merges can delete good geometry or fuse different objects; and the near-perfect oracle claim covers only five object-centric categories.

Why this might matter to me

For spatial reasoning in LLMs and VLMs, the lesson is to treat consistent object memory as a first-class module rather than hoping chain-of-thought or more tools will resolve conflicting masks and depths. The same-frame exclusion trick and overlap normalized by the smaller observation are directly reusable patterns for preventing double-counting when an object reappears.

For world models, meaning internal models that predict how a scene looks and behaves from new viewpoints, DeSpatial suggests the world state should be an explicit, gravity-aligned object list with provenance to source frames, not just latent features. That makes left-right, distance and size queries auditable.

For text-to-CAD and parametric geometry, where you want editable boxes, extents and placements from language or video, the pipeline from dirty pixels to clean oriented boxes with metric scale is the missing front end. Its failure modes — inflated boxes from edge bleed and split parts from occlusion — are exactly what break parametric fitting.

For 3D generation and scene representation, the message is that voxel-overlap association plus appearance-verified fragment linking gives more stable identities than box-overlap or centroid clustering under partial views, which should improve persistent neural fields, Gaussian scenes, or object-centric maps built from video.

Terms worth knowing