The one-paragraph version
This paper asks why vision-language models keep getting video-based 3D questions wrong, and then fixes the most repeatable mistakes without retraining the model. The authors first run a controlled diagnosis: they give the same model the same video question three ways — video alone, video plus machine-estimated 3D clues, and video plus perfect 3D clues from dataset annotations — all written in one shared format. That comparison shows four recurring problems: bad perception, perfect-looking clues that are still missing something essential, picking the wrong geometric measurement, and using the wrong coordinate frame or outdated position. From those failures they build CROSS, a training-free library of small, checked geometric operations plus task-specific recipes that chain those operations together. CROSS can either add a verified calculation to the prompt for a chat-style model, or be called as code tools by an agent. On the ReVSI video benchmark the prompt version lifts the average from 55.9% to 60.2%, and on DSI-Bench the same operations lift an agent called SpatialClaw from 62.8% to 66.3%.
The problem they're solving, and why it matters
Video spatial benchmarks test things like how big an object is, how far apart two objects are, whether something is to your left or right, and how to turn along a route. A single low score on one of those tasks does not tell you what broke.
The paper argues that every such question needs three different capabilities. First, recovery of evidence: finding the relevant objects, room size, and camera path from video. Second, representation of geometry: storing that evidence in a form that can actually answer the question. Third, reasoning over geometry: doing the right calculation in the right reference frame.
Today those three are blurred together as a single “spatial reasoning error.” That matters because the fix is completely different in each case. If the model never saw the chair, you need better perception. If it saw the chair but measured center-to-center distance when the question asked for closest-surface gap, you need better measurement choice. If it measured correctly but in the camera’s frame instead of the person’s frame, you need better frame handling. The authors want to separate what evidence was available from how it was used, so repairs can be targeted and reused.
The key idea, in plain words
Don’t ask a large model to reinvent geometry from scratch on every question. Diagnose the repeatable geometric mistakes once, then give the model small, typed, convention-respecting calculators for those exact steps.
A concrete example from the paper makes this clear. The question is: standing at the nightstand and facing the TV, is the cooktop to your left, right, or behind? A video-only model talks about the kitchen being on the right side of the room and answers “right.” That substitutes room-level left-right for observer-relative left-right. What you actually need to do is place the observer at the nightstand, point forward toward the TV, put up, right, and forward axes around that observer, project the cooktop into that observer frame, and read off the signed angle. In this case the computed bearing is about +58 degrees, where positive means left, so the answer is “left.” CROSS makes that three-step procedure — build observer frame, project, classify — an explicit, reusable skill instead of hoping the model’s chain of thought gets handedness right.
How it actually works, step by step
Step 1: Put all evidence in one shared box. The authors define structured spatial context as an externally built scene description, not model-generated reasoning. It has four parts: a coordinate contract that declares units, axis directions, handedness, and how camera poses transform points; global geometry such as room length, width, and height; viewer trajectory as timestamped camera positions and poses; and entity observations where each persistent object gets a category, 3D center, physical size, and orientation. Ground-truth context fills this from annotations. Predicted context fills the same slots from tools: SAM3 for object masks and tracks, DA3 for metric depth and camera poses. Only the source accuracy differs.
Step 2: Compare three evidence levels. Video-only shows what the model recovers alone. Predicted-context shows the penalty of imperfect estimation. Ground-truth-context shows the reasoning residual — what still fails even when the evidence is correct, on examples where the answer is in principle recoverable from the supplied context.
Step 3: Name the four failure types. Inaccurate perception is the gap between predicted and ground-truth context. Missing information means even perfect context omits what is needed, for example no floor polygon for room area so the model multiplies a whole-scene bounding box, or only 39 stored poses out of 1,155 real poses so path length is underestimated by using short chords. Wrong measurement means using the wrong quantity, for example 225 of 227 ground-truth absolute-distance errors match center distance within 5% when closest-surface gap was asked. Wrong frame or state means left-right mirror, reversing “facing away,” forgetting to rotate an ego-frame displacement into the world frame for pose estimation, or reusing the start heading during route planning.
Step 4: Replace fragile rediscovery with operators and skills. CROSS provides atomic operators grouped into four families: frame operations to build bases and move poses between frames; geometry and shape operations to aggregate boxes and measure surface gaps, oriented extents, and floor area; set and decision operations to deduplicate instances, count, and match an answer only when the margin exceeds estimated error; and temporal and state operations to integrate paths, separate endpoint displacement from total path length, classify local left-right turns, and update position and heading after each step.
A skill is a fixed ordered chain of those operators for one task. Surface distance aggregates reliable boxes then takes the minimum boundary gap. Egocentric direction builds the observer basis then classifies the sector. Pose estimation composes camera motion with the question’s initial pose in one common rigid-motion frame before comparing candidates. Route planning classifies each turn then advances the agent’s heading before the next turn.
Step 5: Gate, filter, or fall back. Every operator reports value plus unit, frame, estimated error, and validity codes. A deterministic safeguard decides to apply the derivation and keep only question-relevant entities and poses, or to abstain and fall back to video-only answering. For example, it falls back if a role phrase cannot be bound, geometry is degenerate, a direction is within 5 degrees of a sector boundary, uncertainty is too high, or no floor footprint exists for an area question. Counting keeps all entities so the filtered list does not give away the answer. For a code-writing agent, the same skills and operators are exposed as callable Python tools.
What they showed — results, honestly, with limitations
Diagnosis used two compact subsets with fixed frames and answer parsers: REVSI-TINY with 823 questions and 32 frames, and RESTI-TINY with 775 questions and 30 frames, both evaluated with Qwen3-VL-32B-Thinking.
Ground-truth context helps a lot but does not solve reasoning. On REVSI-TINY the average rises from 55.9% video-only to 76.2% with ground-truth context, with counting 58.2% to 93.1% and object size 67.9% to 100.0%. But absolute distance only reaches 58.4%, relative direction 52.8%, and route planning actually drops from 36.8% to 29.8%. The authors show these are systematic: almost all absolute-distance failures equal center distance, many route errors differ only by left-right swaps, and pose estimation largely recovers only after correct frame composition. Path length versus endpoint displacement is revealing: endpoint displacement hits 99.2% while path length is only 19.7%, because sparse stored poses preserve endpoints but lose motion between them.
CROSS improves reasoning over the same ground-truth context: REVSI-TINY 76.2% to 92.3%, led by absolute distance +41.6 points to 100.0% and relative direction +47.2 points to 100.0%, plus route planning +19.3 points and pose estimation +28.2 points on RESTI-TINY. It also helps with noisy predicted context and as agent tools, but gains are not uniform. Some tasks do not improve — object size, relative distance, and room size on ground-truth context — and predicted context alone can hurt: on REVSI-TINY it lowers the average from 55.9% to 50.0%, mainly on counting, distance, and size, while on RESTI-TINY it raises the average from 21.3% to 34.3%, suggesting the predictor recovers trajectory better than object identity.
On larger tests, ReVSI average goes 55.9% direct and 52.2% predicted-context to 60.2% with CROSS, and SpatialClaw with Gemma-4 goes 55.6% to 58.9%. VSI-Bench goes 59.8% to 61.4%, STI-Bench 41.6% to 45.3%, and full DSI-Bench with frozen operators goes 62.8% reproduced SpatialClaw to 66.3% with CROSS.
Limitations are stated plainly. The ground-truth intervention shows usefulness of accurate evidence, not independent causal proof for perception versus reasoning. Predicted perception is still limited by open-vocabulary recall, monocular scale, and rotation drift with median gravity tilt 9.2 degrees. The gate is heuristic and tuned on a scene-disjoint development split; its room-scale assumptions may not transfer outdoors. Operators do not cover acceleration or appearance order. Deterministic precomputation enforces conventions but does not prove the base model learned spatial algorithms. Cross-system comparisons differ in backbone and frame budget and are descriptive, and tables report point estimates without significance tests.
Why this might matter to me
For spatial reasoning in LLMs and VLMs, this offers a useful split: stop treating every wrong answer as a reasoning failure and first ask whether the evidence was present, sufficient, and then correctly framed. The four-way taxonomy — perception, missing representation, wrong quantity, wrong frame or state — maps directly onto where to intervene with data versus code.
For world models, the lesson is that trajectory density and state updating matter as much as single-frame geometry. Endpoint position can be perfect while path length, heading, and turn sequence are wrong if motion between samples is discarded and heading is never advanced. Explicit advance-state and path-integral steps are a reminder to persist pose and heading, not just positions.
For text-to-CAD and parametric geometry, the center-versus-surface example is central. Language like “closest point,” “clearance,” “extent along an axis,” “floor polygon versus bounding-box area,” and “frame in which a pose is expressed” has exact geometric meaning. A typed operator library with declared units, frames, and uncertainty is much closer to how CAD kernels and constraints behave than free-form chain of thought. The paper’s validity codes and margin checks resemble sketch-solver failure modes: under-constrained, ambiguous binding, or tolerance larger than option spacing.
For 3D generation and scene representation, the shared schema is the transferable part: coordinate contract plus global extent plus timestamped trajectory plus oriented entities. That is enough to test whether a generated scene or reconstructed memory can support measurement, not just rendering. The cross-benchmark reuse — same operators recomposed for DSI-Bench — suggests building a small, verified measurement layer on top of whatever neural reconstructor you use, rather than retraining for each new question type.
Terms worth knowing
Vision-language model (VLM): a model that takes images or video plus text and produces text answers. Here it must answer metric 3D questions from video.
Spatial context augmentation: supplying an extra structured 3D description alongside video and question, as opposed to relying on video alone.
Ground-truth versus predicted context: the same schema filled either from dataset annotations or from perception tools. Their performance gap is called the perception penalty.
Coordinate contract: an explicit declaration of origin, axis directions, up direction, units in meters, and how a camera pose matrix maps points from one frame to another. Mismatching this causes many reported errors.
Ego-anchored, gravity-aligned frame: the paper’s common frame where the first camera center is the origin, up is +Z, forward is the horizontal projection of the first view, and +X completes the axes. All boxes and poses are converted into it.
Closest-surface gap versus center distance: the distance between object boundaries after accounting for size and orientation, versus distance between box centers. Models often substitute the easier center distance.
Reference frame and spatial state: whose viewpoint a direction is judged from and what the current position and heading are. Errors include left-right reversal, wrong rotation when moving between ego and world frames, and reusing an old pose.
Operator versus skill in CROSS: an operator is one typed calculation like measure surface distance or build observer basis; a skill is a fixed chain of operators for one task like relative direction or route planning, plus gating and filtering.
Gating and filtering: checks that reject an unreliable derivation and fall back to video-only, and if accepted, keep only relevant entities, poses, and the derived value with its uncertainty and role bindings.