The one-paragraph version
Vision-Language Models are good at naming what is in a video but poor at measuring it — how far apart two chairs are, how big a table is, how large a room is, or whether a chair seen now is the same chair seen ten seconds ago. This paper proposes Geometric Code, a training-free add-on that does the measuring outside the model and then hands the results to the model as text. It runs ordinary video frames through two frozen perception models for object masks and for depth plus camera motion, fuses those outputs into cleaned 3D point clouds in a room-aligned coordinate system, merges repeated sightings into single objects, precomputes answers like distances and sizes, and pastes that summary into the prompt as structured JSON. With no training at all, adding this code to 32 video frames lifts average accuracy on the video spatial benchmark called VSI-Bench from 53.1 to 57.1 across four small open models, with the biggest wins on metric tasks like absolute distance.
The problem they're solving, and why it matters
The problem is a gap between semantic understanding and geometric understanding. Semantic understanding means recognizing categories and rough relations, for example "there is a TV and a table." Geometric understanding means metric and persistent structure, for example "the front edge of the TV is 1.18 meters from the table edge" or "the stool in frame 1 is the same physical stool in frame 19 despite camera motion."
The authors argue, building on VSI-Bench and follow-up analyses, that current Vision-Language Models, meaning AI models that take images or video plus language and generate language answers, concentrate their failures in this second kind. They can guess what is present and roughly where, but they cannot reliably estimate distances, dimensions, or keep object identity consistent across frames, and they collapse when forced to take another viewpoint. Bigger models narrow the gap but the metric weakness persists.
That matters because navigation, manipulation, layout planning, and any assistant that talks about physical space need numbers that can be trusted and inspected, not just plausible descriptions. Prior fixes either required extra training on millions of synthetic spatial questions, changed the model architecture to ingest depth or 3D directly, or learned spatial reasoning with curricula and rewards. Those approaches work but leave the knowledge implicit in weights, meaning hidden inside learned parameters where you cannot read or reuse it, and they cost training.
The key idea, in plain words
Do not teach the model to perceive geometry from pixels. Compute the geometry with classical tools and let the model read it.
Think of it like giving a student a word problem plus a measurement sheet. The student still reasons, but does not have to eyeball distances from blurry photos. Here the measurement sheet is called spatial code: a compact text summary listing per-object positions, extents, counts, pairwise distances and rankings, size estimates, room area and outline, appearance order, and sampled camera waypoints.
Crucially, nothing is learned in this pipeline. The segmentation and depth models are used frozen, meaning their weights are not updated, and the fusion step is deterministic, meaning the same inputs always give the same outputs through fixed rules rather than learned guesses. The code can be used alone, replacing the video, or alongside the video. The paper finds it works best as a complement, not a replacement.
How it actually works, step by step
1. Perception layer: masks plus depth and camera. The input is monocular RGB video, meaning ordinary color video from one camera with no lidar or stereo.
For objects, the system uses SAM-3, short for Segment Anything Model 3, applied about 6 times per second with a fixed open-vocabulary list. Open-vocabulary here means the model can be asked to find arbitrary named categories like "TV, table, stool," where the list is built once as the union of all categories in that dataset split, not tuned per scene. SAM-3 returns an instance-level binary mask per detection, meaning a pixel-accurate silhouette for each separate object, even multiple objects of the same kind. Masks are preferred over rectangles because rectangles include background pixels that would corrupt later depth averaging. Each frame is segmented independently, so there is no assumption that object number 2 in one frame is the same as object number 2 in the next.
For geometry, the system uses DA-3, short for Depth Anything 3, to estimate per-frame forward depth, camera intrinsics, camera pose, and confidence. Forward depth means distance along the camera's looking direction, not straight-line distance to the lens. Camera intrinsics means internal camera properties like focal length and image center that map pixels to rays. Camera pose means where the camera is and which way it points. DA-3 looks at patches across frames jointly and is run in streaming chunks of 120 frames with 60 frames of overlap to keep estimates consistent.
2. Geometric engine: from pixels to a shared 3D room. First is back-projection, meaning converting each masked 2D pixel plus its depth into a 3D point in front of the camera using the intrinsics, then moving that point into a single shared world coordinate system using that frame's camera rotation and translation. The result per detection is an object-level point cloud, meaning a set of 3D points sampling the visible surface. Outliers are removed by dropping points that are unusually far from their neighbors compared to the cloud average.
Next is canonical frame, meaning a room-aligned coordinate system where up is really up and the floor is level. The engine finds the floor by sampling confident points and testing about 300 candidate planes with RANSAC, short for Random Sample Consensus, a robust fitting method that tries many random hypotheses and keeps the best. The floor is defined as large and flat with almost nothing below it, scored by how many points lie within a 5-centimeter slab times how many remaining points lie mostly on one side. If that fails, it falls back to the thinnest axis of the scene. Horizontal axes are then the original world axes projected flat.
Then comes instance assembly, meaning deciding which detections across time are the same physical object. Each detection is summarized by a robust 3D box using the 2nd to 98th percentiles to ignore extremes. Two detections of the same class can merge only if their volumes overlap and they never appear together in the same frame — if you see two stools at once, they cannot be one stool. Merging is checked at the group level to avoid chains where A merges with B and A merges with C, but B and C were seen together and so must stay separate.
From those consolidated instances the engine precomputes task quantities. Absolute distance is closest surface-to-surface distance between point clouds, not center-to-center. Relative distance is a full pairwise matrix plus a ranked nearest-to-farthest list so the model does not have to compare numbers itself. Object size uses the single best-observed frame, aligns vertical to gravity and horizontal axes to the object via principal component analysis, meaning finding the object's dominant directions, then reports the longest span. Room size projects floor points down, trims outliers, and fits an alpha shape, meaning a tight outline that can follow concavities, with grid-based cleanup for the outline. Count is just number of instances per class. Appearance order is order of first sighting. Route planning is not solved directly; the engine emits camera positions and headings sampled about every 5 seconds plus object and room layout, and leaves route inference to the model.
3. Prompting: geometry as language. All of this is serialized as JSON with a short legend and units written in the values, like "1.18 meters." For numeric questions the model must output a short phrase with greedy decoding and no extended reasoning. For multiple-choice it gets a longer reasoning budget and a fixed suffix, "The correct option is:", to force a commitment.
What they showed — results, honestly, with limitations
The tests use VSI-Bench, a video spatial benchmark with eight tasks: object count, absolute distance, object size, room size, relative distance, relative direction, route planning, and appearance order. They test four open models — Qwen3.5 in 2B and 4B sizes and InternVL3.5 in 2B and 4B sizes — under three inputs: frames only with 32 sampled frames, code only, and frames plus code.
Frames plus code is consistently best, but gains vary widely: +9.2 points for Qwen3.5-2B from 45.1 to 54.3, +2.1 for Qwen3.5-4B from 57.4 to 59.5, +0.3 for InternVL3.5-2B from 53.1 to 53.4, and +4.1 for InternVL3.5-4B from 56.9 to 61.0. That last configuration is the paper's best and is presented as surpassing prior results at comparable scale. Averaged across the four models, frames plus code moves mean accuracy from 53.1 to 57.1. This is worth stating plainly because the abstract says "+4.1 points" on average and highlights "+24.1 points" for absolute distance; the body shows +3.9 on average and does not make the derivation of 24.1 fully transparent, so the exact averaging behind the abstract claim is unclear.
The pattern by task is clear and honest. Code alone transforms absolute distance, rising from roughly 34 to 43 with frames only to roughly 65 to 67 with code only across models, and also helps room size. But code alone hurts object count, falling from roughly 58 to 70 with frames to 41.4 for all models with code only, and generally hurts object size and appearance order. In other words, precomputed geometry answers measurement questions well, while recognition-heavy questions still need pixels. Split by format, code alone helps numerical estimation slightly but hurts multiple-choice, whereas frames plus code helps both.
Limitations are substantial. Counting is recall-bound: if segmentation misses an object in every frame it can never be counted, so errors skew toward undercounting. Object size often declines with code, suggesting single-view estimates are fragile under occlusion. Route planning remains far below human level — around 38 compared to human 95.8 — and relative direction and route tasks are model-dependent. For precomputed tasks, performance is nearly identical across models, which means the score reflects the engine's quality more than the model's reasoning. Comparability to prior work is approximate due to different evaluation setups, and human average of 79.2 is still far above 61.0.
Why this might matter to me
For spatial reasoning in LLMs and VLMs, this is evidence that small models do not need to learn surveying from scratch: an explicit, inspectable measurement sheet recovers much of the metric competence they cannot extract from pixels, while pixels remain essential for grounding what is what. The division is practical — use deterministic code for numbers, keep vision for semantics — and it suggests future work should train models specifically to read such code, for example with lightweight adaptation, rather than to hallucinate distances.
For world models, which aim to represent how scenes look, move, and evolve, Geometric Code already provides exactly the scaffold those models often lack: object-level positions and extents, room footprint, and camera trajectory in a shared frame. The paper explicitly proposes coupling this explicit interface with predictive models to reason about future states and interactions. The open challenge they note is alignment, because most world models keep space implicit in latent videos or generative features; bridging that to an explicit, text-readable map is unsolved but valuable for planning and verification.
For text-to-CAD and parametric geometry, the approach is a template: segment, lift to 3D with metric depth, rectify to gravity and floor, robustly trim percentiles, fit orientations and outlines, and emit parameters with units. Replacing the final question-answering with extrusion, wall loops, or dimension constraints would turn the same pipeline into a rough reverse-engineering front end, where the JSON becomes sketch parameters rather than hints. The failure modes transfer too — missed detections become missing parts, and single-view size estimates need multi-view fusion.
For 3D generation and scene representation, this argues for hybrid representations: keep neural fields or Gaussians for appearance, but maintain a parallel symbolic layer of instances, boxes, distances, and room polygons that language models can query directly. That layer is debuggable, editable, and cheap to pass as context, and here it lets a 2B model jump by 9 points without retraining.
Terms worth knowing
Vision-Language Model: a model that takes images or video plus text and produces text, for example answering questions about a video. VSI-Bench: a benchmark of video spatial questions covering counting, distances, sizes, room size, directions, routes, and order. Spatial code / Geometric Code: the paper's JSON text summary of computed 3D facts supplied as extra prompt context. SAM-3: a frozen segmentation model used here to produce per-object pixel masks from a named category list. DA-3: a frozen depth model used here to estimate per-pixel depth, camera internals and motion, and confidence. Back-projection: turning a 2D pixel plus depth into a 3D point using camera properties. Point cloud: a set of 3D points sampling surfaces, here one cloud per object detection. Camera pose and intrinsics: pose is where the camera is and where it looks; intrinsics are internal mappings like focal length from pixels to rays. RANSAC: a robust fitting procedure that tests many random hypotheses and keeps the one with most support, here used to find the floor plane. Instance assembly: merging repeated detections of the same physical object across frames while respecting that two objects seen together cannot be one. Alpha shape: a method for tracing a tight, possibly concave outline around a set of points, here used for floor area.