The one-paragraph version
GraFT is a way to make a frozen multimodal large language model — meaning a model that takes both images and text and whose weights are never updated — much better at answering questions about 3D indoor space without any extra training. It first builds, once per scene, a compact 3D scene graph from a walk-through video: a hierarchy of building, floors, rooms, and objects where each object stores its semantic label and its 3D box geometry plus links to the video frames that saw it. For each new question, GraFT picks the kind of evidence that question needs from that same graph: exact numbers are computed by deterministic calculator functions over the boxes, layout and direction questions get a clean top-down drawing made only from the relevant boxes and rotated so “forward” points up, and questions about appearance get the two or three first-person frames that show the queried object most clearly, ranked by geometry. On VSI-Bench, a video-based 3D spatial reasoning benchmark, this lifts frozen open models by 37% to 65% and reaches 51.4 average, ahead of the best proprietary model tested at 45.4 and several models that were explicitly fine-tuned for space; on ScanQA, an open-ended 3D question-answering benchmark, it raises CIDEr, a phrase-overlap caption score, from 58.0 to 73.6 on the same backbone.
The problem they're solving, and why it matters
Multimodal large language models, often shortened to MLLMs, are good at describing images in words but consistently unreliable at three spatial jobs: precise geometric measurement like “how wide is this table in meters?”, converting between viewpoints like “if you stand at the sofa facing the TV, what is to your left?”, and grounding fine appearance like “what color and material is that chair in the corner?”. The usual fixes build the spatial skill into the model itself, either by supervised fine-tuning or reinforcement learning on large curated spatial datasets, or by attaching a dedicated 3D encoder that reads point clouds — the raw sets of 3D points from a depth sensor — into the language model. The authors argue both paths are costly and inflexible: they require data curation and training, they tie the solution to one backbone and one input format, and they commit in advance to a single strategy. Yet different questions want different evidence. A global map helps layout, a well-chosen photo helps appearance, and a calculator helps exact distance. Getting this wrong matters for augmented reality, robotics, autonomous driving, and embodied assistants, where misjudging size, distance, or left-versus-right is not a stylistic error but a physical one.
The key idea, in plain words
Instead of teaching the model geometry, give it the geometry to read. The robotics community already maintains a representation called a 3D scene graph, or 3DSG for short: not a messy point cloud, but a tidy, incrementally maintainable list of what objects are where, with each object reduced to an oriented box — meaning a box with a position, a size along its own axes, and a rotation — plus a label like “sofa” and pointers to frames that observed it. GraFT treats that one graph as a shared middleman. It runs perception once to build the graph, then amortizes, meaning spreads, that cost across all future questions. Then it matches the tool to the task rather than using one rendering for everything: if the answer follows deterministically from box geometry, calculate it and don’t let the language model guess numbers from pixels; if the question is about arrangement, draw an uncluttered, to-scale top-down map containing only the mentioned objects; if it’s about what something looks like, return to pixels but only the clearest views. The frozen backbone then does what it is already good at: calling a tool and reading structured images and text.
How it actually works, step by step
Step 0: Build the graph once and treat it as given. An upstream pipeline turns the input posed video — meaning every frame comes with known camera position, orientation, and lens parameters — into a hierarchical graph with four layers: building contains floors, floors contain rooms, rooms contain objects. This decouples, meaning separates, reasoning from perception: GraFT does not care exactly how you built the graph, only that each object node carries its class label, its box center in world meters, its full width-depth-height extents, its orientation recovered from a stored rotation, a surface direction, and the subset of frames its box projects into.
Step 1: Route the question to one module. The frozen model performs tool selection, choosing among symbolic calculation, top-down rendering, or frame retrieval. The paper calls this a canonical tool-selection step; how the prompt or classifier makes that choice for arbitrary questions is not fully specified, so this routing step is somewhat unclear.
Step 2a: Symbolic module for metric answers. For counting, object size, absolute distance, relative “which is nearer,” and room size, GraFT never asks the language model to do arithmetic from text or images. It runs one deterministic function directly over the stored boxes: a count tool that tallies instances per class, a size tool that returns width, depth, height, a distance tool that measures surface-to-surface gap between two boxes rather than center-to-center, a proximity tool that ranks candidates by closest distance to an anchor, and a room tool that returns width, depth, height and floor area. It makes exactly one call, never chaining tools, so no numerical reasoning passes through the backbone. In plain terms: if you want the distance between a chair and a table, measure the boxes, don’t estimate from a photo.
Step 2b: Bird’s-eye-view module for egocentric-to-allocentric reasoning. Here egocentric means first-person, what you would see standing inside the scene; allocentric means map-like, independent of where you stand; bird’s-eye-view, or BEV, means a top-down orthographic projection. For a direction question like “facing the fireplace from the sofa, where is the stove?”, GraFT draws only the task-relevant objects as filled footprints with their exact proportions, omitting all other scene clutter, unlike prior work that marks objects on top of a full reconstruction. It then rotates the whole drawing about the observer’s position so the facing direction points up, so image top always means front, bottom means back, left means left, right means right, and optionally adds a dashed anchor-to-target arrow plus a textual list of observer-frame coordinates to cross-reference.
Step 2c: Egocentric module for visual attributes. For “what does it look like?” questions, GraFT scores every frame linked to the target object. First a visibility gate keeps only frames that pass three geometric tests together: at least two of the box’s eight corners project inside the image, the camera-to-object distance is within an informative band fixed at 0.5 to 4 meters, and the camera is actually facing the object within a fixed angle. Survivors are then scored as a product of three terms in words: how close the camera is, how centered the object is relative to image center, and how completely it is framed as measured by fraction of corners inside. Because it is a product, failing any one — for example a close centered view that clips three-quarters of the table — drags the score down. Frames are ranked, near-duplicates close in time are skipped for diversity, and only the top-k, two in the main ScanQA experiments, are forwarded.
What they showed — results, honestly, with limitations
The authors isolate each capability under the same backbone before testing the composed system, in both ground-truth perception where the graph comes from human annotations and realistic perception where it comes from automatic reconstruction plus Mask3D and UniDet3D detectors.
Symbolic tools beat letting the model reason over the graph serialized as text on all five metric VSI-Bench subtasks, and crucially make a weak model nearly match a strong one: under ground-truth perception the weaker Qwen2.5-VL-7B trailed GPT-o3 by 33.9 points when doing in-context arithmetic but by only 2.9 points when using the calculators, rising from 55.4 to 92.3 average. The BEV drawing beats raw video by 35.2 points ground-truth and 15.8 points realistic on direction tasks, and beats prior selective renderings like Struct2D and GPT4Scene by 3-4 points. On ScanQA the geometry-ranked top-2 frames improve every caption metric over uniform sampling on the same Qwen backbone, with BLEU-1 from 22.2 to 34.2 and CIDEr from 58.0 to 73.6, even beating a fine-tuned 3D model, and the appendix shows even one selected frame beats eight uniform frames, so selection, not count, drives the gain.
Composed on full VSI-Bench under realistic perception, GraFT with InternVL2-8B reaches 51.4 average, rank 1 in their table, ahead of every general-purpose open baseline — best 40.9 for a 72B LLaVA variant — and every proprietary model — best 45.4 for Gemini-1.5 Pro — and ahead of several fine-tuned spatial models by 0.7 to 6.0 points. A 3B GraFT at 47.5 beats the strongest 72B general baseline.
Limitations are plain. Accuracy is bounded by graph quality: the same system drops from 95.2 to 58.0 average on metric tasks going from ground-truth to realistic perception. It is not uniformly best: on route planning under realistic perception it trails Struct2D, for example 71.4 versus 76.2 with GPT-o3 and 28.9 versus 35.8 with Qwen, and on some counting subtasks it also trails. Evaluation is indoor, static scenes from ScanNet, ScanNet++ and ARKitScenes only, with no test of dynamic or outdoor scenes, and “training-free” still requires a heavy offline perception pipeline whose failures propagate.
Why this might matter to me
For spatial reasoning in LLMs and VLMs, this is evidence that much of the gap is an evidence problem, not a reasoning problem: a small frozen model plus exact calculators and a clean map can beat a giant model guessing from video, and the per-module ablations show you must route each question to the right evidence — tools win numbers, BEV wins direction, retrieval wins appearance order.
For world models, the 3DSG offers a pragmatic middle ground between raw video prediction and full physics simulation: a compact, human-readable, maintainable state — objects as boxes with labels and poses — that is sufficient for counting, distance, direction, and grounding, and that persists across questions without re-running perception.
For text-to-CAD and parametric geometry, the lesson is to calculate, don’t hallucinate: sizes, surface-to-surface distances, and room areas should come from deterministic functions over parametric boxes, and the paper’s trick of making the language model read numbers and maps rather than produce them maps directly to generating extrusions, placements, and constraints from exact solvers.
For 3D generation and scene representation, the selective BEV and visibility score are reusable patterns: render only task-relevant footprints to scale with heading alignment instead of cluttered full-scene renders, and rank generated or retrieved views by proximity times centrality times coverage to automatically pick canonical informative views for inspection or conditioning.
Terms worth knowing
Multimodal large language model (MLLM): a large language model that also accepts images or video alongside text, used here frozen without weight updates.
3D scene graph (3DSG): a structured map where nodes are building, floors, rooms, and objects with containment edges; each object stores semantics plus 3D box geometry and links to observing frames, as opposed to raw points.
Oriented bounding box: a tight 3D box around an object defined in words by its center position, its three side lengths, and its rotation; all metric tools read these boxes.
Egocentric versus allocentric: first-person view from inside the scene versus map-like view independent of observer position; spatial questions often require converting the former to the latter.
Bird’s-eye-view (BEV) rendering: a top-down drawing of box footprints used here as structured visual prompting the model already knows how to read.
Symbolic tool: a deterministic program that computes an answer like count or distance exactly from stored geometry rather than estimating via neural inference.
Serialized graph: a baseline where the graph is dumped as text numbers and the language model itself does the arithmetic, which the paper shows is less accurate than calling calculators.