The one-paragraph version
SpatialChain is a dataset and evaluation method for checking whether thinking-enabled vision-language models — models that take both an image and text as input and produce both intermediate reasoning text and a final answer — actually do spatial reasoning or just guess the right answer from language habits. The authors start from GQA, an existing collection of images, questions, and human-made scene graphs, and keep only questions that need spatial understanding, such as “is there a fence to the left of the person?” Using Claude Haiku 4.5 with access to the scene graph, they generate a step-by-step reasoning chain for each question written as if looking at the image, and keep it only if it ends in the correct answer. That yields 28,350 training examples and 899 test examples. They then evaluate nine model setups on two axes: did the final yes-or-no answer match, and did the reasoning trace engage the correct objects and relations, measured both by simple word overlap and by a scene-graph-aware judge model that scores faithfulness and completeness separately from correctness. The headline result is that high accuracy often hides unfaithful reasoning: four models score at least 79% correct while more than 39% of their correct answers rest on reasoning the judge calls unfaithful. Fine-tuning Qwen3-VL-8B on these chains improves in-domain accuracy by 6.2 percentage points and gives the lowest shortcut rate, but makes the model copy the training style on other spatial benchmarks.
The problem they're solving, and why it matters
Current spatial benchmarks mostly grade only the final answer. That creates a blind spot the authors call the shortcut problem. A model can answer correctly without seeing spatially. For example, if most training photos show cups on tables, the model can answer “yes” to “is the cup on the table?” from word co-occurrence alone, never checking left, right, above, behind, or containment in that specific image.
This matters because reported progress becomes misleading. A model that memorizes that people sit on chairs and dogs sit on floors looks spatially competent while failing when precise visual grounding is needed. Prior work that shows reasoning text, such as rationales for science questions or systems that draw boxes around where to look, still does not provide a checkable reference for what spatial relation should have been mentioned. Without that, we cannot tell the difference between a model that looked at the right region but reasoned about the wrong relation and one that truly followed the spatial chain. SpatialChain is meant to audit the process, not just the outcome.
The key idea, in plain words
The key idea is to give every spatial question a verifiable worked solution and then grade new models against the work, not just the final answer.
Think of a math test where the teacher already has a step-by-step solution key. A scene graph, meaning a structured list of what objects are in the image plus their attributes and pairwise spatial relations such as left of or behind, serves as that answer key in symbolic form. The authors use that key to synthesize a natural-language reasoning chain, for example first find the knife, then find the bread to the right of the knife, then check what is to the right of the bread.
Evaluation then asks two separate questions in words before any numbers: did you get the right final answer, and did your intermediate steps mention the right objects in the right spatial arrangement for the right reasons. The second question is split into faithfulness, meaning grounded in the actual spatial evidence rather than hallucinated or restated, and completeness, meaning covering all steps needed to justify the answer. A correct answer with low faithfulness is counted as a shortcut. That separation is what lets the paper surface failures that accuracy alone hides.
How it actually works, step by step
Source questions come from GQA, which pairs images from Visual Genome with structured scene graphs and compositional questions. The authors filter in two stages.
First, type filtering: they keep only 17 GQA question types that structurally involve relations, such as relation verification, existence with a relation, place verification, and two-object comparison. Applied to the balanced training split this gives 135,061 candidates. For the test split they further favor complexity by keeping only the 1,000 questions with the longest semantic annotation strings, which tend to chain multiple relations.
Second, spatial verification: many relation-type questions need no spatial inference, such as “is the dog sitting?” They use GPT-5-nano as a yes-or-no classifier to keep only questions requiring relative position, proximity, containment, or layout, and discard those answerable by object detection or attribute naming alone. This leaves 86,773 training and 900 test candidates, of which 30,000 are sampled for chain generation.
Chain generation uses Claude Haiku 4.5 in extended thinking mode with the full scene-graph text supplied privately. The prompt forbids mentioning graphs, IDs, or coordinates and requires natural visual language such as “looking at the image, I can see.” The extended thinking trace is kept only when its visible answer matches the GQA ground-truth answer. Chains with wrong answers or malformed output are discarded, leaving the final 28,350 training and 899 test triplets, stored in a thinking format with a reasoning section followed by an answer section. Training chains average 115 words, test chains 145 words, reflecting the harder test selection. Left-of and right-of dominate as the first relation mentioned, about 62.7% of training.
Evaluation has two axes. For answer accuracy, the task is treated as binary yes-or-no question answering with exact matching after lower-casing and article removal, plus separate accuracy on yes-questions and no-questions to detect yes-bias. For reasoning quality, the authors use simple overlap measures — ROUGE, meaning word-sequence overlap with the reference, token-level F1, meaning balanced precision and recall over words after removing common stop words, entity overlap, meaning fraction of scene-graph object names mentioned, and length ratio, meaning predicted length divided by reference length — plus a rubric-based judge. The judge, Claude Sonnet 4.6, sees the question, ground truth, scene graph, candidate answer and candidate reasoning and returns three scores between zero and one for answer correctness, reasoning faithfulness, and reasoning completeness. A second judge from a different provider, GPT-5.6, and 198 doubly human-annotated items are used to validate it.
What they showed — results, honestly, with limitations
Fine-tuned Qwen3-VL-8B leads in-domain at 85.03% accuracy, up 6.2 points over its base, followed by fine-tuned Qwen3-VL-4B at 82.23%. The strongest systems without fine-tuning on this data, GLM-4.1-9B and InternVL3.5-4B, tie at 81.34%, with Claude Sonnet 4.6 at 79.89% and GPT-5.4 lowest at 72.18%.
Chain style falls into three regimes. Fine-tuned models match the reference in length and content with overlap scores above 0.65. Base open-weight Qwen3 and InternVL models produce chains three to seven times longer with moderate overlap. Proprietary Claude and GPT-5.4 produce very short chains under 0.4 times reference length with little explicit step structure.
Within each model, better overlap usually co-occurs with correct answers for seven of nine systems, tested with a rank-based Mann-Whitney U test. The two exceptions are revealing. Claude shows almost no difference because its output is too terse to vary. InternVL3.5-8B shows no difference despite the longest output, about 7.07 times reference length, indicating verbose but decorative reasoning decoupled from the judgment. Single versus multi-hop, meaning one relation versus chaining two relations, is not the main difficulty; perceiving individual relations is.
The judge results show the central warning: four models above 79% accuracy still have shortcut rates above 39%, where shortcut rate means the share of correctly answered items whose reasoning scores at or below 0.5 on faithfulness. InternVL3.5-4B is the extreme at 81.34% accuracy with 58% shortcuts. Verbosity does not help: InternVL3.5-8B has a 47% shortcut rate. Fine-tuned models are most faithful at about 0.63 faithfulness and 0.70 completeness with 19 to 22% shortcuts. The judge signal is reasonably robust: judge-to-human rank agreement on faithfulness matches human-to-human agreement, and the second judge preserves the model ranking on shortcut rate while being uniformly stricter.
Fine-tuning has a trade-off. Qwen3-VL-4B improves in-domain but drops on external ERQA from 0.453 to 0.380 and on EmbSpatial-Bench from 0.791 to 0.757. Outputs adopt the training opener “Looking at...” 100% of the time and shrink sharply, suggesting stylistic specialization rather than proven loss of knowledge. The authors propose adding replay of other data as mitigation, but do not test it here.
Limitations are stated clearly. GQA graphs can contain human errors that propagate. Keeping chains only when the final answer matches checks the conclusion, not each intermediate step, so hallucinated steps can remain. Judge and reference generator share the Claude family, though the second-judge check bounds that concern. The task is binary yes-or-no with a 58% yes-rate, and relations are symbolic left/right/above rather than metric distance or true 3D depth.
Why this might matter to me
If you care about spatial reasoning in language and vision models, this gives you a template for moving from “did it say right” to “did it reason right,” with concrete failure modes to watch for: terse commitment with no work shown, and long fluent description that never traverses the required relation. For world models, where an agent must track what is left of what, behind what, or inside what across actions, the faithfulness versus completeness split is directly reusable: a world model that predicts correctly while hallucinating layout will fail under intervention.
For text-to-CAD and parametric geometry, the lesson is sharper. Symbolic relations alone, such as left of or on top of, are not enough for exact sketches, extrusions, constraints, or mating conditions. SpatialChain tests relational grounding, not metric precision, distance, or depth. That makes it a useful pre-check for part placement and constraint intent, but you would still need benchmarks that score coordinates, dimensions, and geometric validity. Similarly for 3D generation and scene representation, scene graphs are a coarse proxy for meshes, fields, or Gaussian layouts. A promising direction the paper enables is to use its verifiable chains as process supervision or rewards for reinforcement learning, then extend the same idea to metric and 3D-grounded chains where each step must cite a position, size, or camera-consistent relation rather than just a word like “right.”
Terms worth knowing
Vision-language model: a model that takes images plus text and produces text, here both reasoning and an answer. Thinking-enabled means it is prompted or trained to emit intermediate reasoning text before answering.
Scene graph: a structured description of an image listing objects, their attributes, and relations between pairs, such as bread to the right of knife. GQA is the source dataset that provides these graphs, images, and compositional questions.
Chain-of-thought / reasoning chain: the step-by-step text a model produces before its final answer, such as first locate the towel, then look to its left.
Faithfulness, completeness, correctness: correctness is whether the final answer matches ground truth. Faithfulness is whether the reasoning is grounded in real spatial evidence without hallucination. Completeness is whether it covers all steps needed to justify the answer.
Shortcut rate and pass rate: shortcut rate is the fraction of correctly answered items whose faithfulness is low, here at or below 0.5. Pass rate requires both reasonable correctness and average faithfulness plus completeness above threshold.
Supervised fine-tuning with LoRA: further training a base model on SpatialChain examples; LoRA is a parameter-efficient method that trains small added matrices rather than all weights.
Overlap metrics: ROUGE-1, ROUGE-L, token F1, entity overlap, and length ratio compare a model’s chain to the reference in words and object mentions. The authors stress these measure surface similarity, not truth.
Polarity gap and yes-bias: the difference between accuracy on questions whose true answer is yes versus no. A large gap suggests defaulting to confirmation from language priors.
Projective versus multi-hop: projective relations are viewpoint-based directions like left, right, above, behind. Multi-hop means chaining two relations via an anchor, such as find the bread to the right of the knife, then check to the right of that bread.