All summaries · 2026-09-29

Who Is Left of Whom? Tracing Spatial Evidence and Role Binding in Relative-Position Reasoning

Yingjin Song, Denis Paperno, Albert Gatt

arXiv:2609.35486 · abstract · pdf · summarized by meta/muse-spark-1.3-contributor on 2026-09-29

The one-paragraph version

This paper asks why vision-language models and language models can answer a single spatial question correctly — “Is the blue square to the left of the green triangle?” — yet fail when the same relation is probed twice. The authors argue that answering correctly requires two separate jobs: remembering where each mentioned object is, and keeping track of who is who in the question — which object is the target being located, and which is the reference it is located against. Using paired tests that flip the correct answer, plus internal interventions in three open models under matched image and text inputs, they trace a three-stage pipeline from scene evidence to query mentions to final answer. They then show that boosting a small, learned role signal improves consistency on natural images without retraining.

The problem they're solving, and why it matters

Benchmark accuracy alone can be misleading for spatial reasoning. A model can get the original question right but give the same answer again after the world or the question is reversed, which means it did not really compare the two objects in their roles.

The authors make this concrete with two paired perturbations that both should flip the answer from, say, left to right:

The first test mainly stresses location tracking. The second stresses what the authors call role binding — assigning target versus reference — because locations are unchanged but the correct answer reverses. The paper shows this inconsistency is widespread, even in strong models, and then asks what internal representations support or fail to support the two computations.

The key idea, in plain words

Think of the task as: scene in, question in, one-word relation out.

The hypothesized mechanism has three stages:

  1. Scene side stores where things are. Whether the scene is pixels or a sentence like “red circle, blue square, green triangle from left to right,” early model states at the scene tokens hold position information tied to each object.
  2. Question side copies that where-information to the object names. By the middle layers, the mention of “blue square” in the question carries that object’s location, and the mention of “green triangle” carries its location. At the same place, another signal marks which mention is the target and which is the reference.
  3. Late comparison reads the two locations according to the two roles. Subtract reference location from target location along the queried axis, check the sign, output left or right — or above versus below.

In plain terms: location IDs say where each queried object is, while a separate role direction says who is being located relative to whom. You need both, because the same two locations demand opposite answers when target and reference are swapped.

How it actually works, step by step

Matched setups. The authors compare three input conditions for the same underlying scenes: VLM plus image, VLM plus text description with a blank image, and LLM backbone plus text description. Mechanistic work focuses on three families: LLaVA-1.6-Mistral-7B, InternVL3.5-8B, and Pixtral-12B, plus their language backbones.

Controlled data. Synthetic scenes use simple colored shapes on a grid, with two- or three-object horizontal or vertical layouts. Text descriptions list objects in opposite orders, for example left-to-right versus right-to-left, to separate described position from mention order. They also use What’sUp, COCO-Spatial, and GQA-Spatial natural images for behavioral testing and transfer.

Step 1: localize with activation patching. Activation patching here means taking a pair where clean and corrupted inputs want opposite answers, rerunning the corrupted input, but splicing in internal states from the clean run at one layer and one group of tokens, then seeing if the clean answer comes back. Groups include all visual tokens, tokens in the target and reference bounding boxes or rows/columns, description spans naming objects or stating arrangement, joint target-plus-reference query spans, relation option words, and the final token before answering.

Step 2: extract location IDs. A location ID is a prototype vector for a position label like left, middle, right. To build it, the authors pool the internal state for each object at two sites — scene side and question side — then subtract the average state for that same object identity, for example all red circles, to remove what the object is and keep where it is. Then they average what remains across all objects that share the same position. A spatial axis is simply the difference between opposite prototypes, for example right minus left.

Step 3: test source-to-query transfer. They patch corrupted scene states into a clean run and ask: does the downstream question-side comparison shift toward the corrupted locations, and does the final answer flip? They measure both the shift in a target-minus-reference projection and the rate of answer flips.

Step 4: steer locations directly. Keeping the input unchanged, they add the difference between swapped location IDs to the target and reference question spans. If question-side locations cause answers, this should change the model’s preference to the swap-implied answer.

Step 5: find and test the role direction. To separate role from word order, they use two templates with the same roles but opposite mention orders: target-first, “Is the blue square to the left or right of the green triangle?”, and reference-first, “Relative to the green triangle, is the blue square on the left or right?” The role direction is the average difference between the same object’s state when it appears as target versus as reference, averaged across both templates. They disrupt it by pushing target and reference mentions in opposite directions along it, and later amplify it by pushing them along their correct role signs.

What they showed — results, honestly, with limitations

Behavior: high accuracy hides inconsistency. Image input beats text input in all ten models tested. But paired consistency is consistently lower than single-question accuracy. For example, LLaVA-1.6-Mistral on What’sUp reaches about 88.9% original accuracy in the image condition, but only about 81.0% target-reference reversal consistency and 78.3% location-swap consistency. On synthetic data, some Vicuna-based text models score 50% accuracy by almost always answering left or above, yet score 0% on both paired tests. The paper does not claim these particular numbers generalize to all models, only that the gap is systematic.

Patching: staged progression. Under location swaps, effective patches move from scene tokens in early-to-middle layers, to query-object mentions in middle layers, to the final token late. Under role reversals, query-object and late final-token patches dominate, while scene patches do little — expected because the scene is identical across that pair. In images, patching all visual tokens nearly fully restores the answer, while tight bounding-box patches are weaker than broader row or column strips. In text, object-naming description spans matter most on the scene side.

Locations are causal and transfer from scene to question. Extracted location IDs predict held-out positions well above shuffled-label controls, form orderly left-middle-right geometry, and support relation judgments: the sign of target-minus-reference along the extracted axis predicts left versus right or above versus below near 98-99%, versus near chance for orthogonal controls. Patching corrupted scene states shifts question-side location scores and flips answers, most strongly with full visual-token patches. Direct question-side location steering flips beliefs to the swap-implied answer with middle-layer peaks around 75% to 94% for images depending on model, and lower but clear effects for text inputs, well above random, shuffled, or wrong-axis controls.

Roles are a separate, stable, causal direction. Held-out target-versus-reference contrasts align strongly with the estimated role direction across layers, models, and modalities. Disrupting that direction reduces preference for the correct role-sensitive answer more than matched orthogonal random directions, especially in early-to-middle layers and in a strength-dependent way.

Steering roles helps real images. Directions and layers chosen only on synthetic data improve What’sUp-A target-reference reversal consistency in all nine model-by-condition settings, significantly in eight, and improve accuracy and location-swap consistency in most. The largest reported gains are in Pixtral text conditions, for example VLM-plus-text accuracy up 4.9 points, reversal consistency up 7.8 points, and swap consistency up 9.4 points. Similar transfer holds on COCO-Spatial and GQA-Spatial without retuning.

Limitations, stated plainly. The analysis covers only left, right, above, and below in fixed-grid two- and three-object scenes with templated questions, not depth, distance, clutter, or free phrasing. Patching localization uses correctly answered pairs, so it characterizes how correct answers are implemented rather than diagnosing every natural error, though full-benchmark steering partly addresses errors. Only three 7B–12B open models are dissected. Steering is presented as an analysis tool that improves consistency, not yet as a general post-training method. One detail that remains less fully explained is why broad visual context patches so much better than tight object boxes; the paper reports it but does not resolve whether it reflects distributed position coding or attention smearing.

Why this might matter to me

For spatial reasoning in LLMs and VLMs, the separation of where-information from who-is-who offers a practical diagnostic. If your model fails role reversals more than location swaps, the fix may not be better vision grounding but stronger binding of target and reference in the question. Their paired tests are cheap to add to evaluation.

For world models, the source-to-query transfer result matters: scene tokens act as an early spatial memory that question tokens later read. That suggests representing persistent object slots with explicit position tags, rather than relying on the final token to rediscover layout, could make multi-step prediction and planning more consistent when objects move.

For text-to-CAD and parametric geometry, where a description like “plate left of bracket” must become exact constraints, the role-direction finding is directly relevant. Left-of-A-versus-B and left-of-B-versus-A are opposite constraints with identical entities. Amplifying an explicit target-reference signal before constraint solving could reduce flipped mates and mirrored placements, especially when language is the only scene source.

For 3D generation and scene representation, the fact that synthetic-estimated role and location directions transfer to natural images hints that coarse relational scaffolding may be shared across domains. You could imagine using such directions as lightweight probes or steering knobs during layout-to-scene decoding, while leaving detailed geometry to explicit 3D parameters, rather than retraining large backbones for every spatial phrasing.

Terms worth knowing