The one-paragraph version
Spatial-Interactor is a post-training method that teaches vision-language models — AI systems that take images or video plus text as input — to reason about space by learning from interaction, not just from static questions. The authors argue that a video of someone moving through a room is naturally a chain of small changes: what you saw before, what action happened, and what you see after. They turn that chain into training data at three scales: changes in the world while the camera stays still, changes in what you see because you yourself moved, and long trajectories where many small moves must be added up. They build a dataset of about 108,000 checkable question-answer pairs from simulators and real camera and robot logs, called Learning from Spatial Interaction (LSI-108K), then train in two stages: standard supervised fine-tuning for the short changes, followed by a new reinforcement procedure called On-Policy Distillation that uses privileged textual descriptions of local motion to guide reasoning over long videos. Across four Qwen backbones and multiple spatial benchmarks, this improves scores substantially and makes models more sensitive to the order of frames.
The problem they're solving, and why it matters
Today's vision-language models are good at describing a single picture, but weak at staying oriented in a changing 3D world — tasks like remembering a room from multiple views, estimating how far a camera moved, planning a route, or acting in a loop where each move changes the next view.
The paper opens with two simple diagnostics that make the gap concrete. First, a frame-shuffling test: if you randomly reorder the 32 frames of a spatial video, a model that truly tracks motion should get much worse. In fact Qwen2.5-VL-3B and 7B dropped by only 0.9 and 0.8 points on VSTI-Bench, suggesting they rely on order-invariant clues like recognizing objects, not on the sequence of changes.
Second, a local-to-long-horizon comparison: the same models can answer a short interaction question reasonably well on SAT-Real — 54.7% for Qwen2.5-VL-7B and 88.7% reported for GPT-5.5 — but collapse to 8.6% and 23.9% on the long-horizon camera-displacement task in VSTI-Bench that requires accumulating motion over a whole trajectory. Tellingly, when GPT-5.5 was given its own textual summaries of local changes segment by segment, its long-horizon score rose from 23.9 to 35.5.
In other words, existing spatial training mostly asks static questions — what is left of what, how far apart are two things in one view — and provides little direct supervision for how states change and how those changes add up. For embodied uses like navigation, object search, and manipulation, that missing ability matters.
The key idea, in plain words
Learn space the way an infant does: by watching what changes and by moving.
Each interaction is framed as a before-action-after triplet: the observation at one time, the action that connects them such as an object being rotated or the camera turning left, and the next observation. A concrete example is two views where the only difference is that a laptop rotated, or where the camera rotated left by about 50 degrees. Predicting the cause or consequence of that difference teaches a local state transition — plainly, one small update to your mental model of the world or of yourself.
A full trajectory is then just many such updates in order. To learn long-horizon integration — plainly, keeping a running total without losing track or mixing up earlier and later views — the model needs to preserve order and accumulate evidence. The authors therefore split learning into a curriculum:
Level 1 learns passive world changes under a roughly fixed viewpoint, such as an object opening, turning on or off, being removed, swapping position, a pot moving about 20 cm closer, or a robot gripper moving left.
Level 2 learns active self changes when the world is roughly fixed but the observer moves, such as moving forward 0.4 meters, comparing which of two views moved farther, ordering shuffled views in time, or inferring from view B where the sofa is relative to the door.
Level 3 learns to integrate over a complete camera path, such as estimating total path length of about 8.5 meters, describing path shape as forward then turn right, locating when the main turn happened in frames 17-24, or finding which frames revisit the same place.
How it actually works, step by step
First, the team builds LSI-108K from interaction records that already contain ground truth about what changed.
In simulation — ProcTHOR, AI2-THOR, HSSD, Replica, and rendered SIMS-V trajectories — an agent executes valid camera moves or object operations, and the simulator logs the action, camera poses, and object states before and after. In real data — ScanNet, ScanNet++, MultiScan, ARKitScenes, and RoomTour3D for cameras, plus BridgeData V2 for robot arms — they normalize camera poses, object tracks, and end-effector states to derive measured changes in translation, rotation, distance, visibility, operation order, and full path geometry. Deterministic templates then turn each verified target into a question with no language-model invention, and a frozen visual-language checker only rejects visually unanswerable items without rewriting answers. The final count is 107,518 pairs rounded to 108K: about 14.1% Level 1, 64.6% Level 2, 21.3% Level 3; 51.5% simulated, 39.2% real camera, 9.3% real robot.
Second, Stage 1 trains local transitions with Supervised Fine-Tuning, meaning training the model to predict the correct human-verified answer given the before-and-after views. This uses 82,596 Level 1-2 examples plus 80,000 public spatial questions from VSI-590K, MindCube, and VSTI-Bench. Only the language model and multimodal projector are updated; the visual encoder stays frozen.
Third, Stage 2 trains long-horizon integration with On-Policy Distillation inside a reinforcement framework called Group Relative Policy Optimization. In plain terms: for each long 32-frame video, a frozen strong annotator writes four short paragraphs, one per 8-frame interval, describing the environment, diagnostic visual changes, and cautious coarse camera motion, without seeing the question or answer. Those four paragraphs concatenated in time order form a privileged trace available only during training.
During each update the current student policy samples eight reasoning-plus-answer responses on its own. A teacher, which is a frozen snapshot of that same policy before the update, scores the same student-written reasoning prefixes, but with the privileged trace also in its input. The difference in next-token predictions becomes a process loss — a penalty that pushes the student's step-by-step reasoning distribution toward the teacher's better-informed distribution — applied only to reasoning tokens, not the final answer. The final answer still gets a verifiable reward based on exact matching for choices, relative error for numbers, or field-wise accuracy for multi-part answers. The process weight starts small and decays to zero, so early training gets explicit guidance and later training relies more on the model's own reasoning and outcome rewards.
What they showed — results, honestly, with limitations
Across Qwen2.5-VL-3B/7B and Qwen3-VL-4B/8B, Spatial-Interactor improved the average over VSI-Bench, MindCube, VSTI-Bench, and SPBench-MV by 16.4 to 25.0 points over each corresponding base model, reaching 65.9 Overall for the 8B variant. Gains spanned relative distance, route planning, and camera displacement, plus multi-view benchmarks. The authors also report best averages among compared systems on VSI-Bench, MindCube, and VSTI-Bench, while SpatialLadder-3B remained strongest on SPBench-MV.
That headline needs context. Stage 1 included 80K public questions from the same benchmark families used in the main evaluation, so part of Table 1 reflects in-distribution exposure, not purely interaction data. The cleaner transfer test is on four benchmarks absent from training — MMSI, ViewSpatial, SAT-Real, SAT-Syn — where gains were smaller but consistent at 4.3 to 10.6 points, most coming from supervised fine-tuning with an extra 1.0 to 1.4 points from On-Policy Distillation.
Ablations on Qwen2.5-VL-7B help isolate effects: external-data-only fine-tuning raised Overall from 35.1 to 56.0, adding Level 1-2 interaction data raised it to 58.4, standard reinforcement raised it to 59.2, and the full privileged-distillation version reached 60.1, with the largest edge on route planning and camera displacement. Training Level 1 then Level 2 sequentially was the only local setup to improve all four held-out benchmarks. Training dynamics show task reward rising and teacher-student divergence falling together, and trained models become markedly more order-sensitive: shuffling frames now drops scores by 6.6 to 7.0 points versus less than 1 point before. Closed-loop tests on WalkerBench and ESI-Bench also improved success while using fewer actions, but only for the 7B model, and the expanded ESI breakdown shows a few categories where performance stayed flat or declined. The paper does not clearly establish whether the recipe transfers beyond the Qwen family.
Why this might matter to me
For spatial reasoning in language and vision models, this offers a reusable pattern: don't just ask what is where in one frame, supervise the delta between frames and then supervise how deltas compose. The frame-shuffle and trace-order diagnostics are directly adoptable for testing whether a model truly uses temporal evidence versus bag-of-views shortcuts.
For world models, the before-action-after triplet is a lightweight, observable alternative to full physics simulation. It treats video plus logged action or pose as supervision for local transitions, and the privileged textual trace as a scaffold for learning to maintain state. That scaffold-then-remove idea could inform how a predictive world model learns to roll out longer trajectories without drifting.
For text-to-CAD and parametric geometry, the Level 1 tasks — inferring rotation, displacement magnitude, open versus closed, operation order — map closely to editing operations on parts and assemblies. Training a model to ground phrases like moved about 20 cm or rotated left in paired before-after views could improve instruction-driven parametric edits where preserving identity across a change matters more than generating a novel shape.
For 3D generation and scene representation, the Level 2-3 tasks — correspondence across views, parallax-based depth, imagined viewpoint consequences, path length and revisit detection — reward multi-view consistency and trajectory coherence. Even without explicit depth, pose, or reconstruction, learning from ordered interaction records could complement neural scene representations by providing data-driven priors for what should stay stable when only the camera moves versus when the world itself changes.
Terms worth knowing
Vision-language model (VLM): a model that takes images or video together with text and produces text answers or reasoning.
Spatial reasoning: here, perceiving layout and relations and tracking how they change with object motion, viewpoint change, and time.
State transition: one update from a before-state to an after-state. A world-state transition is a change in the environment, like a laptop rotating. A self-state transition is a change in the observer's pose, like the camera moving forward.
Ego-motion: movement of the camera or agent itself, as opposed to movement of objects in the scene. Includes translation such as forward or left, and rotation such as yaw left or pitch up.
Interaction trajectory: an ordered sequence of local records, each containing a before observation, an action, and an after observation. Local windows teach single steps; the full ordered sequence teaches long-horizon integration.
LSI-108K: the authors' dataset of about 107,518 verifiable question-answer pairs derived from simulated and real interactions, organized into the three-level curriculum.
Supervised Fine-Tuning (SFT): training the model to imitate correct answers on labeled examples.
Group Relative Policy Optimization (GRPO): a reinforcement method that samples a group of answers per prompt, scores them with a rule-based reward, and reinforces relatively better answers within the group.
On-Policy Distillation (OPD): the paper's addition to GRPO where a privileged teacher guides the student's reasoning steps on responses the student itself just sampled, rather than imitating fixed offline rationales.
Privileged trace / privileged teacher: a textual summary of local motion per video segment and the teacher branch that sees it during training. Privileged means it is available to the teacher at train time but never to the student at test time.