The one-paragraph version
Vision-Language Models are good at describing flat pictures but bad at reasoning about the physical world behind those pictures, because a photo collapses depth and motion into two dimensions. This paper introduces FactoSR, a training method that teaches a model to unfold that collapsed information back into three separate, checkable skills: matching the same point across different views left-right and up-down, ordering objects front-to-back in depth, and understanding camera motion forward and backward in time. It does this in two stages — first supervised fine-tuning to build basic spatial perception, then reinforcement learning with verifiable rewards where each of those three skills gives its own score. Built on Qwen3-VL-8B, the full system improves about 5.9 points over its base model on both VSI-Bench for video-spatial reasoning and All-Angles-Bench for multi-view reasoning, and reaches the best average among tested open-source models across 3D/4D benchmarks, while mostly preserving general chat and reading abilities.
The problem they're solving, and why it matters
The problem is a dimensional mismatch. Humans perceive the world as multi-view, depth-aware, and continuous in time — we move our head, notice occlusion, feel parallax when near things shift faster than far things. A Vision-Language Model, meaning a model that takes images or video plus text as input and produces text as output, is typically trained on single 2D projections: one image, one question, one answer.
The authors argue this leads to heuristic guessing. For example, a model might say the bigger-looking potted plant is closer, or identify a person across views by shirt color rather than geometry, or answer a navigation question like “did the camera move left or right?” from static cues without tracking how foreground and background shifted.
Prior fixes have not solved this. One family synthesizes large spatial question-answering datasets for supervised fine-tuning, meaning training the model to imitate correct answers, or adds special spatial tokens, meaning extra learned vectors meant to carry 3D information. The paper says these are hungry for explicit 3D data, transfer poorly to 4D scenes that include time, and still rely on single-view questions. A newer family uses reinforcement learning with verifiable rewards, meaning the model tries multiple answers and gets an automatic score for being right rather than imitating one demonstration. But existing versions mostly reward only final correctness, so the dynamics get squeezed by depth distortion and temporal drift. Directly optimizing one unified 4D objective over space plus time would be ideal but, the authors claim, is computationally and algorithmically intractable — too high-dimensional to learn all at once.
This matters for autonomous driving, robot navigation, and world models that need to remember layout, predict motion, and act, not just caption images.
The key idea, in plain words
Divide and conquer. Instead of asking the model to learn “4D world consistency” as one giant thing, split the dimensions that camera projection collapsed, and reward each separately.
Think of unfolding a photo back into the world:
- Unfold left-right and up-down: if you mark a pixel in View 1, where should it land in View 2 given where the cameras were and how far away that point is? This is planar correspondence. A concrete test is: here is a reference point in image one and four candidate points A, B, C, D in image two — pick the one that is geometrically possible, not just similar-looking.
- Unfold depth: from a flat image, which object is truly in front of which? This is depth consistency. A concrete test is: given predicted 3D boxes for a mug, two cans, and a table, is the front-to-back order correct, even if you do not get exact meters right?
- Unfold time: if the camera moved “turn right, then turn back” going forward, does reasoning backward give the inverse “turn back, then turn left”? This is temporal reversibility. A concrete test is: answer both the forward navigation question and the reversed video question correctly, otherwise you were using a shortcut.
By optimizing these three verifiable constraints inside one policy learning loop, the authors turn an ill-posed problem — guessing 3D plus time from 2D — into tangible reasoning steps they call observe, localize, think, and answer, implemented as an “Anchor-Transfer-Verify” chain of thought: anchor on key objects in the first frame, transfer them across views or time using motion and parallax, then verify the conclusion is consistent.
How it actually works, step by step
The backbone is Qwen3-VL, a general open vision-language model, in the 8-billion-parameter Instruct version. Training has two stages.
Stage 1: Spatial Perception Fine-tuning, called FactoSR-SFT.
First, short-answer joint training for one epoch. The data totals 8.2 million samples, about 90.3% short answers and 9.7% long answers. Short answers mix general instruction data from LLaVA-OneVision at 5.1 million with new spatial data at about 1.6 million and math at about 737 thousand. These teach grounding skills like localization, meaning finding where something is, correspondence, meaning matching the same thing across images, and spatial relations like distance, size, position, counting.
Then, cold-start for reasoning over several hundred iterations on about 795 thousand long-answer samples, including 590 thousand newly built spatial chain-of-thought examples. Here the model learns the Anchor-Transfer-Verify pattern for cross-frame alignment, progressing from basic grounding to reasoning. This stage is meant to stabilize the later reinforcement learning stage.
The new 1.2-million-sample spatial set covers single-view depth estimation, multi-view correspondence, multi-view camera-relative motion, and video spatial understanding. The paper’s Figure 3 shows the detailed breakdown, for example position, distance, size, camera motion, and depth tasks across video, single-view, and multi-view.
Stage 2: Factorized Spatial Reinforcement Learning, called FactoSR-RL.
The model is refined with Group Relative Policy Optimization, a reinforcement learning algorithm that samples a group of answers to the same question, here 8 samples per query, scores each with rules, and updates toward answers that are better than their group average, without needing a separate value model. Training uses VeRL software, AdamW optimizer, and a small curated set of 32 thousand samples, about 81.2% spatial and 18.8% general math, split into multi-view, single-view, camera motion, correspondence, distance, and grounding.
The reward is mixed and rule-based. First, a format reward requires structured output with thinking in think tags and final answer in answer tags. If format fails, total reward is zero. Otherwise total reward is a weighted sum:
Total = format-gate times (importance-1 times accuracy + importance-2 times XY + importance-3 times Z + importance-4 times T)
The paper does not report the exact weight values, which is unclear.
In words before symbols:
- Accuracy reward checks final correctness: does the predicted answer equal the ground truth?
- XY reward for point correspondence checks geometric admissibility. Conceptually: take the reference pixel plus its depth, unproject it into 3D camera coordinates using camera intrinsics, meaning the internal parameters like focal length that map 3D rays to pixels, transform to world coordinates using camera pose, meaning position and orientation, then reproject into the second view. If that projection falls outside the image or behind the camera, reward is zero. Otherwise the model gets a soft score that decays exponentially with normalized pixel distance to the true projection, cut off beyond three tolerances to avoid rewarding wild guesses, multiplied by a visibility mask that requires depth agreement within a threshold to rule out occluded points. Even a close guess gets zero if it lands on a non-overlapping or occluded area.
- Z reward for depth order checks relative ranking, not metric meters or box overlap. Conceptually: match each predicted 3D box to a true box using Hungarian assignment, meaning optimal one-to-one matching that minimizes total 3D box cost based on generalized intersection-over-union, extract each box center’s depth in camera coordinates, then compare every pair: did you get the front-behind order right? The Kendall-tau rank correlation, meaning number of agreeing pairs minus disagreeing pairs divided by total pairs, ranges from minus one to one and is normalized to zero to one for the reward.
- T reward for temporal cycle consistency checks reversibility. Conceptually: each sample has a forward question, like actions from door to heater, and a constructed inverse question from end back to start. The model must roll out both. Reward is one only if both forward and inverse answers are correct, implemented as product of the two accuracy checks.
What they showed — results, honestly, with limitations
On 4D reasoning, the paper reports FactoSR-8B-RL at 55.4% average on All-Angles-Bench and 61.4 to 61.5% on VSI-Bench depending on table, up about 5.9 points over the Qwen3-VL-8B-Instruct base at 49.5% and 55.6%. The abstract phrases this as a 5.9% boost on VSI-Bench and 4.5% on All-Angles-Bench, while the main text says +5.9 and +2.5 over the strongest open-source baselines — the baseline for comparison differs by section, so read exact deltas carefully. Gains appear across sub-tasks like attribute identification, pose, relative direction, relative distance, appearance order, and route planning, with large jumps cited such as +23.3 on All-Angles pose and +14.4 on VSI route versus base.
Across 13 benchmarks, Table 2 reports a 3D/4D average of 62.0 versus 59.1 for base, a +2.9 improvement and best among tested open-source models, including BLINK, 3DSRBench, CVBench 2D and 3D, Embodied Reasoning QA, RealWorldQA, and MMSI. General abilities are mostly preserved at 84.1 on MMBench Chinese, 85.7 on MMBench English, 69.1 on MMStar, and 87.6 on OCRBench, but there are small drops versus base on MMStar at minus 1.0 and OCRBench at minus 2.7, so spatial tuning is not entirely free.
Ablations support factorization. Vanilla reinforcement learning without the new rewards gives only +0.2 overall on nine spatial benchmarks. Adding XY alone helps correspondence tasks at +2.7, adding Z alone helps depth tasks at +1.3 including +3.2 on relative depth, adding T alone helps camera-motion and route tasks at +7.9. Full combination gives +1.7 overall and balances all three, while a vanilla grounding reward based on box overlap actually hurts at minus 0.2 overall. Qualitative examples show a shift from size heuristics to occlusion, layering, parallax, and cross-frame verification.
Limitations to note plainly: the method still needs camera intrinsics, poses, and depth maps to compute XY and Z rewards during training, so it is not free of 3D supervision; reward weights, exact data pipeline, and compute cost are under-specified; evaluation is only shown on an 8B backbone with its native prompts, with variance reported as plus-or-minus 0.1 to 0.2 but no real robot or closed-loop test; and some claims like acting as a latent world model are illustrated by one route-planning example rather than proven by systematic world-model metrics.
Why this might matter to me
For spatial reasoning in LLMs and VLMs, this is a concrete alternative to just scaling supervised question-answering or adding spatial tokens. It suggests final-answer rewards are insufficient and that intermediate geometric checks — is the match reprojectable and visible, is the depth order right, is motion reversible — produce more transferable Anchor-Transfer-Verify reasoning you can inspect.
For world models, the temporal cycle trick is directly reusable: requiring forward-backward consistency forces a model to maintain a coherent latent state of layout and ego-motion instead of answering from single frames. The video route example where the model infers leftward pan, motion parallax, and turn-left then turn-right from 16 egocentric frames is a small prototype of decoding scene dynamics for navigation.
For text-to-CAD and parametric geometry, the Z lesson is pointed: optimizing box overlap did nothing, while optimizing relative depth order helped. That aligns with CAD workflows where topology and ordering constraints like front-behind, coincident, concentric matter more than absolute metric regression first. A similar rank-based or constraint-based reward could complement exact parameter accuracy.
For 3D generation and scene representation, the XY reward is essentially a verifiable multi-view consistency loss using unproject-reproject plus occlusion gating. That same machinery — intrinsics, poses, depth agreement, overlap masks — could score or filter generated novel views, NeRF or Gaussian-splat renders, or video diffusion outputs for geometric plausibility without a learned reward model.
Terms worth knowing
- Vision-Language Model: a model that reads images or video plus text and writes text answers.
- Spatial reasoning: answering where things are, how far, what occludes what, and how views connect, not just what is visible.
- 4D: three spatial dimensions plus time, meaning dynamic scenes and camera motion, not just static 3D.
- Supervised fine-tuning: training by imitating labeled examples.
- Reinforcement learning with verifiable rewards: training by sampling multiple answers and scoring them with automatic rules for correctness.
- Group Relative Policy Optimization: a policy-gradient method that compares answers within a sampled group to estimate advantage, avoiding a separate value network.
- Intrinsics and pose: intrinsics are internal camera properties like focal length; pose is external position and orientation. Together they define how 3D points project to pixels.
- Reprojection consistency: a predicted match is valid only if it agrees with where the true 3D point should project under known cameras and depth.
- Overlap or visibility mask: pixels marked as truly seen from another view after checking depth agreement, used to reject occluded guesses.
- Hungarian assignment: optimal one-to-one matching between predictions and ground truth to minimize total cost.
- 3D GIoU: generalized intersection-over-union for 3D boxes, a measure of box overlap and alignment used here for matching cost.
- Kendall-tau: a rank correlation that counts agreeing versus disagreeing front-behind pairs to score depth ordering.
- Cycle consistency: forward reasoning and backward reasoning must invert each other, used here to enforce temporal understanding.
- Anchor-Transfer-Verify: the paper’s chain-of-thought pattern — anchor on key objects, transfer across views or time, verify consistency before answering.