The one-paragraph version
GeoLatent is a method for helping a vision-language model reason about 3D space from a single 2D image without spelling out all of its reasoning in words. It builds on an earlier system called GeoAnchor, which inserts special continuous hidden states — called spatial latents — into the model's thinking trail: one kind for where objects are, one kind for which direction things point, and one kind for overall scene shape. The paper argues that GeoAnchor has two hidden flaws: its overall-shape states all learn almost the same thing, and later answer words can simply ignore the latents and look straight at the image. GeoLatent fixes this with two changes. First, Common-Residual Geometry Alignment, or CR-GEO, separates what is shared across the scene from what is different in different parts, so the shape states are forced to be diverse. Second, routed optimization temporarily blocks direct peeking at the image, forcing answers to go through the latents, then restores full access. With those changes the shape representation goes from effectively one-dimensional to almost four-dimensional, answers become strongly dependent on latents during the blocked stage, and final scores improve to 73.0% on SPAR-Bench and 72.1% on SPBench.
The problem they're solving, and why it matters
A vision-language model, meaning an AI that takes in an image plus a text question and produces a text answer, is often weak at 3D spatial reasoning — tasks like judging depth, distance, whether one object is in front of another, and how the view would change from a different viewpoint. One reason is that when models are taught to show their work, they usually do it as text chain-of-thought, meaning a sequence of words like "the desk is 2 meters away." Words and discrete number tokens are clumsy for continuous geometry. A distance of 2.134 meters versus 2.138 meters, or a slightly diagonal direction, loses fidelity when squeezed into words and can be swayed by language habits.
An alternative is latent reasoning, meaning doing intermediate thinking in continuous vectors — lists of numbers inside the model — rather than in readable words. GeoAnchor took this further by splitting the thinking into three complementary vocabularies: POS latents for object positions, DIR latents for relative directions, and GEO latents for global scene structure. Those latents get extra training signals from geometry teachers. The paper asks: does that supervision alone actually produce good geometry and actually get used? Their diagnosis is no. First, GEO states collapse: they all point the same way, with effective rank near 1.00, meaning roughly one useful direction instead of many. Second, spatial latent underuse: just inserting latents does not make the model rely on them, because under standard attention later words can still attend directly to image tokens and bypass the latents entirely.
The key idea, in plain words
Imagine asking "where is the desk relative to the window?" You need two kinds of help: local facts like the 3D spot for desk and window and the arrow between them, and a global sense of room layout. If you hire eight assistants to describe the room but all eight just repeat "it's an indoor room," you have wasted capacity. That is the collapse problem. And if you write those helpful notes but the final decision-maker never reads them and just glances at the photo, you have the underuse problem.
GeoLatent tackles both. For diversity, it says: capture what is common once, then make each GEO state earn its keep by describing what is left over and different. For example, record "average indoor-scene shape" a single time, then ask state 1 to describe what is floor-like about this corner, state 2 what is wall-like, and so on, instead of letting all states repeat the average. For use, it says: during training, temporarily cut the direct visual shortcut so the answer must be built from the notes, then reopen the shortcut later. Like making a student practice with only their summary notes for a while before allowing open-book plus notes on the final.
How it actually works, step by step
The base is Qwen3-VL-2B-Instruct with a frozen vision encoder and trainable language and latent parts. For each question the model generates an interleaved trail: some text reasoning, plus POS, DIR, and GEO latent spans. A latent here means a small set of continuous hidden vectors — two vectors for each POS and DIR instance, eight vectors for GEO — that are projected back into the model's input space so later words can condition on them.
Local supervision is straightforward. POS hidden states are averaged and mapped to a 3D position in camera-frame coordinates, meaning coordinates centered on the camera that took the picture. DIR states are mapped to a unit direction vector, meaning an arrow of length one pointing from one object to another. Targets come from Depth Anything v3, a depth estimator, by turning depth plus camera details into 3D points. Errors use Smooth-L1 for position and cosine loss for direction, where cosine loss in plain words checks whether two arrows point the same way.
Global supervision is where CR-GEO differs. The teacher is VGGT, a geometry model, whose feature map is pooled at multiple scales — 1-by-1 plus 2-by-2 plus 4-by-4 — to give 21 teacher vectors per image. The old GeoAnchor objective encouraged every teacher vector to be covered by some GEO state plus balanced use of states. The paper proves, in Proposition 1, that if all GEO vectors are identical and equal to the average teacher direction, the balance penalty disappears and the loss is minimized within that collapsed family. In words: repeating the dominant component is explicitly rewarded.
CR-GEO instead does common-residual decomposition in three steps. First, compute a shared direction for teachers and for students by averaging and normalizing. Second, align those two averages once with a term that rewards them pointing together. Third, subtract the shared direction from each teacher and student vector to leave residuals — what makes each patch special — and align residuals with soft assignment. Soft assignment here means each residual teacher softly chooses which GEO state it matches best, with a temperature controlling sharpness. Matching plus a specialization term rewards diverse, complementary assignments. Only the matching term uses stop-gradient on assignments, meaning the assignment weights are treated as fixed when pulling vectors together.
Routed optimization is a three-stage training schedule. In the Joint stage, the model learns answer prediction and geometry together with full attention. In the Bottleneck stage, question, intermediate text, and answer tokens are blocked from directly attending to image keys, while latent tokens keep image access — visual evidence must flow image to latent to answer. Geometry supervision stays on. In the Recovery stage, full attention is restored while geometry supervision continues, so the model can combine direct image access with the latent-mediated route it just practiced.
What they showed — results, honestly, with limitations
On benchmarks, GeoLatent reaches 73.0% on SPAR-Bench and 72.1% on SPBench, which the authors report as +4.6 and +2.4 points over previously reported GeoAnchor. Gains appear in all five SPAR categories, largest on View imagination at +7.5, and +5.6 on SPBench absolute reasoning. ViewSpatial, an external transfer set with camera-centered and person-centered questions, is lower at 43.2% overall, with 53.2% on camera-relative parts.
For structure, under identical initialization and batch order, swapping only the geometry loss raises effective rank — a measure of how many independent directions the GEO vectors really use — from 1.00 to 3.87, and assignment information — a diagnostic that is zero when every teacher picks the same GEO state and higher when assignments are specific and spread out — from near zero to 0.228. After full recovery the representation stays diverse at 3.62 versus 1.00 for the old loss, close to the teacher-residual rank around 4.1.
For use, on a fixed set of 128 direction questions, blocking downstream reading of all latents at the bottleneck drops accuracy from 89.1% to 25.8%, essentially to chance, while the public GeoAnchor checkpoint stays at 82.0%. The drop is only 0.78 points at joint and again 0.78 after recovery, so strong dependence is localized to the bottleneck stage, not permanent. After recovery a usable latent-only route remains: without direct image access the model keeps about 52.2%, 51.3%, and 33.5% on the three benchmarks, beating a no-latent baseline by 5.3, 3.7, and 1.8 points, whereas a control that skipped bottleneck shows about zero gain.
For grounding, local POS and DIR remain decodable and image-specific: 0.338 meter error and 0.899 cosine on held-out scenes, rising from 0.389 to 1.474 meters when the image is swapped. Blanking the image removes 88.3% of GeoLatent's above-chance ViewSpatial margin.
Limitations are stated unusually clearly. Aggregate accuracy alone does not prove the mechanism: a matched no-GEO control with no global latents scores 62.4% mean versus 62.8% for GeoLatent. Person-centered questions remain weak — 28.1% real, near chance when blanked — because the current camera-frame vocabulary does not explicitly encode the observer's orientation. The geometry-capacity sweep suggests 4 GEO vectors is a better compact operating point than the 8 used for compatibility, cross-backbone reproduction improves in-domain scores but drops ViewSpatial by 1.70 points, and overlap checks find no exact duplicates but cannot make a scene-disjoint claim for SPAR-Bench due to missing metadata.
Why this might matter to me
If you care about spatial reasoning in LLMs and VLMs, this is a concrete case where adding interpretable latent slots plus attention routing does more than adding text reasoning — but only when you measure structure and use separately, not just final accuracy. The bottleneck-then-recover pattern is directly reusable as a training recipe for world models: force a compact state to carry dynamics or layout under restricted access, then restore rich observations while keeping the state supervised.
For text-to-CAD and parametric geometry, the POS and DIR story is encouraging — positions and direction arrows can be decoded from latents in meters and cosines and are tied to the specific image — but the GEO story is a caution. Even with supervision, global shape collapses unless shared versus residual structure is explicitly separated. That maps to CAD: do not make eight parameters all re-learn "box-like"; model the common primitive once and spend capacity on residuals like holes, bevels, or placements. For 3D generation and scene representation, CR-GEO's multi-scale VGGT teacher and effective-rank diagnostics offer a practical way to test whether added tokens actually add dimensions rather than repeating the dominant view direction, and the Person-frame failure points to the next vocabulary to add: explicit observer pose.
Terms worth knowing
- Vision-language model (VLM): a model that reads an image plus text and writes text.
- Spatial reasoning: judging depth, distance, relative position, orientation, and layout from images.
- Chain-of-thought: spelling intermediate reasoning as word steps; brittle for continuous space.
- Latent reasoning / spatial latents: doing intermediate reasoning as continuous vectors rather than words. Here POS = object position, DIR = relative direction, GEO = global scene structure.
- VGGT teacher features: geometry-rich vectors from a Visual Geometry Grounded Transformer used as targets for GEO states, pooled here at multiple scales to capture coarse and fine structure.
- Depth Anything v3: a depth estimator used to build POS and DIR targets by back-projection into camera-frame 3D.
- Camera-frame coordinates: 3D coordinates centered on the camera, as opposed to person-centered or world coordinates.
- Effective rank: a number summarizing how many independent directions a set of vectors really uses; near 1 means redundant.
- Assignment information: a diagnostic for whether different teacher patches prefer different GEO states versus all picking the same one.
- Coverage and balance loss: the old objective that rewards covering all teacher patches and using all GEO states equally — which, the paper shows, can still be satisfied by collapse.
- Routed optimization / visual bottleneck: a training schedule that temporarily blocks direct image-to-answer attention so information must pass through latents, then restores it.
- SPAR-Bench, SPBench, ViewSpatial: evaluation sets for spatial questions; the first two are in-domain, the third tests transfer to new camera- versus person-centered phrasings.