The one-paragraph version
This paper argues that current 3D vision-language models are not really looking. When we ask them about an object we usually point to it with a 3D bounding box — a center plus width, height, depth and yaw. That box alone already tells you a lot: a tall skinny box from floor to ceiling is probably a door, a queen-bed-sized box is probably a bed. The authors call this bounding box semantic leakage. They prove it by training a blind language model that sees no 3D scene at all, only box coordinates as text, and it still gets 36.4% class accuracy on Nr3D captioning. Replace the box with a single 3D point on the object, and that blind model collapses to 18.0%, and existing models like LL3DA and GaussianVLM also drop by 15-17 points. To solve this harder point-only task they build P3-VLM, a detector-free, scene-centric model on 3D Gaussian Splats with three visual streams — one for what is at the queried point, one for task-relevant scene detail, one for global context — and it reaches 52.1% on Nr3D points and 56.1% on ScanRefer points, plus best robustness to noisy boxes, best out-of-distribution counting on ScanNet++, and best scores on ungrounded ScanQA and SQA3D.
The problem they're solving, and why it matters
3D vision-language models, meaning large language models given a 3D scene to talk about, are supposed to be generalists. Given a scan of a room they should answer both grounded queries, meaning queries tied to a specific place like caption this object here, and ungrounded queries, meaning questions about the whole scene like is this kitchen tidy.
The field has split into two families. Object-centric models first run a 3D object detector to carve the scene into proposed objects, encode each proposal separately, and feed those to the language model. Scene-centric models encode the whole scene at once, without a detector, which in principle should be more robust.
The authors show both families share the same cheat. Almost everyone prompts or represents location with bounding boxes. For object-centric models, the visual tokens themselves come from detector boxes. For scene-centric models, the inference-time query is still a box center plus extent. In both cases the geometry of the prompt leaks the answer.
Why this matters is that it hides visual blindness. If you can guess door from tall-thin and bed from large-flat without ever processing color, texture, or fine shape, your benchmark score looks good while your visual-spatial reasoning is hollow. That breaks down in the real world where detectors are imperfect, boxes are loose or tight, reconstructions are noisy, and object sizes are atypical — a large cabinet, an oversized window. The paper shows exactly that fragility, and argues we need both a stricter test and a model that cannot lean on the cheat.
The key idea, in plain words
Stop giving the model the size and shape of the answer. Give it only a dot.
Point-based grounding prompting means the location query is just one 3D coordinate, an x, y, z on the target object. No width, height, length, no aspect ratio, no volume. Imagine pointing with a laser pointer instead of drawing a tight 3D cage around the chair. The laser tells the model where to look, but not how big the thing is or what shape it has.
This does two things. First, as an evaluation trick, it strips the categorical shortcut and forces the model to extract semantics from visual features around that dot. Second, as a training principle, it encourages the model to build dense, genuinely visual representations rather than memorizing that boxes of a certain proportion equal kitchens cabinets.
To make that work the model cannot depend on detectors at all. Hence P3-VLM: a scene-centric model that anchors visual features directly to scene voxels, meaning small 3D grid cells with learned descriptors at fixed spatial positions, derived from 3D Gaussian Splats, and then reasons at three levels of granularity — local point, task-relevant region, and whole-room context. The claim is that removing the box shortcut during grounding training even helps ungrounded question answering and out-of-distribution generalization, because the vision encoder becomes load-bearing.
How it actually works, step by step
Start with the input. The scene is represented as 3D Gaussian Splats, meaning a set of soft 3D blobs with position, shape, color and language-aligned features that together render and describe the scene more richly than a bare point cloud. A pretrained sparse encoder called SceneSplat turns those splats into voxel features: each occupied grid cell gets a feature vector describing what is there, plus its 3D coordinate.
Those raw voxel features can be more than 10,000 tokens, too many for a language model. So P3-VLM sparsifies them. It average-pools on a uniform 3D grid down to at most 2048 tokens, then runs a stack of spatially aware self-attention layers where queries and keys are rotated by 3D Rotary Position Embeddings. In plain words, that is a way to bake pairwise 3D distance directly into attention scores, so the network knows which voxels are near each other without looking up a position table. The result is an enriched scene memory.
From there three complementary pathways make language-ready tokens:
Location-aware pathway: the where-dot. Given the query point, find the closest voxel center, take its raw feature, concatenate it with the normalized query coordinate itself, and pass that through a four-layer neural net to make a single location token. The idea is to jointly reason about what is there and where it is. If there is no location query, this pathway is turned off. Crucially it uses raw voxels, not smoothed ones, to preserve locality.
Context-aware pathway: the room summary. Eight learnable query tokens, meaning vectors the model learns to use as summary slots, cross-attend over the whole enriched scene with no language conditioning. They learn to capture room type, layout, extent — the background that lets the rest focus. The authors observe that without this, the task pathway attends broadly; with it, attention sharpens on query-relevant regions.
Task-aware pathway: the task filter. The user language query is first refined by self-attention. Then each scene voxel, tagged with 3D position, cross-attends to those text tokens, so each location can pull in relevant words. The update is added back as a zero-initialized residual with normalization, meaning it starts as no-change for stable training, then learns to highlight task-relevant voxels. That refined scene is then compressed through 128 learnable tokens into task tokens.
Training is in two stages with the same standard next-word prediction loss. Stage one trains only the location pathway on point-grounded captioning on Nr3D and ScanRefer, so the voxel-to-language mapping stabilizes. Stage two turns on all three pathways and jointly trains on captioning plus ScanQA general question answering and SQA3D situated question answering. The language backbone is Qwen3-VL-4B-Instruct tuned with LoRA, meaning small low-rank adapters rather than full fine-tuning, the 3D encoder stays unfrozen, and the pathways train from scratch. They train a point version on object centers and a box version on center plus yaw plus extent for comparison.
What they showed — results, honestly, with limitations
The leakage demonstration is stark. A blind Qwen model with no vision at all, fine-tuned only on box text like center, yaw, size, gets 36.4% on Nr3D and 38.1% on ScanRefer. Replacing boxes with an encoding of just the box as a visual feature gets 44.2%, almost matching the full object-centric LEO at 45.1%, suggesting LEO's 3D encoder adds almost nothing under box prompting.
Under the new point protocol after fine-tuning on points, the blind model falls to 18.0% on Nr3D and 18.4% on ScanRefer, LL3DA falls from 47.7% to 33.1% on Nr3D, GaussianVLM from 46.2% to 29.3%. Training on points does not close the gap — the authors read this as lack of dense visual capacity, not just prompt format.
P3-VLM does close much of it. On Nr3D caption class accuracy, P3-VLM with points gets 52.1%, with boxes 52.4%, beating LL3DA boxes at 47.7% and GaussianVLM boxes at 46.2-47.1%. On ScanRefer, P3-VLM points gets 56.1%, boxes 58.4%, beating baselines at 48.0-51.7%. Caption similarity via Sentence-BERT stays competitive, so fluency is not sacrificed. Class is judged by asking Gemini 2.5 Flash to extract the primary object noun, validated by human check on 5% with no errors found.
Points are not perfectly prior-free — height correlates with class like picture versus chair — but far less leaky. A geometry-only nearest-neighbor test gets 33.0% from full boxes versus 16.5% from xyz center, 17.8% from height alone, 8.1% from horizontal position alone. Only 3 of 38 classes reach 30% recall from centers versus 15 of 38 from boxes.
Robustness favors points. When boxes are shrunk or expanded to simulate detector error and noisy reconstructions, box models crash: LL3DA 47.7% to 28.6% and 33.4%, GaussianVLM 46.2% to 25.3% and 32.2%. P3-VLM boxes is most stable at 36.4% and 39.9%, and point queries degrade gracefully when jittered up to 0.9 times box extent or moved to the surface, dropping only about 5.5 points on Nr3D.
Generalist gains hold. On ScanQA exact-match refined, P3-VLM points gets 31.9% versus 30.9% for retrained GaussianVLM; on SQA3D it ties at 51.8-51.9% but jumps on BLEU-4 from 18.9 to 30.6; on zero-shot ScanNet++ counting out-of-distribution it gets 17.31% versus 9.3% for GaussianVLM and 1.5% for LL3DA, with points beating boxes by 1.89 points. Ablations show each pathway matters and a simplified matched baseline falls to 39.8% versus 52.1% on Nr3D points.
Limitations stated plainly: it requires 3D Gaussian reconstructions, an extra step; evaluation is perception-heavy, not dialogue-heavy; it depends on voxelized SceneSplat features, though a Sonata point-cloud variant is tested. The supplement also shows training on random surface points instead of centers pushes Nr3D to 58.8% and ScanRefer to 64.3%, because center-pooling inside large furniture like under a tabletop yields empty features — an important nuance that the main 52.1% number uses the weaker center protocol.
Why this might matter to me
For spatial reasoning in LLMs and VLMs, this is a cautionary diagnostic and a design pattern. If your grounding prompt includes extent, your model may be doing parametric classification — tall-thin equals door — rather than visual reasoning. Point prompting plus a dedicated location token plus separate global context is a clean way to force where without giving what, and the finding that harder point training improves even ungrounded QA and out-of-distribution counting suggests shortcut removal is a general recipe for stronger world models, not just a captioning trick.
For world models and 3D generation and scene representation, the geometry-oriented versus semantics-oriented contrast is useful. GaussianVLM bakes language into every Gaussian early, which helps semantic QA but weakens precise grounding; P3-VLM keeps geometry primary, fuses position early with 3D RoPE, and lets language reshape rather than select geometry. The fact that the same tri-pathway works with both SceneSplat splats and Sonata point clouds, trading grounding for language alignment, gives you a portable scaffold for grounding in whatever scene representation your generator or simulator outputs — splats, voxels, or points — with a click-like point interface natural for embodied agents.
For text-to-CAD and parametric geometry, the parallel is direct. CAD boxes and parameters also leak category through dimensions — a standardized bed, a door height — just like ScanNet boxes. The paper's residual-prior analysis and the surface-point bonus are immediately relevant: canonical centers carry height priors, interior pooling can land in empty space inside cabinets or under tabletops, while surface points anchor features to visible geometry and boost large furniture by 21.3 points. Future text-to-CAD grounding may want to separate location, extent, and surface attachment explicitly rather than predicting a box as one entangled token.
Terms worth knowing
3D vision-language model: a language model that also consumes a 3D scene to caption, answer, or reason.
Grounded versus ungrounded: grounded queries point to a specific object or place; ungrounded queries ask about the scene at large.
Object-centric versus scene-centric: object-centric first detects objects then encodes each; scene-centric encodes the whole scene at once without a detector.
Bounding box semantic leakage: the paper's term for boxes revealing category through size, aspect ratio and volume, letting models guess without vision.
Point-based grounding: querying with a single x, y, z coordinate instead of a box.
3D Gaussian Splatting: a scene representation as a set of soft oriented 3D blobs with color and features, good for rendering and dense understanding.
Voxel features: descriptors attached to small 3D grid cells at fixed positions, here from the SceneSplat encoder.
Tri-pathway: P3-VLM's three streams — location-aware for the queried point, task-aware for query-relevant scene filtering, context-aware for unconditional room summary.
3D RoPE: 3D Rotary Position Embedding, a way to encode relative 3D distances into attention by rotating queries and keys.
Nr3D, ScanRefer, ScanQA, SQA3D: ScanNet-based benchmarks for referring expressions, dense captioning, general QA, and situated embodied QA.
Class accuracy versus Sim: whether the caption names the right object class versus Sentence-BERT embedding similarity to the reference caption; the paper argues n-gram metrics like CIDEr and METEOR hide wrong-class fluency.
EM-R: refined exact-match accuracy for QA; OOD: out-of-distribution, here unseen ScanNet++ scenes; LoRA: low-rank adaptation, efficient fine-tuning of a frozen language model.