All summaries · 2026-10-28

Zhengrong-Spatial-Aware 1.0: Advancing 3D Multimodal Intelligence

Yu Zhang, Xiaogang Wang, Yuhang Cheng, et al.

arXiv:2610.37216 · abstract · pdf · summarized by meta/muse-spark-1.3-contributor on 2026-10-28

The one-paragraph version

This report does not train a new foundation model. It builds and tests two practical recipes for teaching existing vision-language models to answer metric 3D questions about building scenes — things like how many buildings are there, how tall is that one, which footprint is largest, how far apart are two centers, and what is left of what. The first recipe fine-tunes a 3-billion-parameter model called HiSpatial-3B, which looks at both a normal color photo and an extra geometry layer that gives every pixel an estimated 3D position. The second recipe adapts the general-purpose Qwen3-VL models at 8B and 32B scale using a low-memory tuning method, without any extra geometry input. On their own custom building benchmark, the HiSpatial route jumps from about 1.7% to about 79.9% accuracy after fine-tuning, and also improves on a separate styled image set from 14.9% to 55.1%. The Qwen3-VL route improves from 20.7% to 45.3% for the 8B model and from 30.8% to 47.6% for the 32B model when target data is mixed with general instruction data. The main lesson is engineering rather than theoretical: explicit geometry helps a lot for metric questions, and mixing in general data helps prevent a specialized model from forgetting how to answer normally.

The problem they're solving, and why it matters

Vision-language models, meaning AI systems that take in an image plus a text question and produce a text answer, are now very good at naming what they see. They still struggle when the question requires measuring the physical world from a single flat image. A photo collapses depth, scale, and viewpoint into two dimensions, so asking for a building height in meters or a horizontal distance between centers is much harder than asking for a caption.

The project focuses on that gap for a building-centric dataset. The questions fall into about twelve types, grouped into counting, metric estimation such as height and pairwise center distance, finding extremes such as tallest or largest footprint, ordering such as left-to-right ranking, and relational reasoning such as farthest pair or relative direction. These matter because many real uses need numbers and relations in physical units, not just words like near or big — for example navigation, robotics, construction checking, or design review. The authors want a reproducible software stack that turns raw scenes, images, masks, camera poses, and question-answer pairs into trained checkpoints and comparable scores, rather than claiming a brand-new model architecture.

The key idea, in plain words

The key idea is to compare two complementary ways to get spatial competence, using the same task supervision and the same evaluation.

The first way is to give the model geometry explicitly. Imagine looking at a photo of an apartment block and also holding a second sheet where every pixel says I think this point is this far left, this high up, and this deep. That second sheet is what the paper calls a point map — a dense grid with three numbers per pixel for horizontal, vertical, and depth position. HiSpatial is built to fuse appearance from the color image with structure from that point map, so metric questions have something to ground to. In this work the point map comes from monocular geometry estimation, meaning depth inferred from a single image alone, using a method in the MoGe family that predicts shape without a second camera or lidar.

The second way is to rely on learned priors plus instruction tuning. Qwen3-VL is a strong general multimodal model that already saw spatial and 3D grounding data during pretraining. Here it gets no extra geometry layer. It is simply shown many examples of the target questions phrased as chat, and lightly adapted so it learns the domain vocabulary and answer style. The comparison lets the authors separate two failure modes: did we fail because we could not recover 3D shape, or because the language side did not learn the task format.

A second, cross-cutting idea is mixed-domain training. Instead of training only on building questions, they mix building data with general visual question-answering data, roughly one-to-one in the first HiSpatial setting. The intuition is like keeping up general conversation while cramming for a geometry exam — without general examples, the model learns the narrow answer template too well and loses flexibility.

How it actually works, step by step

In plain steps, the stack looks like this.

First, a shared data layer cleans the provided data. Raw scenes come with color images, styled variants, colored box overlays, instance masks that outline each building separately, camera pose information about where the view was taken from, per-scene annotations, and question-answer records in JSONL format, meaning one JSON object per line for easy large-scale processing. Each question gets a stable identifier that always points to the same image, so training, inference, and testing stay aligned. The paper mentions CAD and GLB-derived 3D assets are also available for future use, but the current results use estimated rather than true CAD geometry. The exact future use is left unclear.

Second, the HiSpatial branch adds geometry. Each image is run through a monocular estimator to produce that XYZ tensor described above, then the image, geometry, and conversation are packed together into WebDataset-style shards, meaning an efficient serialized format for streaming large training sets. Precomputing the point maps once avoids re-running geometry estimation on every training step and makes runs repeatable.

Third, HiSpatial-3B is fully fine-tuned starting from its released checkpoint, not from scratch. HiSpatial-3B itself builds on a PaliGemma 2 backbone, which combines a SigLIP vision encoder for turning image patches into features with a Gemma 2 language model for generating answers. The reported starting recipe uses a total batch size of 32 and saves checkpoints every 400 steps, later extended to 3,600 steps. The answers are trained with supervised fine-tuning, meaning the model is shown the correct answer tokens and taught to predict them one by one given the image, geometry, and question.

Fourth, the Qwen3-VL branch takes a different path. The same building questions are rewritten into a chat format with a human turn containing the question and an image placeholder, and an assistant turn containing the answer. Qwen3-VL has three parts: a vision encoder, a small connector that translates visual features into language tokens, and a Qwen3 language decoder. It also uses improved position modeling for space and time and a technique for injecting multi-level visual features. For adaptation the authors use QLoRA, meaning quantized low-rank adaptation. In plain terms: keep the big pretrained weights frozen in 4-bit low precision to save memory, and train only two tiny added matrices that nudge the behavior. In the lightweight recipe the vision encoder stays frozen while the language-side adapters learn.

Finally, both branches share inference and scoring. A runner loads a checkpoint, maps each test identifier to its image, writes one normalized prediction per line, and then a task-aware evaluator scores differently by type: exact match for counting, numeric parsing with a relative tolerance for heights and distances so 10.2 versus 10.0 can still count, and normalized semantic matching for open-ended direction or relation words. The same interface is said to also support other spatial models for comparison.

What they showed — results, honestly, with limitations

The clearest result is HiSpatial adaptation on the original building benchmark: overall accuracy goes from 1.73% before adaptation to 64.69% at 1,200 steps, 73.90% at 2,400 steps, and 79.89% at 3,600 steps. The very low starting score suggests the released checkpoint did not know this dataset's answer format at all, so most of the gain is learning both geometry use and format together. More encouragingly, on a separately collected styled split — the paper does not fully define what styling means beyond altered appearance variants — accuracy rises from 14.9% to 49.2%, 52.8%, and 55.1% at those same checkpoints, which points to some transfer beyond memorizing training scenes.

For Qwen3-VL with mixed-domain low-rank tuning, the 8B model improves from 20.74% to 45.27%, and the 32B model improves from 30.79% to 47.60%. Larger pretraining helps before tuning, but after tuning the two sizes end up much closer, which the authors present as evidence that the tuning pipeline works at both scales when general data is included.

As a sanity check that their HiSpatial code was working, they report 94.17% on CV-Bench-3D, 94.77% on CV-Bench Relation, 73.90% on EmbSpatial, and 78.05% on RoboSpatial in their own runs, with much less compute than the original HiSpatial training. They are explicit that this is an implementation check, not a full reproduction claim.

Limitations, stated honestly: this is an engineering report on a private custom dataset, not a general benchmark win. We are not given per-task breakdowns, confidence intervals, or the exact tolerance used for metric scoring. The Qwen experiment is described as 1,000 scenes split 900 for training and 100 for testing, giving roughly 10,800 training and 1,200 test questions because each scene has twelve questions, but the HiSpatial split sizes are less clearly specified. One important negative finding is that naively mixing free-form and multiple-choice supervision hurt free-form performance, while an 8-to-2 mixture recovered it — format balance matters, but only one ratio is reported. The paper also warns about narrow-domain over-specialization and motivates retention controls, but does not yet show a systematic study of mixing ratios, of estimated versus true CAD geometry, or of different visual prompts like boxes and masks.

Why this might matter to me

For spatial reasoning in language and vision models, this is a useful existence proof that single-image metric question answering can be substantially improved without training from scratch, and that explicit geometry plus format-aware evaluation does most of the heavy lifting. If you build world models — internal representations that predict how a scene looks and behaves in 3D — the point-map branch is directly relevant: it treats metric structure as an input modality alongside pixels, rather than hoping the language model invents depth. That hybrid of appearance plus structured geometry is a pragmatic middle ground between pure 2D pretraining and full 3D simulation.

For text-to-CAD and parametric geometry, the gap is instructive. The dataset already has CAD and GLB assets and camera metadata, but the current pipeline still uses estimated monocular geometry, not true parametric shapes with exact heights, footprints, and constraints. The jump from 55% on styled images to 80% on the original split suggests estimated geometry transfers only partially when appearance shifts. A natural next step for your interests would be to replace or supervise the estimated point map with true CAD-derived depth, normals, footprints, and dimensions, and to ask questions that require parametric answers like extrusion height, footprint area, or distance constraints rather than free text. For 3D generation and scene representation, the shared JSONL interface, WebDataset packing of image plus XYZ, and task-aware scoring for numbers versus relations offer a template you could reuse to benchmark whether a generative scene model actually preserves measurable structure, not just plausible looks.

Terms worth knowing