The one-paragraph version
This paper asks a deliberately stubborn question about how we build multimodal models: do you actually need to retrain the language model to teach it a new sense like 3D? The usual recipe is to take a frozen 3D encoder that turns point clouds into features, learn a small translator called a projector that turns those features into language-model-friendly tokens, and then also fine-tune the language model itself on 3D instructions. The authors show, for 3D object understanding, that second part is unnecessary. Across three language backbones, training only the small projector — while leaving the language model completely frozen — matches existing 3D systems on object naming and on detailed captioning, and matches their own versions where they also fine-tuned the language model with the same data and same 16-hour computer budget. Projector-only training also runs about twice as fast, sees about twice as many examples in the same wall-clock time, and by definition cannot damage what the language model already knew, whereas their jointly fine-tuned versions forgot a lot, especially on instruction-following, math, and code.
The problem they're solving, and why it matters
A multimodal large language model, meaning a language model extended to accept another input type like images, audio, or 3D, is normally built from three parts: an encoder that already knows how to read that other modality, a language model backbone that already knows how to read and write text, and a projector in between that learns to translate.
The standard practice is two-phase. First align the projector, then unfreeze and adapt both projector and language model on instruction data. That second phase is expensive and risky. Expensive because backpropagating through a multi-billion-parameter language model costs memory and time. Risky because of drift, meaning the language model gets worse at things it used to do well — sometimes called catastrophic forgetting — after narrow fine-tuning.
For 3D this matters practically. 3D instruction data is relatively small and stylized compared to web text: lots of “What is this?” and “Caption this 3D model in detail.” If adapting the whole language model on that narrow distribution breaks general chat, reasoning, coding, or visual question answering, you have traded a general assistant for a 3D specialist. The authors test whether that trade is even needed.
The key idea, in plain words
Leave the brain alone and just train the adapter cord.
Think of the frozen language model as a very capable colleague who only speaks text-token, meaning little numerical vectors that represent words and pieces of words. Think of the 3D encoder as an observer who sees a chair as 8,192 colored dots in space and summarizes what it saw. The projector is an interpreter standing between them. Its whole job is to learn to utter the right sequence of made-up word-like vectors — in this paper 513 vectors per object — so that when the frozen colleague “hears” them wrapped in markers that say “a 3D object starts here and ends here,” it naturally answers correctly.
The authors’ hypothesis, stated as a hypothesis rather than a proof, is that this is a lot like prompt engineering. You do not need to rewire the colleague to get a new behavior; you just need to find the right prompt. Here the prompt is continuous and learned from 3D data, not hand-written English, but the effect is the same: elicit the right answer from knowledge the language model already has.
If that works, you get modularity. The same frozen language model could in principle host many independent interpreters — one for point clouds, one for images, one for audio — without them interfering with each other.
How it actually works, step by step
Start with a colored point cloud, meaning a list of points where each point has a 3D position plus a color. In this work every object is represented as 8,192 points with six numbers per point: x, y, z, red, green, blue. Positions are normalized to fit inside a unit sphere, colors are scaled to a standard range. A table leg and a tabletop are just different clusters of dots.
Feed that to a frozen 3D encoder called Point-BERT, meaning a transformer previously trained with image-text-point contrast to be language-adjacent. Concretely, it chops the cloud into 512 local patches by picking spread-out centers and gathering neighbors, embeds each patch with a tiny point network, then runs self-attention. The output is 513 feature vectors, 512 patch summaries plus one overall summary, each 384 numbers long. The encoder is never updated.
Translate with a small projector, here a three-layer neural network that maps 384 numbers to 1,024 to 2,048 to the language model’s word-vector size — 4,096 for the larger backbones, 2,560 for the smaller Qwen. It uses a smooth activation function called GELU between layers. This is the only thing trained in projector-only mode, roughly 8 to 11 million parameters depending on backbone, tiny next to billions.
Wrap those translated vectors as pretend words between two special markers and prepend them to the user’s question. For example: system instruction plus markers plus translated 3D tokens plus “What is this?” The frozen language model, which can be Llama-3.1-8B-Instruct, meaning a text-only 8-billion-parameter instruction-tuned model, or Qwen3.5-4B or Qwen3.5-9B, meaning vision-language models that already handle images, then predicts the answer word by word as usual. Only answer words count for training loss, a standard next-word prediction objective where prompt words are ignored.
For comparison, build a jointly trained twin. Same encoder, same projector starting point, same stream of training examples in the same order, but also update low-rank adapters inside the language model, meaning small trainable matrices added to every linear layer as an efficient way to fine-tune without updating all billions of weights. Train both regimes for the same wall-clock budget: 16 hours on one A100 80GB graphics card, saving checkpoints at 2, 4, 8, 12, and 16 hours. The training data is the same mix of brief one-line captions and complex long captions, conversations, and part-referring questions from PointLLM-V2, restricted to Objaverse objects, about 1.37 million rows over about 661,000 unique objects, sampled mixed together rather than in two curriculum stages.
Evaluate two ways. For new 3D ability, use generative classification, meaning the model must write out “this is a chair” with no fixed multiple-choice list, judged later by another large model, on ModelNet40, Objaverse held-out, and OmniObject3D, each under an instruction prompt like “What is this?” and a completion prompt like “This is an object of.” And use open-ended captioning on Objaverse, judged against eight rendered views per object for correctness points and hallucination points, combined into precision. For damage to old abilities, strip the adapters back into the base weights and run standard language, vision, and spatial-reasoning suites.
What they showed — results, honestly, with limitations
First, projector-only is competitive with published systems. All three projector-only models beat the original PointLLM-7B and 13B on the authors’ re-run with their judge, in this case GPT-5.6 Luna for classification and an ensemble of three frontier judges for captioning. For example on ModelNet40 instruction-style prompts the projector-only Llama reaches about 62% accuracy versus about 55-58% for PointLLM, and on Objaverse captioning the Qwen-based projector-only models reach around 77-78% precision from that judge versus about 67% for PointLLM. The appendix shows they are also broadly in the range of newer PointLLM-R and MiniGPT-3D. The paper is clear that it could not compare to PointLLM-V2 directly because those weights were not public.
Second, projector-only is competitive with joint training under the same time budget at almost every checkpoint, especially on ModelNet40. Because it runs about twice as fast — for instance about 27,000 versus 12,500 examples per hour for the Llama pair — it sees many more examples in 16 hours. When plotted against examples seen rather than hours, the picture is mixed: projector-only looks more example-efficient on ModelNet40 but less so on Objaverse, though the authors note the joint runs saturate earlier while one projector-only run was still climbing. Neither regime exhausted the data pool, so this is a fixed-compute comparison, not a fully converged comparison.
Third, joint training hurts the backbone. The Llama backbone collapses dramatically after 3D fine-tuning: instruction-following, grade-school math, and code generation scores fall to near zero in places, and outputs become incoherent on open-ended questions. The Qwen backbones degrade less on language but show clear instruction-following failures on open-ended vision and spatial questions, like answering a number question with a caption or answering a coordinate question with a description. Curiously, multiple-choice vision scores go up after joint training, but the authors argue convincingly this is misleading: those tests score by renormalizing probabilities over just the answer letters, so they do not test whether the model can still generate a proper answer.
Limitations to keep in mind: only 3D point clouds were tested, only one frozen encoder, only 16 hours, and only one main hyperparameter setting plus a small sweep. The Qwen4B pair shows an odd dip before 16 hours that the authors say might be seed or tuning noise, without proving it. The captioning advantage for Qwen over Llama is hypothesized to come from Qwen’s prior caption-like training, not demonstrated. And all 3D scores depend on model-as-judge evaluation, which can shift with judge choice.
Why this might matter to me
If you care about spatial reasoning in language and vision models, the drift results are a warning and a tool. Joint 3D tuning broke exactly the behaviors spatial assistants need — following formatting instructions, outputting numbers and coordinates, not rambling into descriptions. Freezing preserves those behaviors for free, which suggests testing any new spatial head projector-first before allowing backbone updates.
If you care about world models, meaning internal models that predict geometry, persistence, and affordance from sensory input, the modularity claim is the exciting part. A frozen reasoner plus independently trained sensory projectors would let you add depth, touch, or motion without retraining the core or risking interference between modalities. This paper only demonstrates the 3D branch, but it is a concrete existence proof for that architecture.
If you care about text-to-CAD and parametric geometry, where outputs must be exact — sketch extrusions, constraints, dimensions — instruction fidelity matters more than vibe. The finding that narrow 3D caption data degrades code and math performance under joint training predicts similar damage if you fine-tune a code-capable model on loose shape descriptions. A frozen code-savvy backbone with a learned geometric projector is a safer starting pattern, and classification-style probes like “what primitive is this, what are its parts” can be added before attempting exact program generation.
If you care about 3D generation and scene representation, the encoder-projector split clarifies where geometry lives. Here geometry understanding lives largely in the projector mapping, not in rewired language weights — removing adapters from jointly trained models still left above-baseline 3D scores. That suggests investing in better encoders, better 3D tokenization, and richer part-level and view-consistent supervision may pay more than larger language fine-tunes.
Terms worth knowing
Multimodal large language model: a language model extended to take a non-text input, here a 3D point cloud, alongside text and to generate text answers.
Encoder: the frozen perception module that turns raw input into features. Here Point-BERT turns 8,192 colored points into 513 vectors.
Projector: the small trainable translator that maps encoder features into the language model’s word-vector space. Here a three-layer network with about 8-11 million parameters.
Backbone: the large pretrained language model that does the reasoning and writing. Here Llama-3.1-8B-Instruct or Qwen3.5-4B / 9B.
LoRA adapters: small low-rank matrices added inside a frozen backbone to approximate fine-tuning cheaply. Updated only in the paper’s joint-training baseline.
Drift / catastrophic forgetting: loss of previously learned abilities after fine-tuning on new data. Here collapse on instruction-following, math, and code after 3D tuning.
Generative zero-shot classification: asking “What is this?” and letting the model write free text, later mapped to a category, rather than picking from a list. Tests whether 3D understanding emerges in language.
Correctness, hallucination, precision for captioning: counts by a judge of correctly stated visual aspects versus contradicted aspects, combined as correctness divided by correctness plus hallucination over the test set.