The one-paragraph version
Today's powerful vision-language models, often called VLMs, are excellent at describing and reasoning about flat 2D images, but they cannot reliably create, edit, or measure 3D shapes when asked directly. This paper introduces 3D-Prog, a system that turns an off-the-shelf 2D VLM into a 3D programmer without any retraining. It does this with two tricks: Canonical Coordinate Framing, which means showing the 3D asset to the VLM as six straight-on views stamped with the same visible X, Y, and Z axes, ruler marks, and origin point; and Task-Adaptive Feedback, which means letting the VLM write Python code to analyze its own work and look at the resulting plots and pictures in a loop. A second VLM acting as a Judge checks each attempt and tells the first one what to fix. With only these visual and code scaffolds, the same system handles completion of missing parts, converting noisy real scans into clean simulation models, finding boxes around objects, splitting objects into parts, editing materials, answering questions about scenes, and building new scenes from text.
The problem they're solving, and why it matters
The 3D field is fragmented. One specialized model does part segmentation, another does text-to-3D, another does detection, and each needs lots of expensive 3D labels and expert tuning. That makes it hard to scale to new open-vocabulary tasks like "find the small table in this laser scan" or "make this scan simulation-ready."
Meanwhile 2D VLMs trained on billions of image-text pairs generalize very well to new language and new pictures. But their sense of space is implicit and flat. They guess in 2D pixels, not in metric 3D meters. If you simply prompt them for 3D coordinates, the authors find the outputs are unstable: axes flip, scale drifts across views, and shapes float with no consistent reference.
Attempts to train native 3D VLMs that take point clouds or 3D features directly have the opposite problem: 3D-text data are small and narrow, so those models do one thing well but do not generalize broadly like 2D VLMs do.
The paper asks: instead of retraining for 3D, can we keep the strong 2D VLM frozen and give it explicit visual grounding plus a way to check its own work, so it can program in 3D reliably?
The key idea, in plain words
Think of hiring a smart assistant who has never used a ruler or graph paper. If you just say "tell me how wide this table is," they will guess wildly. If you first put the object on graph paper with clearly labeled axes and tick marks, show it from front, top, and side, and always use the same corner as zero, suddenly they can read off numbers consistently. That is Canonical Coordinate Framing, often shortened to CCF.
Now imagine the same assistant can also run experiments. They do not have to get it right in one shot; they can write a little script to plot "how many points are at each height," look at that histogram, notice "ah, tabletops create a spike at 0.8 meters," tighten their guess, and try again. And a critical colleague looks over their shoulder comparing their draft in cyan to the target in red and says "still shifted left." That self-checking loop is Task-Adaptive Feedback, often shortened to TAF.
Together, the graph paper grounds the input and output in one shared 3D space, and the experiment-plus-critic loop lets the VLM break out of pure geometry into whatever the task needs — material plots, clustering analysis, layout checks. The VLM never learns new weights; it just writes code that builds and edits scenes in a standard format.
How it actually works, step by step
1. Put the 3D into a 2D picture the VLM can read: CCF. A 3D asset, meaning a point cloud, mesh, or scene description, is rendered as one composite image with six orthographic projections. An orthographic projection means a straight-on view with no perspective shrinking, like an engineering drawing: front, back, left, right, top, bottom, plus an extra perspective view for context. Every view shares the same visible, labeled X, Y, and Z axes, metric tick marks showing real distances, and a fixed origin tied to the object's bounding box. The VLM is then asked to output 3D numbers in that same coordinate system. The paper shows what fails without this: no visible axes leads to orientation ambiguity; no tick marks leads to inconsistent scale; perspective-only leads to metric distortion; text-only 3D numbers with no visual frame lead to floating references.
2. Let the VLM program in Universal Scene Description. Universal Scene Description, often shortened to USD, is Pixar's open hierarchical format for describing scenes as a tree of objects, transforms, and materials. It is text-like code the VLM can write. Conditioned on the CCF image and a natural-language instruction, the programmer VLM produces two kinds of Python: semantic interpretation, and executable OpenUSD code that creates or edits the USD stage. The code runs in a Python environment with common scientific libraries.
3. Let the VLM invent its own diagnostic plots: TAF. In each iteration the VLM also writes analysis code tailored to the task. For example, to "detect the table," early iterations plot a density heatmap of points at typical table height and a height-distribution histogram to find candidate footprints. Later iterations zoom into that footprint and use the cumulative height curve to reject noise and output a final 3D bounding box with minimum and maximum corners. For a teapot segmentation task, the paper walks through using radial distance from the center plus height thresholds plus a clustering algorithm called DBSCAN, which means grouping nearby points into blobs, to separate spout from handle. These plots are rendered and fed back as images alongside the updated CCF views.
4. Add an independent Judge and iterate. A separate Judge VLM sees the comparative overlay — the current draft in cyan over the reference in red when a reference exists, across six views — plus the TAF plots and the original task. It returns a structured verdict: a yes-or-no completion flag, a confidence score, and text listing errors and suggestions. That feedback is appended to the prompt for the next round. The loop runs until the Judge approves or up to 20 iterations are reached. The reported average is about 11 iterations and about 22 seconds per iteration, dominated by VLM calls. Total time ranges from about 2 minutes for question-answering to about 8 minutes for fine segmentation.
What they showed — results, honestly, with limitations
The authors test on both synthetic assets from Objaverse-XL and Pixar's kitchen set and real scans from Articulate3D and ScanNet++, across generation, understanding, and reasoning. Each task score is a mean over only five distinct assets, so sample sizes are small, and editing and composition use a user study with four evaluators scoring 0 to 5 on five criteria each. With that caveat:
For generation, 3D-Prog sharply lowers Chamfer distance, which means the average surface mismatch between prediction and truth where lower is better. On object completion, baselines score 1.17 for Gemini 2.5 Pro, 0.51 for GPT-5.2, 0.42 for Claude Sonnet 4.5, versus 0.19 for 3D-Prog with Claude and 0.16 for 3D-Prog with Kimi K2.5. On Scan2Sim, which means converting a noisy real scan into a simplified simulation-ready asset, all standalone 2D baselines fail to produce valid USD at all, either overflowing context on dense meshes or collapsing geometry, while 3D-Prog scores 0.08 to 0.09.
For understanding, detection and instance segmentation roughly double: for example, instance segmentation mean Average Precision goes from 0.21 to 0.42 with Claude, and semantic segmentation mean Intersection over Union, which means overlap between predicted and true parts, goes from about 0.48 to 0.53. Object editing user scores go from 1.4 for Gemini/GPT and 4.0 for Claude to 4.8 for 3D-Prog.
For reasoning, articulation axis error, which means angular error in the predicted hinge or slider direction where lower is better, drops from near 90 degrees for baselines — essentially random — to 3.5 degrees with Claude backbone and 0.64 degrees with Kimi backbone, with motion type correct every time. Compositional scene scores favor 3D-Prog, which evaluators describe as cleaner, non-intersecting, and correctly scaled, versus baselines with detached parts and bad proportions.
Ablations show both parts matter: removing CCF collapses scene detection from 0.38 to 0.01 and worsens completion from 0.19 to 0.33; removing TAF removes iterative correction and worsens completion from 0.19 to 0.58. Robustness tests with shifting or rotating the input show 3D-Prog holds at 0.02 Chamfer under translations and small rotations while the baseline degrades up to 22 times worse, though both degrade at a large 45-degree rotation. Against a trained 3D VLM called GaussianVLM, training-free 3D-Prog scores higher on situated Q&A accuracy, 60% versus 50%, and slightly higher on captioning.
Limitations are stated plainly by the authors and visible in the supplement: performance inherits the base VLM's coding ability — among tested open models only Kimi K2.5 could reliably write valid USD, while Llama 4 and Qwen3-VL variants failed; the Judge checks convergence, not guaranteed correctness, and can accept suboptimal results; CCF is limited by VLM image context for very large or highly detailed scenes; failures include invalid code, over-simplified "lazy" geometry, and occasional hallucinations where correct analysis is not transferred into output, such as mislabeled legend names retained despite correct shapes.
Why this might matter to me
For spatial reasoning in LLMs and VLMs, this is evidence that explicit canonical framing may be a prerequisite, not an optional aid. It suggests you can get metric 3D behavior — distances, containment, adjacency across turns — from a 2D model by externalizing the coordinate system as visible pixels rather than hoping it emerges internally. The robustness to translation is especially relevant to world models: a persistent Euclidean frame plus execution feedback behaves like a lightweight world state the model can read and write.
For world models and embodied reasoning, the TAF pattern is reusable: let the model generate its own bird's-eye heatmaps, height histograms, and cluster overlays as a scratch space, then condition the next action on what it sees. The table-detection walkthrough is essentially a learned perception program using classical tools under VLM control.
For text-to-CAD and parametric geometry, the choice of USD as a lingua franca matters. All tasks become "write and patch hierarchical construction code until rendered overlays match." That decomposition into interpretable, executable steps — measure, create primitive, transform, validate in six views — maps directly onto parametric CAD workflows where primitives must be axis-correct and proportioned, and where rotated cubes must become true meshes with ordered translate-scale-rotate transforms. The paper's modeling rules and failure modes around invalid USD are directly informative.
For 3D generation and scene representation, Scan2Sim and composition results show a path to simulation-ready, intersection-free scenes with realistic scale and materials from language, without task-specific training. The cost is iteration time and reliance on a strong coder VLM, but the modularity means improvements in base VLMs should transfer immediately.
Terms worth knowing
VLM: Vision-language model. A model that takes images plus text and produces text, here used to produce code.
2D vs 3D VLM: A 2D VLM is trained on flat images; a 3D VLM is trained with explicit 3D inputs like point clouds. This paper keeps the 2D model frozen and scaffolds it.
Canonical Coordinate Framing (CCF): The paper's visual grounding method: six orthographic views sharing visible labeled axes, metric ticks, and a fixed origin.
Task-Adaptive Feedback (TAF): The iterative loop where the VLM writes analysis code to make custom diagnostic images and uses them plus Judge comments to refine its 3D code.
3D-Prog: The full framework combining CCF images, programmer VLM, Python execution, TAF plots, and Judge VLM.
USD / OpenUSD: Universal Scene Description. A hierarchical, composable file format for 3D scenes that both VLMs and graphics tools can read, edit, and render.
Orthographic projection: A view with parallel projection lines and no perspective foreshortening, preserving true relative scale for measurement.
Gauge / coordinate-frame ambiguity: The problem that 3D can be described in infinitely many shifted, rotated, or scaled frames unless you fix axes, origin, and units.
Judge module: A second VLM that scores each draft as complete or not with confidence and error lists, driving early stopping and refinement.
Chamfer distance: A geometric error metric measuring average nearest-surface distance between two shapes. Lower means better fit.
mAP and mIoU: Mean Average Precision for detection and instance segmentation, and mean Intersection over Union for semantic segmentation. Higher means better overlap with truth.
METEOR: A text metric for caption quality used on the Nr3D object-captioning benchmark. Higher is better.
DBSCAN: Density-Based Spatial Clustering of Applications with Noise. A classic algorithm that groups densely packed points and marks sparse points as noise, used here to split spout versus handle.
Scan2Sim: Converting a noisy real-world scan into a clean, simplified, simulation-ready model.
Articulation prediction: Predicting how a part moves, such as its motion type and hinge axis direction, given a marked movable component.