All summaries · 2026-10-07

PrimitiveCAD: An LLM-Based Point-to-CAD Reconstruction with Primitive-Aware Tokenization and Operation Alignment

Jian Gao, Kailin Bi, Jiamin Xu, et al.

arXiv:2610.08698 · abstract · pdf · summarized by meta/muse-spark-1.3-contributor on 2026-10-07

The one-paragraph version

PrimitiveCAD is a three-stage system that turns a 3D point cloud — a cloud of measured surface points — back into an editable, executable CAD program written in CadQuery, a Python library for parametric solid modeling. First it teaches a 3D encoder to see CAD structure by predicting surface normals and which modeling primitive each point came from. Then it fine-tunes a large language model, in this case from the Qwen2 family, to generate Python code from those point tokens, adding a new operation-level check that the mix of CAD operations in the prediction roughly matches the ground truth. Finally it polishes the language model with reinforcement learning that rewards both overall volume overlap and preservation of sharp CAD details like straight edges, arcs, and circles. On the DeepCAD and Fusion360 benchmarks the authors report state-of-the-art scores for code validity, shape overlap, and surface accuracy.

The problem they're solving, and why it matters

Computer-Aided Design, meaning software for precise engineering shapes, normally requires an expert to draw 2D sketches and then apply 3D operations such as extrude, cut, or join to make a solid. The paper is about the reverse direction: you already have geometry as a point cloud, for example from a 3D scan, and you want the editable construction history that would recreate it.

That history matters because a raw mesh or point cloud is hard to edit — you cannot easily change a hole diameter or wall thickness. An operation sequence or Python program is parametric, meaning it is defined by meaningful dimensions and steps you can adjust later.

Recent large-language-model approaches treat this as a generic translation problem: encode points into tokens, meaning chunks of numbers the language model can read as context, and then predict code one token at a time. The authors argue this misses what makes CAD special. Generic 3D encoders like PointNet or Point Transformer do not know about sketch faces, extrusion sides, cylinders, or boxes. Standard next-token training only cares if each word of code looks plausible locally, not if a whole circle or extrusion is missing. And standard reinforcement-learning rewards only care about overall volume overlap, so they can ignore a small but functionally important hole or fillet. The result is code that runs but loses fine structure.

The key idea, in plain words

The central claim is: put CAD awareness into every stage of learning, not just the final output.

Think of rebuilding a piece of furniture from a pile of measurements. First, you want an assistant who can look at points and say “this patch is a flat top face, this patch is a round leg, this patch is the side wall of an extrusion.” Second, when you write the instruction manual, you want someone checking not just spelling but the parts list — “the original used three circles and two rectangles, did you include them?” Third, when you refine your work by trial and error, you want credit not just for getting the overall bulk right but for getting the edges and holes in the right places.

PrimitiveCAD implements those three checks as: primitive-aware tokenization for the point encoder, an operation-frequency alignment loss during supervised fine-tuning, and a feature-line reward during reinforcement learning. Supervised fine-tuning here means training the language model on correct point-cloud to code examples. Reinforcement learning here means letting the model try several codes, running them, and reinforcing the tries that score higher.

How it actually works, step by step

Stage 1: Learn to see CAD primitives in points.

The input is a point cloud sampled from a CAD model. The backbone is Point Transformer v3, abbreviated PTv3, a pre-trained network for general 3D point understanding that is normally used for indoor or outdoor scene segmentation.

The authors fine-tune it with two extra prediction tasks per point. One is normal estimation, meaning which direction the surface faces at that point, trained to ignore whether the arrow points inward or outward. The other is primitive-instance segmentation, meaning labeling each point with which construction piece it belongs to.

For labels they define four primitive families: sketch faces, extrude side walls, cylinders, and boxes, with up to eight instances of each, for 32 possible labels total. Ground truth comes from parsing the true CAD sequence and testing each point against geometric rules — for example, is it on the base or top plane versus the barrel side, or at the right radius and height for a cylinder.

Because instance order does not matter — calling something “cylinder 0” versus “cylinder 1” is arbitrary — they match predictions to truth with the Hungarian algorithm, a classic method for finding the best one-to-one assignment, and then measure overlap with a relaxed Intersection over Union, a soft version of overlap that works with probabilities rather than hard yes-or-no masks. After this stage the encoder outputs compact geometric features the paper describes as 64-dimensional per point, which are then sampled down to 256 representative points with farthest point sampling, combined with Fourier positional encodings — sine and cosine transforms of the xyz coordinates that help preserve precise location — and linearly projected into language-model tokens.

A visualization in the paper suggests these fine-tuned features separate different CAD parts more cleanly than the original PTv3 features, though this is qualitative.

Stage 2: Teach the language model to write CAD code, with a global parts-list check.

The 256 point tokens replace a placeholder span in the input to the Qwen2 language model, so geometry becomes context for code generation.

Training combines two losses, meaning two error signals. The first is the usual autoregressive cross-entropy loss: given previous code tokens and point tokens, how surprised is the model by the next correct token. This enforces local syntax.

The second is the new operation alignment loss. In words: count how often core CAD operators appear in the true code by string matching, estimate a soft, differentiable count in the prediction by summing operator probabilities across positions and token variants, and then penalize mismatch in those frequency vectors with relaxed overlap. For example, if truth has two rectangles and three circles but the model effectively predicts different totals, that is penalized even if each individual token looked plausible. The paper combines them as a weighted sum with a weight on the operation term, but the exact operator set and weight value are not fully specified in the main text.

Stage 3: Refine by execution with detail-sensitive rewards.

After supervised training the model can still be stochastic, meaning the same input can lead to different outputs on different tries. The authors use Group Relative Policy Optimization, abbreviated GRPO, a reinforcement-learning method that samples a group of candidate codes in parallel, runs them, and favors the relatively better ones without needing a separate reward model.

Each candidate that runs successfully into a valid solid gets a reward that mixes two parts: volumetric Intersection over Union, meaning how much the predicted solid and true solid overlap in 3D space, weighted 0.6, plus a feature-line similarity score weighted 0.4. Feature lines here means sharp CAD curves extracted from the mesh and classified as lines, arcs, or circles, then matched between prediction and truth again with Hungarian matching based on endpoints, centers, lengths, and directions. Code that fails to execute gets a fixed large negative reward of minus ten. This stage does not need paired code labels, so the authors say it can use 3D-only data and learn from more diverse shapes, using Fusion360 data at this stage.

What they showed — results, honestly, with limitations

The authors evaluate on DeepCAD, about 16 thousand real designs used for supervised training, and Fusion360, a smaller set of complex realistic models used especially for reinforcement learning, plus about one million models from the CAD-Recode collection in larger-data runs. Metrics are Intersection over Union for volume overlap where higher is better, invalidity rate for share of codes that fail to build a solid where lower is better, and mean and median Chamfer distance for surface error measured from 8,192 sampled points where lower is better.

Quantitatively, under single-pass evaluation without picking the best of many tries, their method is reported as best in the table on both test sets. For example, when trained only on DeepCAD, they report on the DeepCAD test set 86.0 overlap and 0.75 mean surface error with 0.1% invalid, versus the prior Cadrille baseline at 79.4 overlap and 3.43 error with 0.4% invalid. On Fusion360 they report 82.0 overlap and 0.81 error versus 63.7 and 7.61 for that baseline. With combined DeepCAD plus CAD-Recode training they report 90.9 overlap on DeepCAD and 84.9 on Fusion360 with very low invalid rates. Qualitative figures show better holes and right angles than CAD-Recode and Cadrille, but those figures pick the best of 10 tries per method, so they show potential rather than typical single-try behavior.

A direct supervised versus reinforcement comparison on Fusion360 is striking: their model without reinforcement learning already matches the reinforcement-tuned Cadrille on overlap, and with reinforcement learning reaches 86.6 overlap and 0.06% invalid in their table. Ablations suggest both new pieces help: removing PTv3 primitive fine-tuning or removing the operation loss lowers overlap and raises error, and removing the feature-line reward hurts worst-of-10 robustness in particular. They also report that 1,024 input points is a good trade-off, with 512 clearly worse and 2,048 or 4,096 giving only marginal changes, and that GRPO outperforms the alternative Dr. CPPO preference strategy in their test.

Limitations to keep in mind: the authors state the framework focuses on component-level generation and does not yet support complex multi-module assemblies. Key details about feature-line extraction and training hyperparameters are deferred to supplementary material not included here, so the reward is hard to reproduce from the main text alone. The paper also mentions generalization to fillets, chamfers, and shells on a CADEvolve dataset and a comparison to CAD-MLLM, but without numbers in the main text, so those claims are unsupported here. As with all benchmark reports, results depend on dataset splits and CadQuery executability and may not transfer directly to noisy real scans.

Why this might matter to me

For spatial reasoning in LLMs and VLMs, this is a concrete case where generic vision or point tokens are not enough; adding an explicit geometric parsing task — normals plus which primitive a point belongs to — gives the language model a more structured spatial context to reason over. That pattern could extend to other spatially grounded code or plan generation.

For world models, the lesson is to supervise at multiple levels of abstraction: local token correctness, global operation frequency, and geometric detail fidelity. A world model that must predict both actions and their physical consequences could similarly benefit from counting expected high-level events in addition to predicting low-level steps.

For text-to-CAD and parametric geometry, the output choice matters: executable CadQuery code preserves editability and design intent in a way meshes do not. The operation-alignment idea is directly reusable — if you generate CAD from text, you could check that requested holes, ribs, or patterns actually appear in the predicted operation mix.

For 3D generation and scene representation, the feature-line reward points to a gap in purely volumetric metrics. Two shapes can have high volume overlap yet differ in functionally critical edges. Explicitly matching lines, arcs, and circles offers a lightweight way to keep generative models honest about sharp structure, and the reported reduction in worst-case failures after reinforcement learning is especially relevant if you need reliable generation rather than just good average samples.

Terms worth knowing