The one-paragraph version
This paper asks what happens before a CAD model exists, when a person only has a rough hand-drawn sketch and a few sentences in mind. The authors build HiCAD, both a new dataset and a matching method with the same name, that links those messy human inputs directly to executable parametric CAD programs written in CadQuery. Their framework first jointly fine-tunes a vision-language model on four sketch-and-text to code tasks so sketch understanding and text understanding help each other, then refines it with reinforcement learning that rewards real 3D correctness, not just getting the next code token right. Across all four tasks, and especially on real sketches drawn by volunteers, this two-stage system produces more valid and geometrically accurate models than general vision-language models and recent CAD-specific baselines.
The problem they're solving, and why it matters
Parametric CAD modeling, meaning building an editable 3D shape by issuing a sequence of operations like draw a profile, extrude it, cut holes, rather than just outputting a static mesh, is normally slow expert work. Recent AI work has mostly treated it as reconstruction: start from a clean rendering, photo, or 3D scan of something that already exists, and recover the modeling steps.
The authors argue the conceptual design stage is different and harder. There is nothing to copy yet. The input is incomplete and unstructured: for example, a wobbly cross-shaped outline with five circles, plus a sentence like "it is 199 units wide and 16 tall with symmetric through-holes." Sketches give intuitive layout and topology but are ambiguous about size. Text gives semantics and dimensions but is abstract about shape. Turning that combination into code that actually runs and makes the right solid is the gap they target.
Three difficulties make this hard in their framing. First is asymmetric alignment: sketches are sparse and noisy, text is abstract, and they do not line up neatly. Second is logical consistency: a tiny error in code can mean it will not compile at all, or a hole ends up in the wrong place and breaks the topology. Third is sparse feedback: the usual training loss that only checks whether each predicted word matches, called cross-entropy loss, does not know if the final 3D shape overlaps or connects correctly.
The key idea, in plain words
The central idea is to stop treating sketch-to-CAD and text-to-CAD as separate problems, and to stop rewarding only text similarity.
In plain terms: teach one model to handle all levels of human vagueness at once, from sketch alone to sketch plus complete description, so knowledge transfers. A model that has seen full descriptions learns what partial descriptions are missing. A model that has seen sketches learns what words like symmetric or through-hole should look like in space.
Then, after that imitation stage, let the model practice generating code, execute the code to build real 3D shapes, compare those shapes to the intended shapes in 3D space, and learn from that spatial feedback. In other words, judge the model not by "did you write similar code?" but by "did you build a solid that has the right parts, connections, and volume overlap, and does your code even run?"
To make this possible they also had to create data, because they say no public dataset previously contained hand-drawn sketches, textual descriptions, and CadQuery code together for the same objects.
How it actually works, step by step
Think of the work as dataset first, then two training stages.
1. Building the HiCAD dataset. They start from DeepCAD, a large collection of CAD modeling sequences, and reuse CadQuery code for those shapes created by the CAD-Recode project. CadQuery, meaning a Python library where you write code like make a workplane then sketch then extrude to get a solid, is their chosen output language because it is modular, interpretable by language models, and industrially usable.
For each shape they add two new inputs. For sketches, they execute the code to get the boundary representation, meaning the precise mathematical description of faces, edges, and how they join, keep only visible edges in a nominal view to mimic how people actually draw, project them to a 512 by 512 image, subdivide the lines and add small jitter to feel hand-drawn. They also collect 100 truly human-drawn sketches from volunteers for testing generalization. For text, they render an image, compute basic attributes like length, width, height and number of through-holes, feed code plus image plus attributes into a large vision-language model called Qwen3-VL-32B with a strict prompt to write one concise paragraph about geometry only, no color or material. That becomes the complete description. They then randomly delete sentences to simulate users who only specify part of the design, creating a partial description. After removing near-duplicates with image similarity, they keep 82,075 training, 4,536 validation, and 4,015 test samples.
2. Stage one: Cooperative Multi-Task Alignment, or CMTA. Start from a pre-trained vision-language model, specifically Qwen3-VL-4B-Instruct, freeze its visual encoder to keep vision ability, and fine-tune the rest to predict CadQuery code. But instead of training four separate models, they interleave four tasks: sketch alone to code, complete text alone to code, sketch plus partial text to code, and sketch plus complete text to code. The loss at this stage is still token prediction, essentially learning CAD grammar and cross-modal associations, but joint training is meant to map all intents into one shared modeling space.
For example, sketch alone forces geometric reconstruction from noise. Complete text alone forces semantic alignment from words to precise instructions. Sketch plus partial text forces multimodal completion, filling gaps.
3. Stage two: Spatial-Aware Reinforcement Learning, or SARL. Take the model from stage one as the starting policy, meaning the decision-maker that generates code given a query, and improve it by trial and error. For each input they sample 10 candidate programs, execute each with a Python interpreter, and score them with a spatial-aware reward with three parts explained in words first.
Topological consistency asks whether the parts connect the same way. They build a face adjacency graph, meaning a graph where each face of the solid is a node and shared edges are links, using OpenCascade, then measure graph edit distance, meaning how many edits it takes to turn the predicted connection structure into the true one. Closer structure gives higher reward.
Geometric consistency asks whether the solids occupy the same space. They decompose multi-body shapes into pieces, compute intersection volume and union volume, then use Intersection over Union, meaning intersection divided by union, as a smooth overlap score.
Validity asks whether the code runs at all. If invalid, they skip the spatial scores and give a fixed penalty, set to negative 0.1 in the paper, otherwise they combine topology weighted 0.2 and overlap weighted 0.8.
Optimization uses Group Sequence Policy Optimization, or GSPO. In words, rather than weighting each word separately as older token-level methods do, which is unstable for long programs, GSPO gives one shared weight to a whole program based on length-normalized likelihood, clips large updates for stability, and normalizes rewards within each group of 10 to compute advantage. Crucially they do this jointly across all tasks, not per task, to preserve multi-task ability.
What they showed — results, honestly, with limitations
The authors report Intersection over Union, meaning volumetric overlap where higher is better, Chamfer Distance, meaning average point-cloud distance between shapes where lower is better, and Invalid Rate, meaning share of programs that fail to execute where lower is better.
On synthetic test data, their full system beats both general models like GPT-4o, GPT-5-mini, Gemini-2.5, Claude-3.7, Qwen-VL-Max and CAD-specific models CAD-Coder and Cadrille on all four tasks. For example, they report overlap of 69.49% for sketch alone, 74.27% for complete text alone, 76.03% for sketch plus partial text, and 80.90% for sketch plus complete text, with invalid rates around 0.17% to 0.42%. The strongest baseline Cadrille is reported at 66.54% to 72.56% overlap but with invalid rates around 6.8% to 9.4%, so the paper emphasizes roughly an order-of-magnitude drop in invalid code. General models are far behind, often below 55% overlap. Performance improves as more information is added, which they interpret as successful fusion rather than naive concatenation.
Ablations support both stages. Replacing separate single-task fine-tuning with joint CMTA improves overlap, for instance from 66.61% to 72.39% on sketch plus partial text in their table. Adding SARL further lowers invalid rate and raises overlap.
On 100 real human sketches, the gap persists but absolute scores fall, showing the domain shift problem. They report 43.27% overlap for real sketch alone versus 35.72% for CAD-Coder and 17.74% for Cadrille, and 73.37% when complete text is added, with 0% invalid in that setting. General models are reported below 10% overlap on real sketches alone.
Limitations, many stated by the authors themselves, matter. Highly complex structures still fail. Cluttered or non-standard sketch viewpoints mislead the model. Four tasks do not exhaust how humans express intent. Training used only synthetic sketches, so real-sketch robustness is encouraging but tested on only 100 examples. Comparison to general models is also uneven by design: those models were not fine-tuned and were evaluated 2-shot on a 500-sample subset for cost reasons, while CAD-specific baselines were trained on HiCAD. The paper is clear about that protocol, but it means the general-model numbers are not a like-for-like training comparison. Details like exact GSPO defaults and the full sketch jitter settings are left to appendix or prior code, so full replicability is somewhat unclear.
Why this might matter to me
Given your interests in spatial reasoning in LLMs and VLMs, this is a concrete case where token matching fails and explicit 3D grounding helps. The model must infer depth, symmetry, extrusion direction, and hole connectivity from a flat sketch plus words, then produce steps that respect those constraints. The reward design is essentially a way to inject spatial reasoning after language pre-training.
For world models, the lesson is that execution plus comparison can serve as a lightweight world model: the CAD kernel tells the learner what is physically and topologically possible. The face-adjacency versus volumetric-overlap split is useful, separating what connects from where it sits, much like separating scene graph structure from metric geometry.
For text-to-CAD and parametric geometry, the dataset pipeline and choice of CadQuery as an editable program, not a mesh, directly match your focus. Partial descriptions are especially relevant: they formalize underspecification, where the model must complete missing dimensions while preserving editability.
For 3D generation and scene representation, the results suggest a path beyond single-modality reconstruction toward intent-driven generation from scratch. The multi-task finding that joint sketch-text training beats single-task training, and that joint reinforcement beats single-task reinforcement, hints that shared latent geometry transfers across input densities, an idea that could extend from single objects to scenes with layouts, constraints, and iterative refinement, which the authors list as future work.
Terms worth knowing
- Parametric CAD model: A 3D shape defined by operations and parameters, like sketch a profile then extrude 16 units then cut five holes, so you can edit values later, as opposed to a fixed triangle mesh.
- CadQuery: A Python library for writing such modeling steps as executable code. Running the code yields the solid.
- B-Rep or boundary representation: The precise engineering description of a solid by its faces, edges, vertices, and how they join. Used here for rendering and for topology comparison.
- Hand-drawn sketch or HDS: A rough 2D line drawing input. Synthetic means algorithmically wobbled from true edges. Real means drawn by volunteers.
- Complete versus partial textual description or CTD versus PTD: Full paragraph describing shape and dimensions versus that paragraph with random sentences removed to mimic incomplete user input.
- Cooperative Multi-Task Alignment or CMTA: Stage one joint fine-tuning on all four sketch-text tasks to learn shared intent-to-code mapping.
- Spatial-Aware Reinforcement Learning or SARL: Stage two refinement using rewards from executed 3D shapes, not just code text similarity.
- GSPO or Group Sequence Policy Optimization: The reinforcement algorithm used. In words, it compares a group of candidate programs for the same input and updates the model with one stable weight per whole program.
- IoU or Intersection over Union: Volumetric overlap between predicted and true solids, intersection volume divided by union volume. Higher means better geometric match.
- Chamfer Distance or CD: Average distance between sampled points on two shapes. Lower means closer surfaces. Papers often report mean and median.
- Invalid Rate or IR: Fraction of generated programs that fail to run or produce no valid solid. Lower means more reliable code.