The one-paragraph version
Vision2CAD is a system that turns a single image of a part into an editable parametric CAD model, not just a look-alike shape. The core problem it attacks is that previous image-to-CAD systems can copy appearance but lose the hidden engineering logic: which face a new sketch sits on, where that sketch lives in its own 2D coordinate system, and how sketch lines stay glued to older edges when dimensions change. Vision2CAD solves this with an agent loop that pairs a vision-language model, meaning an AI that reads images and text, with a deterministic CAD kernel, meaning the exact geometry engine in FreeCAD that does the math. The model picks things by visible ID labels instead of guessing coordinates, draws in picture coordinates and lets the kernel translate to sketch coordinates, and explicitly ties new sketch points to projections of old edges. To train and test this, the authors built the Geometry Explicit Reference Dataset, called GERD, with 2,700 parts where every modeling step records commands, shapes, and stable IDs for bodies, faces, edges, and vertices. On their GERD test and a 100-part DeepCAD subset, they report the best shape scores and 100% build success, plus better survival after a 10% extrusion-depth edit.
The problem they're solving, and why it matters
Parametric CAD models are histories, not just final meshes. Think of a block, then a sketch drawn on its top face, then that sketch extruded into a boss, then a hole cut through the boss. If you later make the block taller, the sketch, boss, and hole should ride along because they reference the top face and each other. That ability to update while preserving design intent is what makes CAD reusable for analysis, optimization, and manufacturing. Earlier approaches break this chain in three places. First, geometry referencing: DeepCAD-style sequence predictors output operation types and numbers but never say “sketch on face AK,” so downstream features float in space. Code generators using CadQuery or FreeCAD can name geometry, but if they write the whole program blind without seeing intermediate shapes, they pick the wrong face or edge. Second, sketch-plane local coordinates: every sketch lives in its own 2D world with its own origin and axes assigned by the CAD system. The same pair of numbers means totally different 3D spots on a slanted face versus a flat top face. Models that only learn axis-aligned planes fail on inclined planes. Third, sketch constraints to projected external geometry: to stay attached, a new circle center should be constrained to coincide with the projection of an old vertex onto the sketch plane, and a new line should be constrained to be collinear with a projected old edge. Without that link, an edit breaks alignment. As illustrated in the paper, a model without references can look identical at first, then update incorrectly when a parameter changes.
The key idea, in plain words
Instead of asking the vision-language model to do precise 3D math, give it eyes and fingers, and let the CAD kernel do the math. Eyes means every edge and face in the current model gets a visible tag like AG or AK in the rendered view, plus a companion text table that lists face types, which edges bound which face, edge types, and sample points along each edge in picture coordinates. Fingers means the model answers by ID, for example “sketch plane is face AK, revolve around edge AM,” rather than by raw 3D numbers. For drawing, let the model point in the picture it can see, using tick marks around the viewport like graph paper, and have the kernel convert those picture points into true sketch coordinates. For staying attached, show the model where old edges land when projected onto the new sketch plane, so it can place new endpoints right on top of them, then let deterministic rules add persistent constraints like coincidence, collinearity, horizontal or vertical, tangency, equality, and symmetry, plus length and radius dimensions. Wrap all three in a three-stage loop — choose operation, draw sketch, visually check — with retrieval of similar past examples and rejection of bad steps. In short: model reasons about what and where, kernel enforces how exactly.
How it actually works, step by step
The system starts from a target image and an empty FreeCAD document and builds up to 25 steps until it outputs a termination token. Stage 1, selecting the operation: it renders a depth image of the current model, where nearer is whiter and farther is blacker, with solid magenta for visible edges and dashed magenta for hidden edges. It aligns that rendering to the target by adjusting camera scale and translation, a step called registration. From that aligned camera it creates the ID-tagged view and geometry table described above. It also retrieves up to three modeling examples using image similarity that balances current shape and target shape, computed with a frozen DINOv3 image encoder. The model then outputs operation type — one of 16 choices such as extrude add, extrude cut, symmetric extrude, revolve, sweep, loft, fillet, chamfer, offset construction plane, or END — plus whether a new sketch is needed, which face ID is the sketch support, other reference IDs like axis or path, and hints for depth or direction. Stage 2, sketching: the camera is turned to look straight at the chosen sketch face, with extra room reserved using a pseudo-bounding sphere so the new sketch is not cropped out. A fresh tagged sketch view and table are rendered, now including projected locations of external edges. The model draws one or more closed profiles by giving picture coordinates: two endpoints for a line, start-middle-end for an arc, four rim points for a circle. The kernel converts those to sketch-local coordinates. Constraints are then added automatically in order: first to external projections, for example line-to-projected-line collinearity, endpoint to projected endpoint coincidence, endpoint-on-projected-edge; then among new primitives, like endpoint coincidence, horizontal, vertical, tangency, equal length, equal radius, point symmetry; then dimensions for lengths, radii, and offsets from a reference center. FreeCAD solves after each addition and rolls back any addition that over-constrains. Stage 3, verifying execution: FreeCAD executes and renders a preview. The model judges whether this step adds a missing target structure or useful intermediate shape. If yes, the document advances; if no, the system returns to Stage 1 with the rejection reason kept in context for that step.
What they showed — results, honestly, with limitations
The authors test static reconstruction, meaning how close the built shape is to the target, and a parameter-editing test, meaning what happens after all additive and subtractive extrusion depths are multiplied by 1.1 in both prediction and ground truth. Metrics are compilation success rate, meaning share of non-empty error-free documents; mean intersection over union, meaning volumetric overlap where higher is better; Chamfer distance, meaning average nearest-point error where lower is better; and Hausdorff distances, meaning worst-case errors, with the 95th-percentile version ignoring the worst 5% outliers. Crucially, shape metrics are computed only on successful builds, so they must be read alongside success rate. On GERD evaluation data, Vision2CAD gets 100% success with mean overlap 0.4733 and Chamfer 0.0596, versus the next-best baselines at 0.4261 overlap from CAD-Coder on only 18 of 100 parts and 0.0721 Chamfer from CAD-Assistant. That is the reported 11.1% overlap gain and 17.3% Chamfer drop. On the 100-part DeepCAD subset, it gets 0.6987 overlap and 0.0320 Chamfer at 100% success, the reported 5.6% and 41.8% improvements. After the 10% depth edit it stays at 100% success and remains best on both datasets. Ablations that remove each piece show explicit referencing matters most: without it static overlap falls from 0.4733 to 0.4187 and editing success and overlap drop the most. Removing the view-to-sketch bridge drops editing success to 90.0%. Removing projected-geometry constraints barely hurts static shape at 0.4658 but drops editing success to 89%, supporting the claim that copying nearby coordinates looks fine initially but fails to stay attached without persistent constraints. Limitations are stated plainly by the authors and by the evaluation design: only a single target view is used; dense ID labels get cluttered on complex parts; references inside sketches and operations like linear or circular patterns are not yet supported; DeepCAD testing uses only 100 parts because full testing is costly; and as the paper notes, a single depth perturbation cannot fully prove design intent is preserved, only that an important aspect of parametric behavior holds.
Why this might matter to me
For spatial reasoning in vision-language models, this is a concrete pattern for grounding: do not force the language side to hallucinate origins and axes, show it labeled views with coordinate ticks and sampled polylines and let it select and point. That separation of qualitative pointing from quantitative solving is directly reusable for text-to-CAD and parametric geometry work where slanted sketch planes and local frames cause most failures. For world models, the registration plus preview-verify loop is a tiny world model in action: render current belief, align to observation, propose action with retrieved precedent, simulate in FreeCAD, visually accept or reject. The pseudo-bounding sphere trick for keeping future geometry in frame is a useful lesson in viewpoint planning. For 3D generation and scene representation, the message is that appearance metrics alone mislead. Two histories can yield the same mesh but behave differently under edits. GERD as an aligned record of commands, boundary representation states, and IDs at every step, with approximate spline data for cross-system rebuilds, offers a testbed for next-command prediction, face-edge-vertex selection, boundary representation encoding, and edit-stability benchmarks beyond single-shot mesh similarity.
Terms worth knowing
Parametric CAD model means a part defined by an ordered history of features, sketches, dimensions, and constraints, so changing a number updates the shape. Design intent and feature dependencies mean the intended relationships, such as a hole staying centered on a boss, encoded by references and constraints. Boundary representation, often shortened to B-Rep, means describing a solid by its Body, which is a volume; Face, which is a surface piece like plane or cylinder; Edge, which is a curve where faces meet; and Vertex, which is an endpoint. Sketch plane and local coordinates mean the 2D frame in which a sketch is drawn, with its own origin and X and Y axes that the CAD kernel assigns. Projected external geometry means existing 3D edges or vertices flattened onto the current sketch plane, so new sketch points can be constrained to them. ID-tagged view and geometry table mean the paper’s interface where the rendering labels each edge and face and the text file lists types, boundary lists, and sampled view coordinates for selection. Constraint examples include coincidence, meaning points meet; collinearity, meaning lines share the same infinite line; point-on-object; tangency; equality; symmetry; and dimensional constraints for lengths and radii. GERD, GERD-RET, and GERD-EVL mean the new dataset of 2,700 Onshape parts with per-step IDs, its retrieval split for examples, and its image-to-code split with original and 10%-deeper extrusion versions for edit testing.