CADDreamer: CAD Object Generation from Single-view Images

The one-paragraph version

CADDreamer takes one ordinary color photo of a manufactured object and turns it into a clean, editable Computer-Aided Design model — not a lumpy triangle soup, but a watertight assembly of true geometric surfaces like planes, cylinders, cones, spheres, and tori joined along sharp edges. It does this in two stages: first a primitive-aware multi-view diffusion model imagines what the object looks like from six viewpoints, predicting both surface orientation and what kind of primitive each pixel belongs to; then a geometry engine fits exact mathematical surfaces to those predictions, gently nudges them so they meet correctly, and cuts them against each other to build a proper boundary representation. On 1,000 synthetic test objects the authors report far lower shape error and far fewer broken faces than combining existing single-image mesh generators with a CAD fitter, plus qualitative success on real photos.

The problem they're solving, and why it matters

Recent image-to-3D systems can make a plausible-looking mesh from a single image, but that mesh is typically over-tessellated, bumpy, and structure-blind. A term that needs definition here is neural implicit field, which means a neural network that represents shape as a continuous function like signed distance or density at every point in space, later converted to many tiny triangles. That is great for movies or quick previews, but not for engineering, games that need low polygon counts, or manufacturing, where a designer thinks in terms of parts: this face is flat, that part is a cylinder, those two axes are parallel.

The paper targets that gap: single-view reconstruction of CAD objects. A CAD object here means a boundary representation, often shortened to B-rep, which means a description of a solid by its faces which are trimmed pieces of exact surfaces, its edges which are curves where faces intersect, and its vertices where edges meet, all stitched into a closed, watertight volume with no gaps or dangling flaps.

This is hard for two reasons the authors emphasize. First, one view is incomplete — you must hallucinate the back side and infer high-level structure, not just color. Second, even small noise breaks CAD: if two fitted surfaces miss each other by a fraction, their intersection curve disappears or shifts, leaving hanging faces, which means faces with an edge that has no neighbor, or a model that is not closed.

The key idea, in plain words

Think of how a human reverse-engineers a simple bracket: you would say, that is mostly a box, with a cylindrical hole, and a round fillet that is part of a torus, and the flat top must be exactly perpendicular to the side.

CADDreamer tries to give a diffusion model that same kind of dual vision — low-level shape plus high-level part labels — and then enforce the engineer's tidiness afterward.

The trick for the first part is to paint meaning with color. Instead of asking the diffusion model, which means a generative model that learns to denoise random noise into realistic images and here is repurposed to imagine new viewpoints, to output only color or surface orientation, the authors also ask it to output a semantic primitive map, which means an image where the color encodes the primitive type, for example blue for plane, green for cylinder, plus dark lines for feature lines that separate primitives. Because pre-trained image diffusion models already have strong priors about coherent colored regions, this color-coding lets the model leverage what it knows about segmentation to produce consistent part labels across views.

The trick for the second part is stitch-and-straighten. The initial fitted surfaces are wobbly. CADDreamer picks points along where neighboring patches touch on the rough mesh, pulls the mathematical surfaces together at those stitching points, and while pulling, forces expected regularities like two axes staying parallel, perpendicular, or collinear, which means sharing the same line in space. Only after that cleanup does it compute true intersections to get sharp CAD edges.

How it actually works, step by step

The pipeline has two modules, shown in Figure 2 of the paper.

Module 1: Multi-view generation and patch segmentation.

1. Image to orientation. The single input color photo is first converted to a normal map by an off-the-shelf predictor. A normal map means an image where each pixel's color encodes which way the surface points in 3D, which removes confusing texture and lighting and gives the next stage a geometry-focused input.

2. Joint multi-view dreaming. A fine-tuned version of Wonder3D, which means a cross-domain diffusion model that generates consistent multi-view images with attention across views and across output types, takes that one normal map and generates six viewpoints, each with two images: a normal map and a semantic primitive map. The authors fine-tune two separate decoders, which means the network parts that turn latent codes back into pixels, one for normals and one for semantics, and use three-pixel-wide feature lines in the training data.

3. Rough mesh without color. The six predicted normal maps go into NeuS, which means a method that reconstructs a 3D surface by volume rendering a learned signed-distance function, here stripped of its color and texture loss because only shape is needed.

4. From 2D labels to 3D patches. The 2D primitive maps are cut by their feature lines into 2D regions, then back-projected, which means casting each labeled pixel back onto the mesh triangles it would have seen, onto the mesh. A Graph Cut process, which means an optimization that merges many small overlapping projected regions into a few coherent patches by cutting weak links between dissimilar regions, merges them. Similarity is measured by fitting primitives with Efficient RANSAC and comparing parameters. The result is a segmented mesh where each patch should correspond to one plane, cylinder, cone, sphere, or torus.

Module 2: Geometric and topological extraction.

1. Initial fitting. Each patch's vertices are fed to RANSAC, which means a robust fitting algorithm that repeatedly samples small subsets of points to find surface parameters that ignore outliers, to estimate parameters like a plane's normal direction and position, or a cylinder's axis, position, and radius. Definitions are listed in Table 1.

2. Geometric optimization by stitching. Adjacency on the segmented mesh implies intersection: if two patches share a boundary, their true surfaces should intersect. For each such pair the method samples stitching vertices, which means boundary points closest to both surfaces, projects each point onto both surfaces, and minimizes the distance between those two projections by adjusting surface parameters. In words, it pulls the surfaces together at the seam. Parallelism is enforced by sharing one axis, collinearity by expressing one position as the other position plus a distance along a shared axis, and perpendicularity by penalizing non-zero dot product between axes. Optimization uses L-BFGS, a standard quasi-Newton optimizer.

3. Topology guide. From the segmented mesh the method extracts a topological representation: each patch becomes a face, each curve between two patches becomes an edge with direction inherited from the mesh half-edges, and each point touching more than two patches becomes a vertex. Because the input mesh is watertight, this guide is watertight.

4. Topology-preserving intersection. For each guide edge, intersect the two neighboring optimized primitives to get CAD curves, picking the closest candidate if there are several. For each guide vertex, intersect its two CAD curves — equivalent to intersecting three primitives — to get a CAD vertex. Trim curves by vertices to get CAD edges, trim surfaces by edges to get CAD faces, and merge into a final B-rep.

What they showed — results, honestly, with limitations

The authors curated 30,000 watertight CAD models from the ABC and DeepCAD datasets, rendered with textures and backgrounds, split into 29,000 for training and 1,000 for testing, using Wonder3D camera settings. They also tested on real photos taken with a handheld camera, using Era3D for background removal and normal prediction.

Baselines combine state-of-the-art single-view mesh methods — LRM, CRM, InstantMesh, SyncDreamer — with Point2CAD for segmentation and B-rep building. Sketch-extrude methods that only support planes and cylinders are discussed mainly in the supplement via DeepCAD, since the test set includes cones, spheres, and tori.

On meshes and primitives, reported in Table 2: CADDreamer achieves Chamfer Distance, which means average nearest-point distance between sampled points on reconstruction and ground truth, lower is better, of 1.27 versus 3.97 to 5.49 for baselines; Normal Consistency, which means average alignment of surface directions, higher is better, of 92.6% versus 48.9% to 64.4%; vertex label accuracy of 95.7% and correct primitive-count accuracy of 97.9%, versus roughly 30-50% and 33-49% for baselines. On final B-reps, reported in Table 3: hanging-face rate of 2.4% versus 35.2% to 58.5%, and B-rep Chamfer Distance of 1.36 versus 9.74 to 15.4. Figures show sharper edges and fewer missing intersections, and real-image examples look plausible despite lighting and perspective noise.

Ablations support the design: removing cross-domain semantic prediction hurts segmentation; using color photos instead of normal maps as diffusion input works on synthetic data but generalizes poorly to real photos; removing stitching or the parallel-perpendicular-collinear constraints sharply increases hanging faces.

Limitations are stated plainly and matter. Coverage requires all faces to appear in the six fixed, mostly horizontal 256x256 views — a top face invisible in all views is lost. Extremely thin faces narrower than about one pixel become fragmented. Heavy occlusion, unusual viewpoints, and complex fine structures can fail. The optimizer only handles intersection, parallelism, perpendicularity, and collinearity, not symmetry or more global regularities. Training is heavy: eight 80GB A100 GPUs. The paper does not provide uncertainty estimates or a test of how errors scale with viewpoint difficulty, so robustness boundaries remain unclear.

Why this might matter to me

For spatial reasoning in vision-language models: the semantic primitive map is a compelling intermediate — it forces a generative model to name parts and boundaries, not just pixels, and enforces multi-view consistency of those names. That is a concrete pattern for grounding language like parallel, coaxial, or fillet in images.

For world models: compact B-reps are the opposite of dense point clouds or neural fields. They are low-dimensional, symbolic, and simulation-ready — watertight solids with exact axes and radii that a physics or planning model could grasp, collide, or assemble. CADDreamer suggests a path from photometric prediction to actionable contact geometry.

For text-to-CAD and parametric geometry: while this work is image-to-CAD, its second module is reusable. Any front end that proposes noisy patches — whether from text, sketch, or language-guided editing — still faces the stitch-and-constrain problem. The formulation of intersection as distance-minimization over stitching points with axis constraints is a template for turning neural guesses into valid parametric solids.

For 3D generation and scene representation: it reframes quality as editability and topological validity, measured by hanging faces and primitive-count accuracy, not just Chamfer Distance. If future object or scene generators output primitive-labeled multi-views plus a topology guide, downstream tools could keep scenes light, sharp, and manipulable rather than frozen meshes.

Terms worth knowing

B-rep / boundary representation: solid described by trimmed faces, intersection edges, and vertices forming a closed volume; the standard for CAD and manufacturing.

Geometric primitive: ideal surface type with few parameters, here plane, cylinder, cone, sphere, torus. Example: a mug body as cylinder, handle as torus.

Feature line: curve separating two primitives, where surface direction jumps; used to split 2D maps before 3D segmentation.

Normal map: image whose colors encode surface orientation rather than color; robust to texture change.

Semantic primitive map: image whose colors encode primitive class and feature lines; lets diffusion priors do part segmentation.

Diffusion model / Wonder3D / NeuS: Wonder3D is a pre-trained multi-view diffusion model fine-tuned here to dream consistent normals and semantics; NeuS reconstructs a mesh from those views via volume rendering of a signed-distance function.

RANSAC: robust fitter that tolerates outliers by testing many random minimal point sets; used for initial primitive parameters.

Graph Cut: optimization for merging overlapping 2D-to-3D projections into coherent 3D patches based on similarity.

Stitching vertex / primitive stitching: boundary points used to pull neighboring surfaces together during optimization.

Chamfer Distance, Normal Consistency, SEG(V), SEG(P), HF: evaluation metrics for point proximity, direction alignment, per-point label accuracy, per-object primitive-count accuracy, and fraction of faces with unjoined edges.