All summaries · 2026-09-17

SketchGraphs: A Large-Scale Dataset for Modeling Relational Geometry in Computer-Aided Design

Ari Seff, Yaniv Ovadia, Wenda Zhou, et al.

arXiv:2007.08506 · abstract · pdf · summarized by meta/muse-spark-1.3-contributor on 2026-09-17

The one-paragraph version

Parametric computer-aided design, the way most mechanical parts are actually engineered, does not start with pixels or triangles but with a two-dimensional sketch: simple shapes plus rules that lock them together. This paper introduces SketchGraphs, a dataset of 15 million such sketches pulled from real public models on the cloud CAD platform Onshape, each stored not just as final geometry but as the designer’s construction history. A sketch becomes a geometric constraint graph, meaning a network where the items are geometric primitives and the connections are designer-imposed constraints. The authors also release Python tools for handling, visualizing, and rendering these sketches, including noisy hand-drawn rendering, and they show two first uses: autoconstrain, which means guessing good constraints for bare geometry, and unconditional generation, which means sampling brand-new sketch graphs from scratch.

The problem they're solving, and why it matters

Most machine-learning work on 3D shapes before this paper used what the shape looks like at the end: collections of filled cubes called voxels, or surfaces made of triangles called meshes. Datasets like ModelNet, ShapeNet, and even the parametric ABC dataset support that kind of appearance modeling, but they lose the how: which line was drawn first, which two endpoints were declared to touch, which circle was made equal to another.

That how matters for two reasons. For engineering, a mesh sample you cannot edit is not very useful. In parametric CAD, which means design defined by adjustable parameters plus lasting relationships, you draw a rectangle as four line segments plus rules like these sides stay horizontal, these sides stay vertical, these corners stay touching. If you later drag one corner or change one length number, the rules propagate the edit and keep design intent. Without the underlying rules, you cannot do that propagation.

For AI research more broadly, the authors argue parametric CAD is a well-scoped example of constraint programming, meaning specifying what relationships must hold rather than giving step-by-step imperative instructions, and of program induction, meaning learning to infer an underlying program from examples. It is more structured than freehand drawings of cats or buses in datasets like QuickDraw and Sketchy, but more tractable than full general programming, making it a useful test bed for learning modularity, symmetry, and long-range relationships.

The key idea, in plain words

Treat a real CAD sketch as a social network of geometry. The members of the network are primitives, a word the paper uses for basic drawable pieces like a point, a line segment, a circle, an arc, a spline, and an ellipse. The friendships between members are constraints, meaning explicit rules imposed by the designer like coincidence, which means two points must touch, horizontal or vertical, parallel, perpendicular, tangent, mirror, equal length, or a fixed distance or angle value.

Concretely, imagine a simple house outline: four lines for a square plus a triangle on top. The graph would have a node, meaning a graph member, for each line plus extra nodes for important subparts like each line’s endpoints and center. Edges, meaning graph connections, would record that the bottom of one wall touches the end of the floor, that opposite walls stay parallel, and so on. Some edges loop back on one member, for example setting a circle’s radius. Some edges join three members, for example a mirror constraint that says this left shape mirrors that right shape across this middle line as the symmetry axis.

A second key idea is that CAD gives a natural order for free. Unlike a generic graph where there is no correct order to list the nodes, here we know the order the designer added primitives. Early primitives tend to be hubs that later geometry anchors to, and neighbors in time are much more likely to connect than random pairs. That order makes it natural to train sequential, step-by-step predictive models.

How it actually works, step by step

Acquisition, meaning how the data were gathered: the authors queried Onshape’s programming interface for public documents created from 2015 to 2020, found over two million document identifiers, downloaded each component design file called a PartStudio, and extracted every sketch with at least one primitive and at least one constraint. Three-dimensional extrusion and other solid operations were not kept. The result is over 15 million 2D sketches.

Representation: each sketch is stored as a multi-hypergraph, meaning a network that allows multiple connections between the same members and connections that join more than two members. Primitives carry parameters, meaning adjustable numbers describing placement, such as a point’s x and y coordinates or a circle’s center and radius, plus a flag called isConstruction that marks whether a piece is physically realized or is just invisible reference geometry shown as dashed lines. Constraints carry their member references plus any needed values, such as a length number for a distance constraint or an angle number plus direction flags for an angle constraint. If a rule applies to part of a primitive, for example the endpoint of a line, that sub-primitive becomes its own node so the reference is unambiguous.

Ordering into sequences: the primitive order is known from Onshape. The relative interleaving of primitives versus constraints is not recorded, so the authors canonicalize it by inserting each constraint right after its members appear, mimicking the habit of constraining as you draw. They also note an alternative ordering with all constraints at the end, which mimics importing a bare scan and then adding rules.

Tooling: the open-source pipeline provides structured domain types for primitives and constraints, a local renderer for viewing sketches, functions to query a geometric constraint solver, which means software that adjusts coordinates to satisfy the rules, and integration with deep-learning frameworks.

Example baseline models: both baselines use message passing networks, meaning neural networks that repeatedly pass information along existing graph connections to build context-aware embeddings. The autoconstrain model reads bare primitive shapes plus a growing constraint graph and repeatedly predicts the next partner node to connect to plus the constraint label, meaning the rule type. At test time it is given a mask of which constraints are geometrically satisfied so it only proposes rules that would not deform the sketch. The generative model does the same edge prediction plus an extra module that predicts the next primitive type to add or a stop signal to finish, including constraint numerical parameters but not primitive coordinates, which are left to the solver starting from a uniform initialization.

What they showed — results, honestly, with limitations

The dataset itself is the main result. Sketches range from a single circle to large assemblies. To give a feel: lines dominate at about 68.5% of primitives, followed by circles at about 10%, arcs at about 9.5%, and points at about 8.6%. The most common constraint by far is coincidence at about 42%, followed by projected geometry at about 9.7%, distance, horizontal, mirror, vertical, and parallel. The authors report that sketches are often close to fully constrained, meaning enough rules to remove all free movement except rigid shifting and rotating, with a Pearson correlation, meaning a linear association measure, of 0.598 between total degrees of freedom and degrees removed by constraints.

On a filtered subset of about 2.2 million sketches limited to at most 16 primitives, only point, line, circle, and arc types, and only one- and two-member edges, the autoconstrain model achieved average edge precision and recall of 0.74 each, average F1 score, meaning harmonic mean of precision and recall, of 0.71, and 0.495 bits per edge, where fewer bits means more predictable, compared to 6.09 bits for guessing uniformly among valid constraints. Qualitatively, dragging edited geometry preserved intended touching, alignment, and equality, though in the shown example it missed a 7 millimeter distance between top and bottom of a rectangle.

The generative model achieved 28.2 bits per graph on held-out data versus 85.6 bits for a standard LZMA compressor, meaning a general file compressor, applied to integer sequences, so the learned model compresses much better. Sampled primitive counts, constraint counts, and type distributions roughly track the training data, but rendered samples look simple and repetitive.

Limitations stated or clear from the methods: baselines use only the small, simple subset and ignore three-way constraints like mirror; constraints involving external default axes are not retrievable from the API; coordinate initialization is not modeled, which limits visual diversity; evaluation of generation is only likelihood plus histograms, not human ratings of usefulness; and the paper is unclear on how well results would transfer to full 3D feature histories, which are future work.

Why this might matter to me

For spatial reasoning in language and vision models: most vision-language training sees finished renderings, not the relational program that keeps a design coherent under edits. SketchGraphs provides millions of human examples where symmetry, parallelism, touching, and mirroring are made explicit as edges. That is supervision for learning that a rectangle is not four independent strokes but a loop held together by rules, which could ground spatial terms like aligned, centered, or mirrored in executable relationships rather than pixel statistics.

For world models: a world model needs to predict how scenes behave when poked. A constraint graph plus solver is a tiny physics-like world where dragging one primitive propagates through declared dependencies. Learning to predict those propagations, or to infer missing constraints that make behavior stable, is a clean analogue of inferring joints, supports, and attachments in messier 3D scenes.

For text-to-CAD and parametric geometry: current text-to-3D often outputs meshes that an engineer cannot open, dimension, and modify. SketchGraphs points toward text-to-sketch-graph, where a model outputs editable types plus values like line to arc with tangent and 45-degree angle, which a solver then realizes precisely. The paper’s finding that common angle and length values are highly concentrated, for example 45 degrees and standard millimeter and inch sizes, suggests strong priors a language-conditioned CAD model could exploit, while still leaving exact numbers controllable.

For 3D generation and scene representation: 2D sketches are the anchors for 3D operations like extrusion into solids. A scene or part could be represented hierarchically as sketch graphs plus operations, preserving part structure, reuse, and long-range symmetry that triangle soups lose. The construction order, early hub primitives anchoring later ones, and the hand-drawn rendering pairs also offer a bridge from noisy visual input to structured inference, relevant to reverse-engineering scans or sketches into editable CAD.

Terms worth knowing