The one-paragraph version
This paper treats a 2D CAD sketch as if it were a sentence to be written by a language model. In mechanical CAD, a sketch is not just lines on a page: it is a set of geometric pieces called entities, like points, lines, arcs and splines, plus a set of rules called constraints, like “these two lines stay parallel” or “these endpoints stay touching,” that capture what the designer meant to preserve when dimensions change. The authors collect more than 4.7 million real sketches from public Onshape documents, describe them with a structured data format called Protocol Buffers, turn them into sequences of tokens, and train Transformer models to predict the next token, just as in text prediction. They show two ways to do the tokenization, a raw byte version and a smarter triplet version that knows about fields and pointers, for both making sketches from scratch and reconstructing a sketch from an image of it. The triplet model wins on likelihood and produces plausible, symmetric, mostly closed shapes with sensible constraints, though a meaningful fraction are still invalid without cleanup.
The problem they're solving, and why it matters
Every 3D CAD model starts from 2D sketches. A sketch lives on a plane and later gets extruded, cut, or lofted into solid shape. Onshape data cited in the paper suggests sketching is about 35% of feature creation operations, so it is a high-leverage place to help.
The hard part is not drawing lines, it is capturing design intent. Design intent means the shape should still behave like the intended object when you drag or resize it. A heart should stay a heart. That behavior is encoded by constraints.
Constraints interact globally. If you move one point, the solver — the CAD program’s numerical engine that adjusts all geometry to satisfy all rules at once — may have to move everything. It is easy to add a contradictory rule, such as requiring the same two lines to be both parallel and perpendicular, which no drawing can satisfy. For a given geometry there are also many equally valid rule sets, but only some preserve intent cleanly. That combination — must be grammatical like a valid rule system, and must be meaningful like preserving intent — is why the authors compare sketching to language modeling.
Prior machine-learning work either focused on meshes and ignored parametric history, used small or duplicate-heavy sketch datasets, required an external solver to place geometry, or only decided how to extrude a sketch that was already given. Image-to-CAD systems were mostly hand-coded heuristics, not learned end-to-end. This paper aims for a learned model that emits both geometry and constraints together, at real-world scale.
The key idea, in plain words
If sketching is like writing sentences, then use the tools that write sentences well: Transformers trained on lots of real examples.
To do that, the authors need a bridge between highly structured CAD objects and flat sequences. Their bridge is Protocol Buffers, a widely used system for defining structured messages, plus a custom interpreter that walks through those definitions step by step.
Think of building a sketch as answering a series of fill-in-the-blank questions: is this next object an entity or a constraint? If an entity, what kind — line, point, circle-or-arc, spline? If a line, is it construction geometry that will not show in the final drawing? What are its start and end coordinates? If a constraint, what kind — coincident, parallel, mirror, and so on — and which pieces does it point to?
The model’s job is simply to answer the next question given all previous answers. Because the interpreter always knows which question comes next, every answer the model gives leads to a structurally valid message. That is the core trick: offload syntax to the serialization protocol, so the neural network can spend its capacity on design patterns instead of brackets and commas.
How it actually works, step by step
1. Define the vocabulary of sketches. The authors support 4 entity types — point, line, circle-or-arc, and interpolated spline — and 16 constraint types, including fix, coincident, concentric, equal, parallel, tangent, perpendicular, mirror, distance, length, diameter, radius, angle, horizontal, vertical, and midpoint. They drop constraints that point to so-called external entities outside the sketch plane. A circle-or-arc, for example, is defined as a center plus either a radius for a full circle or start, end and clockwise direction for an arc. A mirror constraint is defined as an axis plus one or more pairs to be kept symmetric.
2. Resolve messy real data. Raw Onshape data often stores numbers as symbolic expressions tied to variables elsewhere in the document. The authors scrape about 6 terabytes of public documents from October 2020, propagate variable assignments, and evaluate expressions with the SymPy library when possible. They rescale sketches into a square from negative one to positive one in both axes to make learning easier. They keep sketches with 4 to 100 entities, then split randomly into about 4,671,037 training examples and 50,000 each for validation and test.
3. Reduce duplication. Public CAD is full of copies: tutorial parts and simple rectangles. About 15% are simple axis-aligned rectangles, which they remove. They then group remaining sketches that share the same sequence of object types, render each to a 128 by 128 binary image, cluster near-identical renders within each group using hierarchical clustering with Jaccard distance, and keep one per cluster.
4. Turn a sketch into tokens. Entities come first, then constraints that refer to them, or in the better-performing interleaved ordering, each constraint is moved to sit right after the entities it refers to. The authors test two encodings. The byte encoding serializes the Protocol Buffers message into raw bytes, each 0 to 255 plus an end marker, for about 456 tokens on average and up to 1,987. The triplet encoding uses one triplet per field: a discrete choice part, a continuous value part where only one is active, and an end-of-list flag for repeated fields like the list of objects. For example, a line plus a point becomes a short sequence where early triplets choose object kind and entity kind, middle triplets fill booleans and quantized coordinates, and a final flag closes the list. Field identity and position inside the object are fed in as extra context, and continuous values are quantized to 8 bits and treated as categories for stability.
5. Model the sequence autoregressively. Autoregressive here means predict each token from all earlier tokens and multiply those predictions to score the whole sketch. Both models use a Transformer decoder, the attention-based sequence model behind modern language models. The unconditional models use 384-dimensional embeddings and 24 layers. Discrete tokens get categorical predictions. Pointers — references from constraints to entities — are handled specially in the triplet model with Pointer Networks, a mechanism that scores which earlier item to point to by comparing the current output to the history of outputs. Because a pointer might mean a whole line, an endpoint, or a midpoint with no dedicated token, the authors insert special referrable tokens after each entity to serve as clean pointer targets.
6. Sample with an interpreter. For bytes, sampling is like normal text generation. For triplets, a state machine generated from the Protocol Buffers spec tells the model which token group is legal now, picks the active part of the predicted triplet, fills that field, and feeds the choice back in until the outer list is closed.
7. Add image conditioning. For drawing-to-sketch, a visual Transformer, the image version of the same attention architecture, cuts a 256 by 256 render into 16 by 16 patches into 256 vectors. The sketch Transformer can attend to those patch vectors at every layer, so it can look at relevant image regions while emitting each object, rather than compressing the whole image to one vector.
What they showed — results, honestly, with limitations
The main quantitative measure is log-likelihood, expressed as bits per object and bits per sketch, where lower means the model assigns higher probability to held-out real sketches. Both learned models beat uniform baselines by a huge margin. The triplet interleaved model is best unconditional, at about 4.09 bits per object and 122.8 per sketch, ahead of byte interleaved at about 4.27 and 128.1. Interleaved ordering beats putting all constraints at the end in both encodings, supporting the idea that early design-intent clues help predict what comes next. The image-conditional model reaches about 2.62 bits per object, much lower because the image removes much uncertainty, though ambiguity remains since many token orders can render to the same picture.
Qualitatively, 10,000 samples closely match the data on counts of objects, entities, constraints, closed regions, arcs, and common constraint types when sampled without filtering. With nucleus sampling with top-p of 0.9 — a decoding trick that keeps only the most probable tokens covering 90% of probability mass to avoid weird tails — distributions skew somewhat but samples get cleaner. Even so, many raw samples are malformed, meaning unparseable, or unsolvable by the solver: about 36% for bytes and 14% for triplets without filtering, dropping to about 25% and 6% with that filtering. Renders often look regular, closed, symmetric, and detailed, and inferred constraints like coincident connections and multi-entity mirrors are often reasonable. Running samples through a solver can fix small coordinate errors, as shown in before-and-after figures.
The conditional model nearly perfectly reconstructs simple sketches and mostly errs on fine detail. On an out-of-distribution smiley face drawn by the authors, it stopped too early because training sketches are shorter, but when forced to keep generating up to 1,024 tokens and searching over 64 samples for best image match after smoothing, it recovered a near-perfect sketch and even inferred sensible coincident and parallel rules the original only had by accident.
Limitations are plain. Trying to shorten byte sequences with Brotli compression made learning fail, suggesting aggressive compression destroys learnable structure. Like typical teacher-forced models — trained always on correct history and thus brittle when their own early mistake shifts them off track — it cannot recover from early errors, and reinforcement learning with solver feedback is left as future work. Training imitates valid examples but does not directly optimize for solvability or stability. Coverage is limited to the supported types, size range, and cleaned data, and ordering choices and quantization mean the model must learn arbitrary linearizations of what is fundamentally an unordered set.
Why this might matter to me
For spatial reasoning in language and vision models, this is a concrete case where next-token prediction learns non-trivial geometry: closure, symmetry, parallelism, and long-range references, not just local strokes. The gap between high likelihood and still-high invalid rates is instructive — syntactic validity via the interpreter does not guarantee geometric solvability, a sharper test of spatial understanding than image similarity.
For world models, the solver-in-the-loop direction the authors sketch matters. A true design world model would predict how the constraint solver and later 3D operations will react to an edit, and could be trained with environment feedback rather than pure imitation. Their failure to learn from Brotli-compressed bytes is also a lesson: representations that hide causal structure hurt world-model learning.
For text-to-CAD and parametric geometry, the triplet-plus-interpreter pattern is directly reusable. It shows how to make language models emit parametric, editable history — entities plus constraints with pointers — rather than dead meshes or pixels, and how interleaving relations early helps. Conditioning here is on images, but the same cross-attention slot could take text, dimensions, or partial sketches for autocomplete and constraint inference.
For 3D generation and scene representation, the paper anticipates lifting the same machinery to extrusion and other operations that, like constraints, are nested messages with references to existing geometry. That points toward language-like 3D programs where objects, relations, and operations share one sequence interface, with renders or solver states as grounding.
Terms worth knowing
Sketch: a 2D drawing on a plane that serves as the starting point for 3D CAD operations like extrusion.
Entity, also called primitive: a geometric piece in a sketch, here point, line segment, circle or circular arc, or interpolated spline.
Constraint: a rule relating entities, such as coincident meaning touching, parallel, perpendicular, tangent, mirror meaning symmetric, horizontal or vertical, or fixed dimensions like length and diameter.
Design intent: the behavior the designer wants preserved under resizing or dragging, encoded by a good choice of constraints.
Solver: the CAD numerical engine that adjusts all entity coordinates to satisfy all constraints at once; a sketch is unsolvable if no arrangement satisfies them.
Protocol Buffers: a system for defining structured data types and serializing them compactly; here used to specify sketches instead of hand-writing syntax.
Token: the unit the model predicts at each step, either a byte value or, in the smarter encoding, a field value.
Triplet format: the authors’ encoding where each field becomes discrete choice plus continuous value plus end-of-list flag, with only the relevant part active and field identity supplied as context.
Token group: a set of similar fields, like all x and y coordinates, that share the same embedding and output distribution.
Pointer Network: a mechanism for predicting references by scoring the current state against earlier states, used here for constraints pointing to entities.
Referrable token: an extra placeholder inserted after each entity to give pointers a clean target when entities have multiple parts like whole shape, endpoints, or midpoint.
Interpreter-guided generation: sampling where a state machine derived from the data spec tells the model which field is next and fills it in, guaranteeing structural validity.
Autoregressive model: a model that builds a sequence one token at a time, scoring the whole by multiplying the probability of each token given prior ones.
Interleaved versus concatenated ordering: whether each constraint is placed right after the entities it uses, or all constraints are lumped after all entities; interleaved performed better.
Nucleus sampling: a decoding filter that samples only from the smallest set of most likely tokens covering a chosen probability, here 0.9, to reduce strange outputs at the cost of diversity.