The one-paragraph version
Parametric CAD Bench v3 is a 100-task test of whether AI agents can do real parametric CAD work inside FreeCAD, not just write code that runs or make a shape that looks right. It covers three jobs: build a part from a text description with named dimensions, build a base part from text and then edit it to meet new requirements, and rebuild a mechanical part from a dimensioned engineering drawing. Agents work in a fixed FreeCAD 1.1.0 environment and must save native FreeCAD documents with an editable feature history. A separate verifier scores each result on two axes — how close the solid geometry is to a hidden reference, and how many of the required dimensions and features are actually measurable and correct in the native model — combined into a continuous reward from 0 to 1. In the September 24, 2026 snapshot of 13 complete model-plus-agent systems, Claude Code with Claude Opus 5.5 scored highest overall at 61.03% mean reward and 84.66% on drawing reconstruction, while two-stage create-and-edit remained hardest with five tasks solved by no system.
The problem they're solving, and why it matters
Professional mechanical design lives in parametric CAD systems like Siemens NX, CATIA, SolidWorks, and the open-source FreeCAD. Parametric here means the model stores more than final shape: it stores a geometry kernel that handles curves, surfaces and solids, a constraint solver that enforces dimensional relationships, and a feature history that records how sketches became extrusions, cuts, holes and patterns so later parameter changes propagate. The paper argues that current evaluations cheat that reality. A generated script can execute but produce wrong dimensions, missing holes, broken dependencies, or a frozen imported shape instead of an editable single-body PartDesign model. Conversely, two very different feature trees can both be valid engineering solutions, so demanding an exact copy of a reference history is too strict. Earlier versions of this bench tested only text-to-CAD creation. Version 3 adds editing and reconstruction from engineering drawings, tightens isolation so agents cannot see hidden reference geometry, and reports partial credit rather than binary pass-or-fail, to better reflect engineering intent across creation, modification, and drawing interpretation.
The key idea, in plain words
Don't grade the script, grade the native CAD artifact on both shape and specification. An example makes this concrete. One creation task is a trussed upper plate: a thin 280 by 549 by 8 mm frame made from three rectangles, with trapezoidal side recesses, 18 internal openings including 17 triangles plus one six-sided hole, plus mounting holes and a center bore, controlled by 86 named parameters. Another is an asymmetric clevis joint connector: a central block with a large split-ring clevis on one end, a smaller fork on the other, and a long slotted side arm, controlled by 44 parameters. A pretty render is not enough; the holes must be the right size, in the right place, cut from the right face, and exposed as editable sketch and feature dimensions the checker can measure. For scoring, think of two report cards multiplied in a strict way. One card measures geometric closeness, the other measures whether the named engineering requirements are present and correct. If either card is zero, the task gets zero, formalized as a harmonic mean that punishes imbalance. An edit task scores base and target separately, then combines those two scores the same strict way.
How it actually works, step by step
Tasks and inputs: 30 text-to-CAD creation tasks give a prose description plus key parameters. 30 text-to-CAD create-and-edit tasks give a base description then an edit instruction, for example build an axisymmetric flanged inlet housing with a 50.8 mm bore and then add a fourfold-symmetric front plate, four bolt holes and four rear openings while leaving the body unchanged; the agent must save both base and edited models. 40 image-to-CAD tasks give only an engineering drawing with orthographic, section and isometric views plus dimension labels, such as a linkage arm with 56 mm length and unequal 3 mm and 2 mm eye holes, and require an editable single solid.
Execution contract: the agent writes a FreeCAD Python script in the agent container, for example to /app/answer.py, that builds the model as parametric features inside one PartDesign Body producing exactly one solid. Reference models, drawings tolerances, and parameter-checking code live in a separate verifier container using validator version 0.6.0, so agents cannot peek.
Geometry check in words first: the verifier compares candidate versus reference on volume, total surface area, area broken down by surface type like planes versus cylinders, distribution of mass captured by normalized principal moments of inertia, and alignment of face centers after iterative closest point registration. Volume must match within 0.1% for full credit and gets zero beyond 1% error, with similar 1% to 10% bands for the other terms. Before weighting, hard gates can zero the score: bounding-box extents off by 10% or more, not using an allowed native Body with one solid, more than 5,000 faces, or face or vertex counts differing by more than half. Passing scores are then a weighted sum with an alignment multiplier.
Specification check in words first: CAD engineers wrote per-task checking code, called param_check.py, that measures the hidden required dimensions directly from native sketches and features. Most checks are generic per kind of dimension, some are task-specific, plus a few direct witnesses on final geometry. Default tolerance is 1% relative error. If n is the number of required parameters and f is the number failed or not found, the spec score is zero if there are none, otherwise maximum of zero and one minus f divided by the smaller of n and 10. In plain terms, for large specs with 10 or more parameters each failure costs 0.1, so 10 failures exhausts all credit and prevents large parameter lists from hiding errors.
Final reward in words first: let geometry be g and spec be s, both 0 to 1. The task reward is their harmonic mean, which is zero if either is zero and otherwise twice their product divided by their sum. Create-and-edit nests this: score base as harmonic of its g and s, score target the same way, then take the harmonic of those two stage scores. The reported overall percentage, called S, is 100 times the average reward over all 100 tasks, equivalently 30% creation plus 30% edit plus 40% drawing by task counts.
What they showed — results, honestly, with limitations
The leaders differ by workflow, and no system is reliably correct. Best overall point estimate was Claude Code with Claude Opus 5.5 at 61.03%, followed by mini-swe-agent with Opus 5.5 at 57.96%, Codex with GPT-6 Astra at 56.87%, and Claude Code with Claude Fable 5.1 at 56.75%. By workflow, Fable 5.1 led creation at 55.48%, Astra led create-and-edit at 44.37%, and Opus 5.5 led drawing reconstruction at 84.66%. Paired by task, Opus 5.5 beat Opus 5 within Claude Code mainly on drawings: 22 of 40 drawing tasks improved by more than 5 points, median gain 7.62 points, accounting for about 87% of its 10.18-point overall gain. Claude Code versus mini-swe-agent with Opus 5.5 fixed showed no decisive overall difference, with interval -1.15 to +7.31 points; Claude Code led creation while edit and drawing were essentially tied. Drawing tasks separated systems most sharply, ranging 8.18% to 84.66% across systems versus narrower bands for text workflows. Yet difficulty remains: across 1,300 final trials 30.23% earned zero, worst in create-and-edit at 41.54%. Twenty-four tasks got zero from at least 7 of 13 systems, five edit tasks got zero from all systems, and only 15 tasks ever earned a perfect 1.0. Failure analysis shows zeros usually come from scoring, not crashes: 326 of 393 zeros involved zero geometry, 33 failed spec alone. The two worked examples illustrate both modes: the trussed plate failed mostly on spec with 12 to 80 of 86 parameters unmeasurable despite decent shape similarity, while the clevis passed dimensions for many systems but earned only 0.175 to 0.500 geometry. Astra offered best observed efficiency at $5.59 per score point and 14.42 minutes mean trial time, versus longer and costlier runs for the top scorer. The authors stress limits: 100 curated FreeCAD tasks do not represent all engineering, exclude assemblies, simulation and manufacturability certification, test two-stage construction from scratch rather than editing someone else's document, use heuristic proxies that do not guarantee future editability, and compare complete dated systems confounded by prompts, tools and budgets with only one official trial per task.
Why this might matter to me
For spatial reasoning in LLMs and VLMs, the drawing-to-CAD split is telling: top systems exceed 80% on 40 dimensioned drawings requiring reconciliation of orthographic, section and isometric views, while weaker systems sit below 31%, and the same-agent Opus 5 to 5.5 jump is concentrated in drawings. That suggests genuine progress in reading views, dimensions and geometric relations, but the paper cautions the score bundles perception with tool use, image transport and failure recovery, not vision alone. For world models, the create-and-edit bottleneck matters more than the drawing high scores: preserving a bore, hub and flange envelope while adding a non-circular arc-bounded plate and two opposite-face hole patterns with signed seed centers and angular spacing tests persistent state, constraints and forward prediction of how local operations affect a solid — exactly where all systems drop to around 40%. For text-to-CAD and parametric geometry, the native-document contract plus param_check.py is a useful template: it demands dimensions be measurable in sketches and features, not just visually plausible, and shows that shape similarity without spec consistency earns zero under the failure budget of 10. For 3D generation and scene representation, the contrast with mesh or Blender-script benchmarks is sharp: here boundary-representation solids with topology, surface-type areas, moments and face-center alignment replace perceptual similarity, and the nested harmonic reward offers a way to require both global shape and local dimensional compliance without insisting on one true construction history.
Terms worth knowing
Parametric CAD means a model defined by editable parameters, constraints and operations so changes propagate, not just a static mesh. FreeCAD is the open-source parametric modeler used here, built on the Open CASCADE Technology geometry kernel that handles boundary-representation geometry. Boundary-representation, or B-rep, describes a solid by its faces, edges and how they connect topologically. A feature tree or history is the ordered list of modeling steps like sketch, pad which extrudes a sketch into solid, pocket which cuts, hole, groove, loft, pattern and mirror. A PartDesign Body is FreeCAD's container that should yield one connected solid from those features. An engineering drawing is the traditional 2D input with multiple views, sections and dimension labels. Geometry similarity is the shape-closeness axis, specification consistency is whether named dimensions are present and correct, and task reward is their strict harmonic combination. A witness check measures a dimension directly on final geometry, while a native check measures it from the underlying sketch or feature.