All summaries · 2026-09-26

AgenticCADedit: A Stateful, Tool-Mediated Agentic Approach to Multimodal 3D CAD Editing

Saptarshi Neil Sinha, Mika Silvan Goschke, P. Kühn, et al.

arXiv:2609.29621 · abstract · pdf · summarized by meta/muse-spark-1.3-contributor on 2026-09-26

The one-paragraph version

AgenticCADedit is a different way to let a language model edit an existing 3D engineering model from a realistic, messy human request — someone talking while spinning the part, pointing at a face, and sketching on top of it. The prior benchmark for this task, called neuralCAD-Edit, lets the model try again and again, but each try rewrites one complete program from the original model and throws away partial progress. AgenticCADedit instead keeps one living model in memory and edits it in small steps: run a little code, look at the resulting faces and edges as numbers, render pictures with the intended faces highlighted in red to check the selection was right, and undo just that last step if it was wrong. On the fixed 192-task neuralCAD-Edit benchmark, and holding the model and evaluation method constant, this stateful approach improved every reported average score for all three tested language models, with the biggest jump for the weakest open model, and it cut billed cost because it reuses cached input and emits far fewer new output tokens. Human experts are still far ahead.

The problem they're solving, and why it matters

Computer-aided design, meaning the software in which industrial products are specified before manufacturing, is mostly about editing, not creating from blank. A reviewer asks for a thicker rib, a moved hole, or four added legs; a designer finds the right surfaces, changes them, and repeats if the intent was misunderstood. That communication is inherently multimodal, meaning it uses more than typed text. In the neuralCAD-Edit data the authors reuse, professionals speak aloud while rotating the model, pointing the cursor at the face they mean, and drawing temporary or permanent sketches that indicate where to cut or add. Most neural CAD research has ignored that reality and focused on unconditional generation, meaning making a shape with no instruction, or text-conditioned generation, meaning making a shape from a sentence like “make a table.” Even the neuralCAD-Edit baseline is, in the authors’ words, stateless. Stateless here means no memory of geometry between tries: each iteration writes a whole CadQuery program — CadQuery being a Python library that builds solid geometry with operations like extrude, fillet, and Boolean union — runs it starting from the original part, looks at the render and error, then starts over from scratch. If step one of three was correct, that work is lost. The model also cannot ask “what faces do I have now, how big are they, where are they?” or “did I select the right edge?” or “take back only that last fillet.” The paper asks whether keeping a persistent, inspectable state fixes that.

The key idea, in plain words

Turn editing from “guess the whole final script” into “do a sequence of small, checkable moves on one evolving part,” much like a human designer would. Concretely, instead of regenerating everything, the model keeps a single current shape, called result in the code, alive across turns. It adds a load block, then looks at the new shape. It adds bolt shafts on top of that updated shape, then refines them into heads and shanks. If a selector grabbed the wrong edge, the render with red highlighting shows it, and the model can revert only that operation rather than discarding the good block underneath. Later actions therefore build on, rather than replace, earlier geometry. The tools are organized as a loop the authors call perception–decision–action–verification: perceive with numbers and images, decide to inspect, edit, or stop, act with code, verify with a new inspection or render.

How it actually works, step by step

The system has three pieces: what the model is told, the workshop it works in, and when it stops.

First, context. Before looping, the language model gets an instruction that it lives inside a persistent parametric session — parametric meaning defined by editable operations and dimensions, not just triangles — where CadQuery is already imported, the part to edit is already loaded, variables carry over, and the current geometry is always in result. It is advised to inspect and render before editing, keep code steps small, visually confirm faces and edges before operating, verify after each change, and only declare done when the request is matched. The task itself arrives as text plus video of the designer speaking, pointing, and drawing.

Second, the agentic loop and workshop. A program called the harness shuttles messages between the language model and a Model Context Protocol server — Model Context Protocol, abbreviated MCP, being simply a standard way to expose tools like “run code” or “render” to the model. The harness launches that server as a subprocess that imports CadQuery once and remembers variables. Before every code run it pushes the state onto a history stack, so a crash restores the prior good state automatically.

The action space, meaning the allowed moves, is eight tools:

In each iteration the model sees full history and tool schemas, may call several tools at once — for example edit then immediately inspect — the harness runs them in order and appends results, with images passed back as an extra user message because tool messages cannot carry images. Each tool has a time limit.

Third, exit. The loop ends when the model calls finish, when it answers with no tool call which is treated as implicit done, or when a budget of 100 language-model iterations is exhausted. In all cases the current result is exported and scored against the reference edit.

What they showed — results, honestly, with limitations

They changed only the interaction paradigm, keeping the same 192 expert tasks, spanning text-only, interactive speech-plus-pointing, temporary-drawing, and static-drawing requests, across easy, medium, and hard levels and single versus assembly models, and the same evaluation protocol. Evaluation uses validity, meaning the run produced an exportable solid; Chamfer distance, meaning in words the average nearness between sampled points on the produced and reference shapes where lower is better; voxel IoU, meaning in words the overlap of occupied cubes where higher is better; DINOv2 similarity, meaning similarity of image features from isometric renders where higher is better; and vision-model ratings for instruction-following and model quality on a 1-to-7 scale normalized to 0-to-1, with acceptance requiring both at least 5.

Averaged over all tasks with failures scored as zero, every model improved on every metric. The weakest baseline gained most: with the open-weight model named qwen3.6-27b, validity rose from 51.0% to 94.8% and acceptance from 1.6% to 12.0%. With gemma4-31b, validity went 40.1% to 57.3% and acceptance 2.6% to 6.8%. With the proprietary model named gpt-5.6-luna, validity went 96.4% to 99.0% and acceptance 50.0% to 53.6%. The number of valid outputs also grew substantially — for example 98 to 182 for qwen — while median geometric scores over only the valid outputs improved by smaller margins, suggesting the method mainly turns broken attempts into usable ones rather than making good ones perfect. A stricter pairwise check on only tasks both systems solved validly shows the same pattern, except mean Chamfer distance is noisy because a few badly displaced outputs dominate the average.

Fine-grained acceptance falls with difficulty, and interactive and temporary-drawing requests are hardest for open models, likely because transient cursor motion must be recovered from video frames; permanent static sketches help, especially for the strongest model. Qualitatively, for “add four legs equally spaced,” their method preserves size and grid where the baseline mis-sizes or misplaces; for “merge a cross with a 90-degree rotated copy,” it produces a true four-way cross where the baseline leaves it unchanged or misaligned; for a hard wheel task requiring de-fillet, enlarge to 50 cm, and add treads, open models often do only the enlargement.

Cost with gpt-5.6-luna: their loop processed more nominal input tokens, 301.37 million versus 99.39 million, but 94.8% were served from prompt cache versus 0.2% for the baseline because the append-only session reuses a long shared prefix, and it emitted 66.7% fewer output tokens, 1.80 million versus 5.39 million. Recorded API cost was $11.00 versus $26.30, a 58.2% reduction.

Limitations are plain: the human baseline at 82.3% acceptance and 100% validity remains far ahead; long composite requests and sketch intent are only partially solved; the evaluator itself is a vision model, Opus 5 medium inside Claude Code, which the authors say reproduces expert ratings on the human baseline more closely than prior evaluators, but that is still an automatic proxy.

Why this might matter to me

For your interests in spatial reasoning in large language and vision-language models, this is a clean isolation test: same model, same sampling, same benchmark — only the ability to ground words and gestures to addressable geometry changes. The gains come not from a bigger model but from letting the model ask “which face, how large, where, with what normal?” and then see its selection painted red. That pattern — numeric inspection plus visual verification plus undo — is directly reusable for studying where VLMs fail at pointing, transient sketch tracking, and multi-step spatial plans.

For world models and text-to-CAD plus parametric geometry, the lesson is that editable representation matters. They deliberately stay in executable CadQuery rather than meshes or radiance fields because only operations like fillet, chamfer, shell, and Boolean carry design intent and support selective revert. If you think of a world model as a persistent state that actions update and later actions condition on, this is that idea applied to CAD: result plus history stack is a tiny world model, and persistent variables are its memory.

For 3D generation and scene representation, the cost result is practical. Multi-turn visual grounding looks expensive because input tokens balloon, but with prefix caching it was cheaper in dollars while producing far fewer output tokens. That suggests future text-to-CAD or assembly-generation agents should budget for inspect-render-undo loops and explicit planning over operations, plus tighter grounding of annotated video to geometric entities, which the authors name as next work.

Terms worth knowing