All summaries · 2026-11-04

SketchMate: Agentic Geometric Reasoning for Zero-Shot Precision Insertion

Jiaming Wang

arXiv:2610.41844 · abstract · pdf · summarized by meta/muse-spark-1.3-contributor on 2026-11-04

The one-paragraph version

SketchMate is a robot system for inserting a part it has never seen before — like a USB plug, power plug, or peg — using only a spoken instruction and camera views, with no CAD model or prior training on that part. Instead of asking a vision-language model, meaning an AI that understands images and language, to directly guess where to move, it splits the job: the vision-language model decides what to measure, and precise geometric software does the measuring. The model builds a “sketch,” meaning a very simplified description made of a few block and tube shapes plus two coordinate frames that mark how the parts mate. That sketch is reused for later placements by aligning old 3D scans, and after grasping, the robot takes close-up looks at both mating faces to fix rotation mistakes and tiny offsets before pushing the part in with a gentle, wiggling controller.

The problem they're solving, and why it matters

Insertion is where general-purpose assembly breaks down. A plug must enter its socket with only about 0.1 to 0.3 millimeters of clearance per side and tolerate only one or two degrees of twist error, in the paper’s tasks 0.2 to 0.45 mm sideways and 1.3 to 4.6 degrees. Many parts are also keyed, meaning they only fit in one rotation — think which side of a USB-A plug holds the little plastic tongue, or which way the ground pin of a NEMA 5-15 power plug points.

Industry today solves this by engineering each part separately with CAD models, fixtures, and hand-taught poses. Learning methods train a specialist policy per part family. Zero-shot vision-language methods try to generalize without that work, but they ground language in coarse spatial outputs: labeled dots and grid tiles on an image, keypoints linked by written constraints, or blocky 3D value maps. Those are limited to several millimeters by image patches and voxels, and they force the language model to guess 3D correspondence from a flat picture — a known weakness. The paper shows this is not just imprecise, it misses the opening entirely.

The key idea, in plain words

Let language do semantics and geometry do measurement.

The vision-language model is good at knowing which object is the plug, which hole on a board is the right receptacle, and which feature matters — for example, “the rim of that round hole” or “the flat on that shaft.” It is bad at reporting exact millimeters and degrees.

So SketchMate has the model build a sketch model once per task. A sketch model here means the minimal geometry a planner needs: a handful of solid shapes that roughly cover the moving part for grasping and collision-checking, plus a mate frame for each part. A mate frame means a tip point where contact starts, an approach axis along which the part moves, and a reference direction sideways to that axis that pins down the twist, plus a symmetry order and an insertion depth.

Crucially, a sketch describes the parts themselves, not where they sit on the table. Build it once, then move the parts and you can just carry the sketch along. And because the first sketch from far cameras often gets the twist wrong, look again up close after you have the part in hand, when both mating faces are visible, and let shape fit decide the final twist and nudge.

How it actually works, step by step

A new task starts with an instruction such as “pick up the orange NEMA 5-15P plug and insert it into the blue task board, NEMA 5-15R receptacle,” plus three color-plus-depth views.

1. An agent builds the sketch with tools. A vision-language agent loops, calling geometric tools that run on fused depth from all views. In plain terms: - view shows an image or crop with a pixel grid. - xyz returns the 3D point, surface direction, and object at clicked pixels. - region summarizes an object: height range, fitting upright box, fitting vertical tube, flat faces. - hole fits the rim around a pixel inside an opening: center, circle radius, rectangle, height, visible depth. - top gives a top-down height map with a millimeter grid, free of perspective, where slots and keys are clear. - fit fits a plane, circle, line, tube, box, or pose to chosen 3D points. - check validates a draft sketch against depth: does each shape agree with what was seen, and how much of the part is explained.

The prompt tells the model to take numbers from tools, not from eyeballing images, and to check before answering. Example from the paper: for a DC barrel plug, the model’s first guess for the body tube radius scored poorly on agreement, so it sampled image points, fitted a tube, and corrected the radius from a self-invented value to a measured 4.05 mm.

2. A memory reuses sketches without calling the model. Each run stores the observed 3D points of both parts in table-centered frames, called templates, plus the sketch expressed in those frames and the part colors. On a new placement, the system segments the scene the same way, finds similarly colored and sized objects, and aligns each template by trying many yaw rotations followed by a standard alignment algorithm called ICP, short for Iterative Closest Point, which iteratively nudges one cloud onto the other. Fitness means the share of scene points close to the template. If best fitness is high enough, it transforms the stored sketch into the new scene. Otherwise it calls the agent again.

3. A standard planner and compliant controller execute. The planner samples top-down grasps across the rough shapes, prefers any grip the agent suggested, and plans collision-free paths with a common planner called RRT-Connect. The final placement is computed from the mate frames: axes opposed, reference directions aligned up to symmetry, tip at depth past the entry. A spring-like impedance controller lowers the part, and if blocked, does a spiral and rotary wiggle like classic peg-in-hole search. This can absorb tenths of a millimeter, but not a wrong hole or a 90-degree twist.

4. Look, then mate, after the grasp. This is the correction step. The robot holds the mating end over an upward camera to compute a profile, meaning for each tiny grid cell looking along the insertion axis, the lowest point of the part and fingers in the gripper. It moves a wrist camera close over the socket — about 0.16 mm per pixel — to compute a height map, meaning for each grid cell the highest observed surface, marking cells as open where camera rays pass through, to see into narrow holes.

It then scores every candidate twist and tip position by shape complementarity, meaning how well protrusions of one fit recesses of the other. The score rewards outline cells that sit in free space touching a wall, and penalizes part material that would sink into solid socket material, summed over several depths and counted by cells to tolerate tight fits. It searches twists exhaustively, then refines. Among near-tied fits — symmetric rotations, identical ports, sliding along a bar — it picks the one closest to the sketch. If the sketch twist is far from all fits, it discards it as a misreading. In short, the sketch supplies the opening, axis, depth, and tie-breaker; geometry supplies the twist and sub-millimeter position.

What they showed — results, honestly, with limitations

All tests are in simulation with a Franka arm, using noise-free depth, which the authors flag as a limit. Tasks: 69 development episodes — 42 AutoMate assemblies, 13 procedural connectors and pegs, 10 NIST-style board receptacles, 4 industry-challenge placements — plus 18 held-out connector placements from six new types the method was not tuned on.

In a strict kinematic test that requires within 2 mm and 5 degrees of the assembled pose with no excess penetration, SketchMate inserts 61 of 69 development parts, about 88%. The ported baselines ReKep and MOKA, given the same language model, planner, and controller, insert 0. Their final poses miss the opening by a median of 25 mm and 13 mm, often with wrong keypoints on holders or board edges, or with fixed yaw that cannot express keyed rotation.

Sketch alone inserts 50, with median error 0.15 mm. The close look adds 11 and lowers median error to 0.10 mm. With perfect ground-truth sketches, the same pipeline inserts 64, so the agent-built sketch loses only four parts where it picked the wrong feature or receptacle. On held-out connectors, the full method inserts all 18 in both kinematic and physics tests, versus 6 and 10 for sketch alone. For keyed parts, sketched rotation is within tolerance in 57 of 86 episodes, rising to 83 after the look.

Reuse works: on 32 new placements of eight tasks, reusing one stored run inserts 30 strictly, matching calling the agent each time, but takes 3.4 seconds to produce the sketch instead of about 90 seconds with about 17 tool calls.

Failures are instructive. The agent consistently misreads certain twists — NEMA 5-15 in 7 of 8 placements, XLR in all three — suggesting a stable wrong belief, not noise. The look misses an internal post that hides walls, a duplex SC connector whose two twists differ only by a key below camera resolution, a plug sent to a looser wrong receptacle, and fits tighter than about one close-view pixel. The method also assumes straight insertion along one axis, a rigid grasp after the look, and two extra cameras, and real depth blur is untested.

Why this might matter to me

For spatial reasoning in LLMs/VLMs, this is a clean design pattern to copy: do not ask the model for millimeters or 3D correspondence. Ask it to name objects, openings, and measurement targets, then let typed tools return metric values. The jump from 0 to 61 versus baselines with the same model makes the point that representation, not a bigger model, was the bottleneck, and the consistent misreading of NEMA and XLR twists gives you a concrete probe of where model priors fail.

For world models, the sketch plus memory is a minimal, actionable world model: not a video predictor, but a persistent, transferable state — shapes plus mate frames plus templates — that supports planning, collision checking, and reuse by registration. The 90-second to 3.4-second drop shows the value of caching decisions rather than re-reasoning.

For text-to-CAD and parametric geometry, the sketch is essentially a tiny parametric CAD program inferred from language and depth: upright boxes defined by center, extents and yaw, tubes by two end-face centers and radius, frames by tip, axis and reference. The hole, top, fit, and check loop is a template for language-directed reverse-engineering: propose features in words, fit parameters in geometry, verify coverage automatically.

For 3D generation and scene representation, the two envelopes are worth noting: a lower envelope profile of the in-hand part including fingers, and an upper envelope height map with free-space carving to see into occluded holes. Mating by counting contact versus penetration, rather than taking a max, is a robust way to compare generated or sensed shapes under resolution limits and sensor noise.

Terms worth knowing